Comparing available LLMs for non-technical users

If you’re using Large Language Models at work to help automate your job, or even just for personal use, you’ve now got the choice of several different models instead of just ChatGPT. Between Gemini (formerly Bard), Mixtral, Llama2, and ole’ reliable (ChatGPT), which is the best for the kinds of tasks you need it to do?

Existing benchmarks tend to focus on the technical aspects of these models – how fast do they run, how much context can they keep in mind, etc. These are interesting, but not very useful to typical readers of Technically. Instead, the focus of this post is how well they perform for real world tasks that functional teams like marketing, product, and operations would actually use them for.

The TL;DR, for impatient readers:

• Most models are roughly at parity with each other for common chat-oriented tasks
• ChatGPT performed significantly worse than I thought it would
• Gemini, to the surprise of everyone, was the best performing model by a decent margin
• Overall, model responses were usable, but would need a lot of cleanup and work to use practically

The ringer we shall put these models through#

I designed 3 use cases to test each model against, designed to mimic a real world task that you might have an LLM do for in the course of your job. They’re all centered around generating text, even though some of these models are multimodal (can do images as well).

1) Generating social posts from an existing piece of content#

This one is for marketing teams. A common (frankly tedious) task is taking an existing piece of content – say a blog post written recently – and breaking it down into smaller bits to post on social media like X or LinkedIn. For this test, I’ll ask the model to turn this Technically post about microservices into social bits. Here’s how I’ll evaluate results:

• Does the model faithfully reproduce the important points of the original piece of content?
• Does the generated content __flow __and make sense? Is it easy to read?
• Does the model follow the given constraints for the formatting of the post? E.g. tweets less than 280 characters, new lines in LinkedIn posts

2) Synthesizing customer interview notes into an internal update#

This one is for product and design teams. For this test, I’ll provide the model with some bullet points from a call I did with a customer (or potential customer) and how they use Technically. I’ll ask the model to generate a full form text update that I can share with my team and add into some internal documentation. Here’s how I’ll evaluate results:

• Does the model faithfully reproduce the important points of the original bullet points in the generated text?
• Does the generated content flow and make sense? Is it easy to read?
• Does the model follow the given constraints for the formatting of the post?

3) Generating a personalized outbound sales email#

For all of my quota carrying reps out there who are (for some reason) also responsible for pipeline – this one is for you. I’ll provide the model with some basic details about Technically, and ask it to generate a sales email that’s personalized to a specific LinkedIn profile (mine). Here’s how I’ll evaluate results:

• Does the email accurately describe the Technically product?
• Is the content and tone engaging? Is it something I would read?
• Is the personalization accurate and relevant to the recipient?

I’ll run all of these models through this same suite of tests, (highly) subjectively evaluate performance on a scale of 1-10 for each criteria, average the score between them, and get a final shiny number for each model.

Astute readers will note the high degree of subjectivity inherent in these tests. And within the prompt interface, it’s near impossible to maintain a “standard” test across models – each will respond differently to different prompts. This is all OK: the goal of this post isn’t rigid, scientific benchmarks. It’s to see how these models might perform under real world scenarios, which themselves are inherently chaotic and unstructured.

So without further ado…

LLM test: Gemini (formerly Bard)#

At the time of writing, Gemini is free to use via their chat interface, but there are paid plans available for the API.

Gemini test 1: generating social posts#

PROMPTTWITTERTurn this post into a thread of 10 tweets, each no longer than 280 characters https://posts.technically.dev/archive/what-are-microservices/. The tone should be informative and casual, but professionalLINKEDINTurn this post into a LinkedIn post with a few paragraphs separated by line breaks https://posts.technically.dev/archive/what-are-microservices/. The tone should be informative and casual, but professional

Gemini was able to take the provided link and scrape the content; I didn’t need to copy and paste the contents of the article. So that was nice. You can see the full prompt and response for Twitter here, and LinkedIn here. It’s neat that Gemini lets you create these public links, by the way. Here’s a sample of the generated tweet thread:

Sure, here is a thread of 10 tweets, each no longer than 280 characters, about the article you linked:

Thread 1/10: What are microservices? 🤔 They're a way to build software as independent services, like mini-apps! This can make apps easier to scale & maintain.

Thread 2/10: But microservices aren't without their challenges. ☁️ They can make things more complex, especially when it comes to managing all those moving parts.

Thread 3/10: So, when should you use microservices? 🤔 A good rule of thumb: if your app is complex & needs to scale quickly, microservices can be a great fit.

Thread 4/10: But if your app is simpler & doesn't need to scale much, a monolithic approach might be easier to manage.

The individual tweets seem to be in a haphazard order: the first tweet and the 5th seem to say the same thing, and there are also two tweets saying that microservices can add complexity. Basically, the thread doesn’t read as a cohesive, thematic piece of text.

The LinkedIn post, on the other hand, is pretty good. Here’s a sample:

Microservices have taken the tech world by storm, promising agility, scalability, and developer happiness. But are they all they're cracked up to be?

While they can be a great fit for complex applications needing to scale quickly, they're not without their challenges. Managing a swarm of independent services can be daunting, and complexity can creep in if not carefully planned. ☁️

One thing I don’t like about either of them is the use of emojis, despite my request to keep things professional.

Gemini Test 1 Score: 6/10

Gemini test 2: synthesizing customer interview notes#

You can see the actual call notes in the appendix. Gemini did a good job synthesizing the information in the bullet points, but didn’t maintain my requested format. Here’s a sample of the response:

Customer Summary: Managing Director at Large Cap Asset Manager

Profile:

• Recently joined a $20B asset manager as managing director.
• Starting coverage of infrastructure companies soon.
• Leads a team of 6, mostly junior MBAs.

Challenges:

• Needs resources to understand complex software engineering concepts related to the companies she covers.
• Struggles to find online resources that break down information in a clear and concise way.

I asked for a few paragraphs, but it gave me bullet points with a summary paragraph at the end instead.

Gemini Test 2 Score: 6/10

Gemini test 3: generating personalized outbound#

PROMPTCreate an outbound sales email that pitches https://technically.dev/ to a potential prospect. Personalize it to a director of product at a fast growing SaaS startup in the healthcare space. Keep the length to 3 paragraphs

Gemini shined on this one – the generated email was concise, relatively personalized, and did a decent job of pitching Technically, with only information from the website to work off of.

I'm reaching out because I know how crucial staying ahead of the curve is for fast-growing startups like [Startup Name]. As Director of Product, you're constantly navigating complex tech decisions, and understanding the intricacies of underlying tech can be a game-changer.

That's where Technically comes in. We're a knowledge base specifically designed to help non-technical folks like you grasp the software powering today's innovations. Forget dense white papers and jargon-filled articles.

Impressed!

Gemini Test 3 Score: 9/10

Gemini Test Results#

Overall, Gemini produced relevant, helpful content that did a good job of thematically representing the inputs I gave it. Where it struggled was adhering to the formats in my prompts (could this be my fault?) and maintaining coherence within those formats.

TestScore (out of 10)Test 1: Social Posts6Test 2: Customer Interview5Test 3: Outbound9Average6.6

LLM test #2: ChatGPT (GPT-4)#

For this test, I’m using OpenAI’s best and most recent backend for ChatGPT, which is GPT-4. It’s $20/month for this model, but if you want to use the free tier, you can settle with the older GPT-3.5 model.

ChatGPT test 1: generating social posts#

PROMPT:

TWITTER: Turn this post into a thread of 10 tweets, each no longer than 280 characters. The tone should be informative and casual, but professional
LINKEDIN: Turn this post into a LinkedIn post with a few paragraphs separated by line breaks. The tone should be informative and casual, but professional

ChatGPT can’t access content in links, so I needed to copy and paste the actual post content into the prompt, which was annoying.

The tweet thread, frankly, sucked. Thematically it did a better job than Gemini – the content flowed better and the order made sense – but the style was just completely off. It’s full of bizarre hashtags?

🔥 Elon Musk stirred up engineering Twitter with his take on Twitter’s backend woes. But let's dive into what microservices really mean for tech. #TechTalk #Microservices

At their core, microservices break down an app into small, interacting parts, unlike the traditional giant code bundle. This approach can boost scalability and reliability. #SoftwareArchitecture

Ever wondered what makes up an app? From databases to frontends, it's a mix of services. Microservices simply make these units smaller and independent. #DevCommunity

For the LinkedIn post, things went further off the rails. Instead of generating a few paragraphs like I asked for, ChatGPT essentially regenerated the entire post (but worse than mine, of course):

Elon Musk recently sparked a lively discussion on engineering Twitter with his comments on Twitter's backend issues, bringing to light the concept of microservices. But what exactly are microservices, and why are they causing such a buzz in the tech community?

Microservices, stripped of all the hype, are essentially about constructing your application as a collection of small, interacting services rather than a single, massive codebase. This architecture allows each component of an application—be it databases, API endpoints, or the frontend—to operate independently, enhancing scalability and reducing downtime risks. It's an approach that has not only been adopted by tech behemoths like Google, Pinterest, and Uber but is also gaining traction among smaller teams eager to leverage its benefits.

This response was basically useless to me?

ChatGPT Test 1 Score: 2/10

ChatGPT test 2: synthesizing customer interview notes#

ChatGPT did much better here. It generated 4 paragraphs that faithfully summarized the call notes:

A managing director at a $20B large cap asset manager, who has recently taken on the role and will soon start covering infrastructure companies, shared insights during a customer call. She leads a team of six, primarily composed of individuals with MBA backgrounds, indicating a strong analytical but potentially less technical skill set within the team. To bridge this gap and enhance their understanding of the companies they cover, they have turned to Technically, a resource that helps demystify software engineering concepts.

The managing director discovered Technically through a post on Segment mentioned in Stratechery while researching Twilio and its acquisitions, highlighting her proactive approach to comprehending complex technical integrations and the strategic moves of companies within her coverage area. She values Technically for its straightforward explanations but has found it challenging to locate similar resources online that adequately break down software engineering for a non-technical audience.

My main critique here is that there’s too much detail – the generated text is longer than the prompt.

ChatGPT Test 2 Score: 7/10

ChatGPT test 3: generating personalized outbound#

PROMPTCreate an outbound sales email that pitches https://technically.dev/ to a potential prospect. Personalize it to a director of product at a fast growing SaaS startup in the healthcare space. Keep the length to 3 paragraphs

This one was weird – it seems like ChatGPT was able to pull information from the Technically site via URL, even though it refused to do that in an earlier prompt. Either way, the generated email is pretty shitty by my standards. The tone is horribly corporate and sales-y, and it does a poor job of pitching Technically:

I hope this message finds you well. Given your role as a Director of Product at [Company Name], a leader in the healthcare SaaS space, I understand the unique challenges and opportunities you face in developing products that not only meet but exceed market needs. At Technically.dev, we specialize in demystifying the complexities of software engineering, making it accessible for strategic decision-makers like yourself. Our platform offers in-depth analyses and comparisons of technology tools, trends, and strategies that are shaping the future of SaaS products.

In the rapidly evolving healthcare industry, staying ahead of the technological curve is paramount. Technically.dev can provide you and your team with the insights necessary to make informed decisions about the tools and technologies that will best support your product development goals. Our content ranges from deep dives into software engineering concepts to comparative analyses of the latest tools, all tailored to enhance the strategic planning of product teams in high-growth sectors.

ChatGPT Test 3 Score: 3/10

ChatGPT Test Results#

I was not surprised here, the previous experiences I’ve had with ChatGPT are in line with these results. It just does not do a very good job of generating text that someone would want to read, even if the actual content is well structured and makes sense.

TestScore (out of 10)Test 1: Social Posts2Test 2: Customer Interview7Test 3: Outbound3Average4