What is Generative AI?

If you’re reading this, you’ve probably already a decent grasp on what run of the mill Machine Learning is. But how do you go from a model that does something tractable, like predicting a stock price or identifying bugs in a cornfield, to a model that creates words from scratch? Or in other words…what is a Generative AI model actually doing?

First, I’m going to run through in general how GenAI works and how models create things. Then I’ll go into each of the two most popular types of GenAI – image generation and text generation – and cover each in a bit more depth.

What is a Generative AI model doing?#

Old school Machine Learning was all about prediction. You’d train a model on a dataset, and use it to predict what’s going to happen when new data enters the fray, like a new day in the stock market or a new image of some corn in your field.

Generative AI is also all about prediction. But instead of predicting a highly scoped, specific thing – like a number, or a yes/no answer – it predicts entire sentences, paragraphs, images, videos, or even audio. It’s trained to generate entire swaths of new data based on your prompts.

So even though the techniques and style have changed quite a bit over the past 5 years, GenAI and old school ML aren’t so far apart; they’re both learning patterns in data, and then using those patterns to do something.

Types of GenAI models#

There are a bunch of different types of GenAI models, some of which have been around for a while. You’ve got Generative Adversarial Networks (GANs), where one model creates something and a sister model critiques it. You’ve got Variational Autoencoders (VAEs), which use probability distributions. Then you have Recurrent Neural Networks (RNNs), which is a special type of neural network that predicts sequences of words. There are even more with even longer acronyms.

But most of the advances over the past few years have come from two specific types of GenAI models:

1. Transformers – mostly for text generation
2. Diffusion models – mostly for image and video generation

Like any software system, each has things that it’s good at and things that it’s less good at. Let’s run through each type of model, where they came from, and how they work. In practice, many state of the art models today are using some combination of both.

Text generation models and transformers#

Most of what’s wow-ing people today in GenAI is LLMs, or Large Language Models. ChatGPT, Claude, Gemini, and the like are all based on a specific type of model called a transformer.

Transformers and attention #

You’ve probably heard of neural networks: they’re a type of ML model that powers most of what people are doing in AI today. Standard neural networks process information sequentially: think, one word after another after another.

To illustrate (😈) how sequential processing works, imagine if you’re reading a book, but could only keep the previous word in memory. If every time you moved on to a new word, you forgot the previous one. You’d never understand what you’re reading, because human language has long term dependencies. What happened earlier in the sentence, or even 2 chapters ago, could be critical to understanding what’s happening right now! And so neural networks struggled to generate decent sentences and paragraphs, because they lacked that context.

Enter transformers, via a seminal paper in 2017. They’re a clever way for models to handle those longer term relationships in text: instead of models processing word by word, they’re able to focus on the most important part of the sentence, no matter where it is logistically. Researchers called this the attention mechanism.

LLMs are pre-trained on massive, massive datasets (in many cases, the entire internet). With transformer-based architectures, they’re able to focus on the meaningful relationships between words, paragraphs, and even chapters instead of just learning word by word. And the result, increasingly, is models that can create text that’s really compelling.

How text generation models actually work#

Speaking of generating text, how does that actually work in practice? It’s surprisingly simple: The way that ChatGPT and LLMs generate these entire paragraphs of text is by playing the word guessing game, over and over and over again. Here’s how it works:

1. You give the model a prompt
2. It predicts a word based on the prompt
3. It predicts a 2nd word based on the 1st word
4. It predicts a 3rd word based on the first 2 words

…and so on and so forth. There are a few frontier models that generate more than 1 word Technically, it’s 1 token, but most of the time that equates to one word. at a time, but generally this has been state of the art for a while. It’s really very primitive when you get down to it. But it turns out that the word guessing game can be very powerful when your model is trained on all of the text on the entire internet.