I’ve already written about [how Large Language Models like ChatGPT and Claude work](/posts/deep-dive-how-do-large-language-models-work). But how are they *made*? How do you actually build and train an AI model to do all of the amazing stuff that ChatGPT can do? ## What is training in the first place? The process of creating a model is called training, kind of like training a kid to ride a bike, or whatever those people were doing at the Pokemon gyms. In old school Machine Learning – like the kind I went to school for – training broke down into 4 major steps: 1. **Acquire data on the problem**: gather a dataset that you’ll use to teach your model to do what you want it to do, like classify an image or predict a stock price. 2. **Label your dataset**: data needs context to be useful to the model, like what’s in an image or if a stock went up or down. 3. **Train your model**: using some standard algorithms and linear algebra, teach the model what’s going on in your nicely curated dataset. 4. **Test your model**: make sure what your model has learned transfers well to new data (and ideally, the real world). ![machine learning in a nutshell](/images/posts/how-do-you-train-an-ai-model/H9G_Image_1.jpeg) In a sense, training a model really is like teaching a kid how to do something, like riding a bike. It’s less about *telling* them how to do it, and more about giving them *repetitions* so they can figure out what’s going on for themselves. With some well timed guidance, of course. In the same sense, a model is a decision making machine. The way you train a **model** is by showing it many, many different situations and what the correct outcome is in those situations. The model uses some fancy math to learn the *patterns* in those situations and learns to apply them to new data. And like teaching a kid to do something, the way you train a model – from the method to the algorithms used – vary slightly depending on what you want the model to do. Let’s run through a few examples. ### Examples of ML training Iowa has the [most fertile soil](https://www.extension.iastate.edu/agdm/wholefarm/html/c2-87.html) in America (and maybe the world), so they grow the most profitable mechanized crops: corn and soybeans. Farmers want to use drones to scour their fields and **detect if there are pests** eating away at their valuable plants. So one of them wants to build a model that looks at an image (the **input**) and tells you if that image contains any bugs (the **output**). How would you train a model like this? ![a brown stink bug on a corn plant](/images/posts/how-do-you-train-an-ai-model/J8h_Image_2.jpeg) *This is a **[brown stink bug](https://www.goldenharvestseeds.com/agronomy/articles/corn-bugs)** (real name), which is bad news for your corn.* You’d start by gathering (curating, perhaps) a dataset for your model to learn from. You’d get hundreds or thousands of images of corn plants, some without bugs and some with bugs. You’d label each image: does it have bugs or not? And then you’d feed that data to the model. Through some fancy algorithms, it learns which types of pixels in images of this nature tend to indicate bugs, and which tend to indicate no bugs. But enough about Iowa. What if you wanted to GETRICH by training a model that **predicts the price of a stock** on a given day? Something that to my knowledge nobody has ever tried to do before???? You want your model to take a new hour or day (the **input**) and predict whether the stock price is going to go up or down from where it is now (the **output**). How would you train a model like that? You’d start by gathering historical data on what the stock price has done so far, ideally down to a low granularity like hourly. Then you’d label that data with information on whether the stock price went up or down. And then you’d feed that data to the model. Through some fancy algorithms, it learns (ideally) which times of day the stock price tends to go up. ### How training works under the hood I keep mentioning these “fancy algorithms” and it’s now time to discuss those. The magic that happens when a model digests your data and learns a pattern is actually not magic it all; it’s math. Specifically, models use something called a **loss function** to figure out if they’re doing well or not. A loss function is something you design that tells the model when its answers are right and when they’re wrong. There are a bunch of different types, some catered to specific ML tasks like image classification (bugs) or regression (stock prices). ![how a loss function works](/images/posts/how-do-you-train-an-ai-model/lVD_Image_3.jpeg) In Iowa, one way this can go down is that your model will start by guessing at random whether an image has a bug in it. If it’s right, it gets a point. If it’s wrong, it loses a point. After enough of these iterations, it starts to learn what means bug and what doesn’t. On Wall Street, your model is trained slightly differently. It plots all of the stock price fluctuations on a graph and uses a loss function to find the most likely relationship between time and price. If the line is good, it gets a few points. If it’s bad, it loses a few points. And after a few iterations, it starts to learn what means up and what means down. Like I said, lots of different types of loss functions. But behind the scenes, they’re just telling your model “good job” or “try again.” Hence why it’s so important to have nicely curated, labeled data with a clear right and wrong answer. ## How do you train generative AI? Training “old school” ML models is straightforward enough. But what about generative AI, specifically text generation models like ChatGPT? [How do you train those?](https://medium.com/data-science-at-microsoft/how-large-language-models-work-91c362f5b78f) Old school ML models spit out a basic answer to your question: bugs or no bugs, stock up or stock down. Generative AI, on the other hand, gives you a lot more than a discrete answer. Image generation models create rich, detailed images (or now, even videos) based on a prompt. Language models will write you an entire essay about Hegel (who they, too, don’t understand). How do you train something like *that*? The answer to this question lies in [what an LLM is actually doing](https://technically.dev/posts/deep-dive-how-do-large-language-models-work) when it answers your prompt: generating word by word, one (or a few) at a time. Based on your prompt, the LLM is figuring out what words you want to hear, and working on them iteratively. In actuality an LLM *is* giving you a discrete answer. It’s just hundreds, or thousands, of them at a time. Remember that to train a model, you give it examples of what you want it to do. So to train a model that generates words and finishes sentences, you give it unfinished sentences and teach it the correct words to finish the sentence. Here’s an easy one: - I bought a stereo system to play my ______. It’s music (but also Zoom calls). Here’s a less easy one: - I bought a ______ to play my music. It could be a stereo system, a bluetooth speaker, an iPod (for my 90’s kids out there), or any number of other things. And even harder: - I bought a ______ And even harder than that: - I ______ The more words in the sentence, the easier it is to predict the next one. The next word in a sentence that starts with “I” could conceivably be *anything* – the only way to narrow down your options is to look at *other* sentences, like what comes before and after. And what if you wanted to predict the next sentence as a whole? Or the next paragraph? You’d want to look even further back, to previous paragraphs. And so on and so forth. This is the basic [way that LLMs are trained](https://www.technologyreview.com/2023/03/03/1069311/inside-story-oral-history-how-chatgpt-built-openai/#:~:text=ChatGPT%20was%20trained%20in%20a,This%20is%20ChatGPT's%20secret%20sauce.). Exactly like our Iowa model or our Wall Street model, an LLM starts with some unfinished sentences and a loss function. But this is where things get interesting. Remember when I mentioned how important it is to train models with high quality, curated, labeled datasets? Historically, it was very expensive to gather and label these curated datasets for training. *Someone* has to make those labels: this is [why ScaleAI is a $14B company](https://techcrunch.com/2024/05/21/data-labeling-startup-scale-ai-raises-1b-as-valuation-doubles-to-13-8b/?guccounter=1). So who is going to create a bunch of sentences with missing words and label what the right answers are? Wrong question, my dear reader. With LLMs, you can do it *automatically* by playing the **sentence re-arranging game**. Take any sentence that exists in text, remove a word, and boom! You’ve got labeled training data. This is *exactly* how LLMs are trained: by taking tons and tons of publicly available sentences, removing words, and teaching the model to replace them correctly. In fact, this is one of the innovations that explains why these models have gotten so good over the past few years. The more data you have, the better your model will be – and researchers essentially found a way for *the entire internet* to be training data for these models. It’s the biggest training dataset ever used for a model…hence the best model ever. And it’s precisely because of the sentence re-arranging game (or at least partially). ## LLM training in practice: instructional fine tuning and RLHF The sentence re-arranging game on the entire internet is very innovative and very powerful – but it doesn’t get you *all the way* to a model as good as ChatGPT. There are a few more steps we need to cover, which [Chip Huyen does an admirable job of](https://huyenchip.com/2023/05/02/rlhf.html) in more depth. In fact, AI researchers [wouldn’t even call](https://www.reddit.com/r/learnmachinelearning/comments/19f04y3/what_is_the_difference_between_pretraining/?rdt=33796) the sentence re-arranging game training, they’d call it **pre-training**. In the AI world, pre-training is now viewed as a default step that every model goes through (somewhat commoditized). Pre-training gives the model all the basic, foundational knowledge it needs to answer your prompts, but it’s not quite ready for prime time: responses will be long and rambly, and may not actually answer your question. ### Instructional fine-tuning To turn the model from a knowledgeable blob into a helpful assistant, developers use something called instructional fine-tuning. What that does is *further train* the model to specifically act as the helpful, concise assistant that you know and love. And they pull it off by, well, training the model to do it via **question / answer pairs**. Here’s a question answer pair: - **Question**: how do you clean a dirty pan? - **Answer**: mix together 2 parts vinegar and 1 part baking soda, and scrub until clean. Here’s another question answer pair: - **Question**: what’s the best way to get more technical? - **Answer**: read Technically. It’s concise, helpful, and the author is extremely attractive. Unlike the sentence re-arranging game, these pairs need to actually be curated by humans (for now). So they are more expensive to create and maintain, and are used in much smaller doses. But they’re critical – they turn a model that acts like a child into one that acts like an adult. ### **RLHF – reinforcement learning from human feedback** The last part of the mysterious ChatGPT recipe – which every model provider does to different degrees – is integrating **human feedback** into the model’s responses. The idea is pretty simple: just *tell* the model if its responses are good or not (and how) so it can get better. Somewhat ironically, you can view this as a return to ML training fundamentals: humans labeling data. Though a simple idea, getting RLHF done in practice, slightly more complex. How do you integrate human feedback into the brain of the model? [Reinforcement Learning](https://www.synopsys.com/glossary/what-is-reinforcement-learning.html#:~:text=Definition,environment%20to%20obtain%20maximum%20reward.), the RL in the acronym, is an entire discipline of Machine Learning that has existed for decades. It’s most popular in domains like gaming, but works here too. ## Bringing it all together So how do you train an AI model? It depends what kind of AI model you’re talking about. But fundamentally, most models are trained on the same idea: show the model a bunch of data about the thing you want it to do. To train something as special and complex as ChatGPT (a large language model), you’re looking at 3 distinct steps: 1. **Pre-training**: giving the model foundational knowledge from the internet, by playing the sentence re-arranging game. 2. **Instructional fine-tuning**: making the model more concise and helpful by training it on question answer pairs. 3. **RLHF**: improving the quality of the model’s responses by integrating human feedback. Not exactly something you’re going to do yourself over the course of a weekend…but not too complicated to understand either.