Why couldn't it count the R's?#
The counting Rs problem was part of an interesting paradox for LLMs. They seemed superhumanly good at some things. But on basic stuff, like comparing which of two numbers was bigger, they’d mess up often. How do you explain that discrepancy?
It was mostly technical. Older models couldn't count Rs, or struggled to compare numbers, for two reasons:
- First of all, they don't see text like we do. That text is translated into a complex series of machine-readable numbers, and that’s what the model sees.
- And second of all, they only got one snap decision to figure it out instead of being able to think about it.
Models don't read letters. They read tokens, and “strawberry” gets split into chunks like "str," "aw," and "berry." So the model can’t “just see the Rs” like we can.
Before reasoning was introduced, models got one chance to guess the answer. Just one forward pass through the model – a single run through of the model’s weights doing their math – could decide the critical token that determines whether the answer is right or wrong. Remember, models generate one token at a time!
Researchers began to discover what they called the rationalization problem. The rationalization problem is when a model produces an answer before reasoning about it and then, after the fact, comes up with a reason for why it gave that answer. I mean hey, humans do it too.
This behavior pattern was very common with pre-reasoning models. If you asked them why they had done something, they would just make something up. That's how you got a model that said strawberry had two R's, then gave a lecture if you asked why it thought that.
Training a model to reason instead#
So in essence, models got simple things wrong because they couldn’t think. But what if they could think? You could sort of ask models to do this in the past, like asking them to use chain of thought, but it didn’t really work – because these models weren’t trained to think. Until…
In September 2024, OpenAI released o1, a reasoning model that thought internally before responding. It wasn't just simple chain of thought prompting, the model had actually been trained to reason. So how did they do that?
Well, OpenAI, of course, didn't give us all the details. But DeepSeek did. In January 2025, DeepSeek published DeepSeek-R1, their first reasoning model, and they open-sourced it, along with a paper that explained the details of how it was trained. They used a technique called RLVR, or reinforcement learning with verifiable rewards.
The basics go like this. You give the model an environment where it can attempt to answer questions where the answer is already known by humans. With arithmetic, for example, you can easily verify whether or not the model’s answer was correct in absolute terms.
This verifiability is powerful, because it means you can then train the model to find the right answer consistently, and it can learn any techniques it needs to get there by itself. You basically give the model a destination, and RLVR optimizes the model until it can consistently get there. You specifically do not tell it how to get there, you make it learn itself, rewarding it along the way when it gets things right.
This ended up working really well. Researchers discovered new reasoning techniques like backtracking that weren't explicitly trained for. They didn't give R1 made-up examples of aha moments. It came up with that on its own during training.
The model wasn't under any optimization pressure to make the chain of thought something that a human would like – it didn’t even need to be in English! It could focus purely on effectiveness.
This is part of why technical domains like math and coding have arguably been the areas where models have performed the best (in addition to economic incentives for the labs).
Math and coding are highly verifiable domains. It’s pretty easy to verify, for sure, that a model answered a question correctly. The solution is either right or wrong. And when there's a test suite like this you can use to verify that a problem is solved in code, that becomes something you can train on (try, score, adjust, repeat).
Math is the easiest to make into verifiable problems. But other domains can be trained for as they become more and more verifiable, like legal or biology. So creating evals (tests that score a model's answers) and these RL environments is what drives a lot of progress in models. There are companies like AfterQuery that do this for the labs on a large scale.
How reasoning works, and what it looks like#
Here's what that actually looks like in the model’s mind. This example is from the DeepSeek paper. Part way through a math problem, the model says "Wait, wait. Wait." and goes back to re-check its own work.
Internally, models aren’t always reasoning in human-readable format. Neuralese can refer to the language in which different sections of the model talk to each other, the raw numbers passed around inside it. This is an example of a chain of thought from Claude Fable, which doesn't look anything like what you would see if you asked a non-reasoning model to think step by step.
Strange languages aside, you can get an intuition for why this capability has dramatically improved model quality. Some problems don't fit in just one pass through a model's "mind." They take cycles of iteration to understand properly and complete.
That's what happened to strawberry. The model doesn't have to guess in one shot anymore. It can spell the word out, count the R's, and go back and check if the number looks off. It's the same thing the model is doing in the screenshot, just on an easier problem.
One interesting thing to note is that those cycles of iteration don't necessarily have to be in the form of additional readable tokens that lay out its thinking. In fact, researchers showed in 2024 that models can be trained to do some of this reasoning with a series of just the same token over and over again, with a little dot.
And it's reported that OpenAI's GPT-6 Astra loops parts of the model internally before producing each token, so it gets more cycles per token. OpenAI hasn't confirmed it, though.
Is it actually thinking?#
Beyond the typical sociocultural debates about whether models thinking is really thinking in the human sense, some have argued that the reasoning we see from models isn’t the real deal.
In June 2025, Apple published a paper called The Illusion of Thinking. In a nutshell, it says that reasoning models "face a complete accuracy collapse beyond certain complexities." In other words, reasoning is shallow.
But the paper was deeply flawed. Because the max output tokens of the models were limited at the time, they couldn't manually go through every step in the task Apple gave the models to complete before running out of tokens. A later replication found that limits like this explain part of the collapse, and that some of Apple's tasks were literally impossible.
The title, and the coverage that followed, then implied that all of the reasoning these models were doing was actually just fake imitations of reasoning. Not true!
That would be like trying to measure whether you have comprehension of math by seeing what number you can count to in your head within 25 minutes.
The Cope Corner: If you are one of those Gary Marcus or Yann LeCun types that feels the urge to yell that it's just predicting the next token and say it's not really thinking, then whatever, I don't care. This is the best way to describe what's going on.
Saying that AI is just predicting the next token is like saying human brains are just orchestrating electrochemical reactions.
It doesn't tell you anything about what they're doing, it's just saying how they're doing it. You don't fare any better in a tiger attack by knowing that the tiger is just made out of flesh and bones.
Can we trust the scratch pad?#
There's a better critique of reasoning, though.
Chains of thought, the text a model writes on that scratch pad while it reasons, are not entirely honest. Anthropic found that chains of thought often don't acknowledge cheating behaviors, such as influence from a suggested answer. In one test, Claude 3.7 Sonnet mentioned a hint it had used only 25% of the time, and DeepSeek R1 39%.
So it's not a perfect window into the mind of a model. But it is better than nothing, and it's our best tool for monitoring what a model is doing.
One interesting phenomenon is that if models are being trained and know that their chains of thought are being monitored, we quickly lose one of our best tools for safety. OpenAI tested this in 2025. It penalized models during training for writing down plans to cheat. The models kept cheating and just stopped writing it down.