That's easy to explain, and it's shocking how many people are baffled by this and use it as proof that LLMs can or can't reason when it has nothing to do with that, but just with the input that LLMs get.
LLMs don't actually "see" individual input characters, they see tokens, which are subwords. As far as they can "see", tokens are indivisible, since the LLM doesn't get access to individual characters at all. So it's impossible for them to count letters natively. Of course, they could still get the question right in an indirect way, e.g. if a human at some point wrote "strawberry has three r's" and this text ends up in the LLM's training set, it could just use that information to answer the question just like they would use "Paris is the capital of France" or whatever other facts they have access to. But they can't actually count the letters, so they are obviously going to fail often. This says nothing about their intelligence or reasoning capability, just like you wouldn't judge a blind person's intelligence for not being able to tell if an image is red or blue.
On the other hand, writing code to count appearances of a letter doesn't run into the same limitation. It can do it just fine. Just like a blind programmer could code a program to tell if an image is red or blue.
> just like you wouldn't judge a blind person's intelligence for not being able to tell if an image is red or blue.
I would judge a blind person's intelligence if they couldn't remember the last sentence they spoke when specifically asked. Or if they couldn't identify how many people were speaking in a simple audio dialogue.
This absolutely says something about their intelligence or reasoning capability. You have this comment:
> LLMs don't actually "see" individual input characters, they see tokens, which are subwords.
This alone is an indictment of their "reasoning" capability. People are saying these models understand theoretical physics but can't do what a 5 year old can do in the medium of text. It means that these are very much memorization/interpolation devices. Anything approximating reasoning is stepping through interpolation of tokens (and not even symbols) in the text. It means they're a runaway energy minimization algorithm chained to a set of tokens in their attention window, without the ability to reflect upon how any of those words relate to each other outside of syntax and ordering.
>This alone is an indictment of their "reasoning" capability.
I'm not sure why it says anything about their reasoning capability. Some people are blind and can't see anything. Some people are short-sighted and can't see objects which are too far away. Some people have dyslexia. Does it say anything about their reasoning capability?
LLMs "perceive" the world through tokens just like blind people perceive the world through touch or sound. Blind people can't discuss color just like LLMs can't count letters. I'm not saying LLM's can actually reason, but I think a different way to perceive the world says nothing about your reasoning capability.
Did humans acquire reasoning capabilities only after the invention of the alphabet? A language isn't even required to have an alphabet, see Chinese. The question "how many letters in word X" doesn't make any sense in Chinese. There are character-level LLMs which can see every individual letter, but they're apparently less efficient to train.
The reason it is an indictment of their reasoning capability is that—no matter how much energy is spent trying to say they are not—these are really stochastic parrots: they do not understand the symbols they are operating with. They operate below that level.
The fact they can't operate on full symbols reliably but require sub-symbols via tokens is concrete proof of that. They may add heuristics or build more CoT sub-chains to get around some of these trickier issues later, but this is the state of affairs right now.
All efforts so far require exponential increases in training size to receive logarithmic increases (at best) in accuracy. And now with o1, it requires exponential compute at inference to scale with that sub-logarithmic accuracy.
People have a short memory these days, but around GPT-3, the majority of people on HN and tech "luminary" founders were saying that these would actually have exponential output and diverge. They were wrong. These models are quickly converging to a training set because they are and always were a curve fit. And even there, they are notoriously unreliable for use cases without a human in the loop, because of the intrinsic amount of information entropy that can be packed into the size of these models. But there is nothing mysterious about them.
Way weirder than this is that LLMs are frequently correct in this task.
And if you forgo the counting and just ask it to list the letters it is almost always correct, even though, once again, it never sees the input characters.
Much has been written about how tokenization hurts tasks that the LLM providers literally market their model on (Anthropic Hiaku, Sonnet): https://aclanthology.org/2022.cai-1.2/
The problem is not the addition, is that the LLM has no way to know how many r's a token might have, because the LLM receives each token as an atomic entity.
For example, according to https://platform.openai.com/tokenizer, "strawberry" would be tokenized by the GPT-4o tokenizer as "st" "raw" "berry" (tokens don't have to make sense because they are based on byte-pair encoding, which boils down to n-gram frequency statistics, i.e. it doesn't use morphology, syllables, semantics or anything like that).
Those tokens are then converted to integer IDs using a dictionary, say maybe "st" is token ID 4663, "raw" is 2168 and "berry" is 487 (made up numbers).
Then when you give the model the word "strawberry", it is tokenized and the input the LLM receives is [4463, 2168, 487]. Nothing else. That's the kind of input it always gets (also during training). So the model has no way to know how those IDs map to characters.
As some other comments in the thread are saying, it's actually somewhat impressive that LLMs can get character counts right at least sometimes, but this is probably just because they get the answer from the training set. If the training set contains a website where some human wrote "the word strawberry has 3 r's", the model could use that to get the question right. Just like if you ask it what is the capital of France, it will know the answer because many websites say that it's Paris. Maybe, just maybe, if the model has both "the word straw has 1 r" and "the word berry has 2 r's" and the training set, it might be able to add them up and give the right answer for "strawberry" because it notices that it's being asked about [4463, 2168, 487] and it knows about [4463, 2168] and [487]. I'm not sure, but it's at least plausible that a good LLM could do that. But there is no way it can count characters in tokens, it just doesn't see them.
Tokenization does not remove information from the input[1]. All the information required for character counting is still present in the input following tokenization. The reasons you give for why counting characters is hard could be applied to essentially all other forms of question answering. Ie, to answer questions of type X in general, the LLM will have to generalize from questions of type X in the training corpus to questions of type X with novel surface forms which it sees at test time. [1]tokenizers can remove information if designed to do so, but they don't in these simple scenarios
As far as I know, that's not the case. The tokenizer takes a bunch of characters, like "berry", identifies it as a token, and what the LLM gets is the token ID. It doesn't have access to the information about which letters that token is composed of. Here is an explanation by OpenAI themselves: https://help.openai.com/en/articles/4936856-what-are-tokens-... - as you can see, "Models take the prompt, convert the input into a list of tokens, processes the prompt, and convert the predicted tokens back to the words we see in the response". And the tokens are basically IDs, without any internal structure - there are examples there.
If I'm missing something and you have a source for the claim that character information is present in the input after tokenization, please provide it. I have never implemented an LLM or fiddled with them at low level so I might be missing some detail, but from everything I have read, I'm pretty sure it doesn't work that way.
A sequence of tokens can be converted back to the sequence of tokenized characters without loss of information. Eg, how do you think text is rendered for the user based on sequences of tokens generated by the LLM? Different tokenization schemes arrange that information differently and may make it (hand waving here) harder or easier for the model to reason about details like raw character counts that are affected by tokenization. If the training set included sufficiently many examples of character counting Q/A pairs, an LLM would have no trouble learning how to do this task.
Thank you for taking the time to write this response. Unfortunately, even though I agree that tokenization makes it pretty hard for the LLM to count characters, I'm still not convinced that it is a fundamental problem for doing so. I think the lack of (or limited amount of) symbolic processing is an even more important factor.
> But there is no way it can count characters in tokens, it just doesn't see them.
If that is the case, then how can most LLMs (tested with ChatGPT and Llama 3) spell out words correctly?
Might that also be the answer to why it says "2"? There are probably sources of people saying there are two R's in "berry", but no one bothers to say there is 1 R in "raw"?
It doesn't see "straw" or "berry". It sees a vector which happens to represent the word strawberry and is translated from and to English on the way in and out. It never sees the letters, 'strawberry' is represented by a number, or group of numbers. Try to count the Rs in "21009873628" - you can't.
I'm aware of this. The network could, and apparently does, associate single characters with words. It can associate "red" with "rose", and might associate "r" with "straw", and it might even associate some kind of embedding of "two r's" with "berry".
Nope, I'm not missing that particular fact. I'm aware that sentences (and words) are split into tokens, which are vectors.
I don't understand how most LLMs can spell out words though, nor do I understand what is causing the failure to count characters in words. I was not convinced by the comment I was responding to.
> it's shocking how many people are baffled by this
Is it? These stupid word generators are marketed as AI, I don't think it's "shocking" that people think something "intelligent" could perform a trivial counting task. My 6 year old nephew could solve it very easily.
Weird that no one explicitly added embedded ascii/utf-8 directly to LLM training data for compression. Given that high dimensional spaces are built as vector spaces fully describable by basis vectors, I would assume somewhere these characters got added.
Perhaps it's an activation issue (i.e. broken after all) and it just needs an occasional change of basis.
One can define "reasoning" in the context of AI as the ability to perform logic operations in a loop with decisions to arrive at an answer. LLMs can't really do this.
Yeah, it would be like writing python to count how many vertical pen-strokes are in a string of byte-characters. To an eye, you can just scan and count the vertical lines. Python sees ASCII or UTF data, not lines, so that would be super difficult, analogous to a token-based system not seeing byte-chars.
> Just like a blind programmer could code a program to tell if an image is red or blue
Uh I'm sorry but I think it's not as easy as it seems. A pixel? Sure it's easy just compare whether the blue is bigger than red value. For image, I don't think it's as easy.
This is outside the bounds of the point OP was making, but FWIW an image is ultimately just a grid made up of pixels. You need only to loop your pixel classifier over the 2d array and tabulate.
LLMs don't actually "see" individual input characters, they see tokens, which are subwords. As far as they can "see", tokens are indivisible, since the LLM doesn't get access to individual characters at all. So it's impossible for them to count letters natively. Of course, they could still get the question right in an indirect way, e.g. if a human at some point wrote "strawberry has three r's" and this text ends up in the LLM's training set, it could just use that information to answer the question just like they would use "Paris is the capital of France" or whatever other facts they have access to. But they can't actually count the letters, so they are obviously going to fail often. This says nothing about their intelligence or reasoning capability, just like you wouldn't judge a blind person's intelligence for not being able to tell if an image is red or blue.
On the other hand, writing code to count appearances of a letter doesn't run into the same limitation. It can do it just fine. Just like a blind programmer could code a program to tell if an image is red or blue.