Think about the difference between what an LLM is doing and a human is doing in response to being told they are wrong.
If you tell a human, “You are wrong on this, and here is why.” They may reject it, but their mental model is updating with the new information that you, the speaker, think X for YZQ reasons. Their response is based on a judgment of how trustworthy you are and the credibility of the evidence.
For an LLM, the response is not based on these logical connections, but simply the additional prompt context of YZQ tokens being close to each other.
This is not “logic” in any traditional sense or in the sense of how a human incorporates and responds to this new information.
The LLM’s method of responding is also inherent to the architecture of the model. It’s predicting tokens based on input. It’s not reasoning.
Critically, this flaw is inherent in all LLM output. Giving an LLM’s output the power to affect real world activities means trusting that the decision can be made by sophisticated word association rather than more complex reasoning.
There may be lots of decisions where word association is all you need, but I doubt that is the case for all decisions humans make.
If you tell a human, “You are wrong on this, and here is why.” They may reject it, but their mental model is updating with the new information that you, the speaker, think X for YZQ reasons. Their response is based on a judgment of how trustworthy you are and the credibility of the evidence.
For an LLM, the response is not based on these logical connections, but simply the additional prompt context of YZQ tokens being close to each other.
This is not “logic” in any traditional sense or in the sense of how a human incorporates and responds to this new information.
The LLM’s method of responding is also inherent to the architecture of the model. It’s predicting tokens based on input. It’s not reasoning.
Critically, this flaw is inherent in all LLM output. Giving an LLM’s output the power to affect real world activities means trusting that the decision can be made by sophisticated word association rather than more complex reasoning.
There may be lots of decisions where word association is all you need, but I doubt that is the case for all decisions humans make.