> In practice, however, LLMs don't seem to have any problem interpreting natural language instructions
I can think of a couple of reasons this may be the case.
1. There is a subset of English that you use unknowingly that has a socially accepted formal definition and so can be used as a substitute for programming language. LLMs have learned this definition. Straying from this subset or expecting a different formal definition will result in errors.
2. The level of detail in your English description is such that ambiguity genuinely does not arise. Unlikely, you would not consider that "natural language".
3. English is not ambiguous when describing program features, and formal definitions can be skipped. Unlikely, because the entire product owner role is built on the frequently exclaimed "that's not what I meant!".
I think its #1, and I think that makes the most sense: through massive statistical data LLMs have learned which natural language instructions cause which modifications in codebases, for a giant amount of generic problems that it has training data on.
The moment you do something new though, all bets are off.
I can think of a couple of reasons this may be the case.
1. There is a subset of English that you use unknowingly that has a socially accepted formal definition and so can be used as a substitute for programming language. LLMs have learned this definition. Straying from this subset or expecting a different formal definition will result in errors.
2. The level of detail in your English description is such that ambiguity genuinely does not arise. Unlikely, you would not consider that "natural language".
3. English is not ambiguous when describing program features, and formal definitions can be skipped. Unlikely, because the entire product owner role is built on the frequently exclaimed "that's not what I meant!".
I think its #1, and I think that makes the most sense: through massive statistical data LLMs have learned which natural language instructions cause which modifications in codebases, for a giant amount of generic problems that it has training data on.
The moment you do something new though, all bets are off.