> A model that can reason should be able to understand the documentation and create novel examples. It cannot.
That's due to limitations imposed for "security". "Here's a new X, do Y with it" can result in holes bigger and more complex than anyone can currently handle "in time".
It's not about "abilities" with LLMs for now, but about functions that work within the range of edge cases, sometimes including them, some other times not.
You could still guide it to fulfill the task, though. It just cannot be allowed to do it on it's own but since just "forbidding" an LLM to do something is about as effective as doing that to a child with mischievous older brothers, the only ways to actually do it result in "bullshitted" code and "hallucinations".
That's due to limitations imposed for "security". "Here's a new X, do Y with it" can result in holes bigger and more complex than anyone can currently handle "in time".
It's not about "abilities" with LLMs for now, but about functions that work within the range of edge cases, sometimes including them, some other times not.
You could still guide it to fulfill the task, though. It just cannot be allowed to do it on it's own but since just "forbidding" an LLM to do something is about as effective as doing that to a child with mischievous older brothers, the only ways to actually do it result in "bullshitted" code and "hallucinations".
If I understood the problem correctly, that is.