>3) LLMs do not have a mechanism for sampling from given probability distributions. E.g. if you ask LLM to sample a random number from 1 to 10, it will likely give you 3 or 7, as those are overrepresented in the training data.
I am not sure that is true. Yes it will likely give a 3 or 7 but that is because it is trying to represent that distribution from the training data. It's not trying for a random digit there, it's trying for what the data set does.
It would certainly be possible to give an AI the notion of a random digit, and rather than training on fixed output examples give it additional training to make it to produce an embedding that was exactly equidistant from the tokens 0..9 when it wanted a random digit.
You could then fine tune it to use that ability to generate sequences of random digits to provide samples in reasoning steps.
That requires tool use or some similar specific action at inference time.
The technique I suggested would, I think, work on existing model inference methods. The ability already exists in the architecture. It's just a training adjustment to produce the parameters required to do so.
I am not sure that is true. Yes it will likely give a 3 or 7 but that is because it is trying to represent that distribution from the training data. It's not trying for a random digit there, it's trying for what the data set does.
It would certainly be possible to give an AI the notion of a random digit, and rather than training on fixed output examples give it additional training to make it to produce an embedding that was exactly equidistant from the tokens 0..9 when it wanted a random digit.
You could then fine tune it to use that ability to generate sequences of random digits to provide samples in reasoning steps.