There are papers including one by Yann LeCunn which shows that the overwhelming majority of data found within high dimensional spaces is within decision boundries created by AI models. This implies that the overwhelming majority of things we think as humans are "novel" or "extrapolative" are in fact not novel and are nothing more than fancy interpolation.
Basically, we humans cannot tell when something is "new to us" or "new to the universe" and it honestly hardly matters in practical reality.
Finally, LLMs can totally "extrapolate" with techniques which encourage it to avoid its traditional training loss functions. A high temperature would do this, but it'll mostly just make your model crazy.
