> I'm just not clear why this process is allowed (or more importantly, why it can't be removed).

Because we don't know how to make CPUs do pixel perfect images every single time. (I wrote a little more above)