And when gaming latency is a big issue, so buffering a few frames to have margin for a complex one would be unacceptable.

In this case the console would just stream full frames.

It could also send deltas and achieve much higher variable frame rates.

The need to stream pixels at a constant rate feels like scanning a CRT and makes generating video signals a lot more complicated than it needs to be.

> what happens if you exceed the bandwidth limit for several frames in a row?

There's no way deltas would exceed deltas that are just full frames. If you can't send a full frame over the wire, then you are trying to push too many pixels and need to downsample on any of the 3 dimensions you have to play with.