I assume your concern with GPU passthrough is that each VM needs a whole GPU?
You can use GPU-PV to split your GPU between VM instances.
Then the main bottleneck becomes how thin you split out your VRAM.
Wouldn't virtualbox or vmware's paravirtual GPUs be a better fit for this use case? Unfortunately the offerings with qemu/libvirt still lag vmwares by a lot.
I know those offer virtual GPUs, but I am unfamiliar with any paravirtual GPU offerings from VMWare or VirtualBox. The virtual GPUs are much more limited in performance and graphics API support.
More info here:
https://web.archive.org/web/20231107182321/https://mu0.cc/20...
https://youtu.be/XLLcc29EZ_8?t=570
https://github.com/jamesstringer90/Easy-GPU-PV