I checked out Eric's work just - he's doing an amazing job! Much more thought put into the subject (security for GLSL being a huge topic on its own). The lack of MMU in the VC4 architecture is a huge pain point that probably sucks up most engineering cycles when it comes to arbitrary application environments using GPU resources like X11 or Wayland - when memory runs out, what to do? You can throw engineering resources at it, but engineering talent really needs access to VMCS on the VideoCore side todo any worth while work.
When we ported Android to the VC4 architecture the first time (~2010), the low memory killer in Linux was subverted to understand to kill Android applications based on their use of the VideoCore GPU memory and it worked pretty well, yet it would still close the primary running app once in a while. Run monkey over Android and all hell broke loose - really tough situations to defensively code fore. For example, for CX reasons, you had to ignore some GLES errors in the primary client due to low memory, then the system had to kill another application that was using the memory, then it would kill the EGL context for the primary application so it would refresh the entire app lifecycle using an almost suspect code path inside Android. Good times! Imagine Wayload has very similar challenges for normal desktop use.
The open source driver doesn't use VMCS, it instead puts aside a fixed block of memory (typically 256 MiB, which I think is also a limit due to some hardware bugs) for use through the Linux Contiguous Memory Allocator (CMA) that it then draws from.
VMCS only comes into the picture if you use video decode, but I think Dave Stevenson from the foundation hacked the firmware side to support importing Linux allocated memory blocks into VMCS so that you can do zero-copy decode and import into EGL (or more likely HVS, the EGL support for the formats is pretty limited).
(I really liked the design of the HVS - having pretty much scripted planes is a fresh approach over similar hardware blocks that have a fixed number of planes each with own idiosyncracies and limitations)
The HVS was a cool design - the only real issue if I recall was its limited line buffers meaning it was hard to determine what could be composited in real time, so we ended up always rendering to a triple buffer. It also did some amazing scaling in realtime, but this came with some crazy artifacts that were super painful to triage. The occasional "flash of bad pixels" in a live composition screen was really painful. I just remember I wrote the DPI and CPI peripherals that we brought up the HVS with - on a 4MHz FPGA complex, we had YUV camera input scaling up to a WVGA DPI screen running at 60fps on a distributed FPGA platform through the HVS. Fun times.
