I don’t think your problem is RGB instead of BGR. That’s just the compositor’s work area and your monitor never sees it (it includes an alpha channel). Have you tried KDE Plasma? It sounds like KWin uses 10-bit planes by default when available. Maybe Ubuntu’s compositor (Mutter?) doesn’t support 30 bit color or must be configured? Or maybe you need the nvidia driver >= 580.94.11 for VK_EXT_hdr_metadata (https://www.phoronix.com/news/NVIDIA-580.94.11-Linux-Driver)
I'm actually surprised that YouTube HDR works on your side - perhaps it's tied to the ABGR2101010 output mode being available.