The browser loads images into the X11 server. Those images consume memory in the X11 server process. That bloats the X11 server up to be many gigabytes.

There is no reasonable way to account for the memory.

As a tab-hoarding Firefox user on machines running X11 with 2-4GiB of RAM who has spent time investigating Firefox and Chrome memory usage issues, I am having a really hard time believing your claim. Right now on the machine I am typing on there are 38 tabs open in Firefox 58.0, uptime 15 days, Xorg is taking up only 66088 KiB (66 MB) resident memory total.

The "reasonable way" to account for memory is to read the top(1) manual page to find out about the different memory usage statistics, then run top to find out which processes are using how much memory.

I use top. I run top, and it shows that the X11 server is gigabytes in size. Now what? Nothing tells me which X11 client windows are at fault. I can't tell if the memory is even legitimately in use; it could be a leak.

Your 66 MB is really low. Maybe your video driver is better than mine.