Before OOM killer activates, the system will swap out all other applications including desktop shell. If you are unlucky to use a spinning hard drive, your system will freeze for a long time.
This is a problem with popular Linux distributions, they do not have protection against application swapping out important system applications.
Linux has disk quotas and CPU scheduler but it doesn't have fair memory and swap management (or this is not configured out of the box). For example, desktop shell doesn't have protection against being swapped out.
Ah yes, that is indeed very annoying. That's why I usually don't configure a swap partition on Linux, although that probably ends up wasting a lot of memory on write-once-read-never pages that could otherwise be used for the page cache.
A per-process real memory maximum that automatically invokes the OOM killer unless somehow opted out would probably be useful.
This is a problem with popular Linux distributions, they do not have protection against application swapping out important system applications.
Linux has disk quotas and CPU scheduler but it doesn't have fair memory and swap management (or this is not configured out of the box). For example, desktop shell doesn't have protection against being swapped out.