My first thought is the kernel should pre-allocate some space for running a recovery/cleanup/analysis process when malloc fails. Is anything like this done already? Can it defer to the user to decide what to do when that happens?
Well on the first case there is, the OOM killer is preallocated, the memory deallocation algorithms are preallocated, etc.
On the second... not so much. It comes back to the same problem. How do you notify the user without using the memory subsystem again (allocating text, or graphics buffers). How do you differentiate a memory call used to notify the user from a normal one? It's the same problem the OOM killer was having.
