A "server engine" (usually called a kernel actually) is like a userspace operating system kernel that is purpose-built for a class of server workloads and attempts to optimally manage system resources for that workload. The server application is built on top of that kernel. When people refer to e.g. "database kernels", this is what they are referring to. It reimplements the operating system resource management services through the OS APIs.
Writing an excellent server kernel requires a high level of technical ability and quite a bit of low-level code since you have to reimplement most of the operating system resource management services most developers take for granted. Few pieces of open source server software are built on userspace kernels and the ones that do like PostgreSQL are (currently) only partial kernels that still rely on the OS to do significant things a full kernel would reimplement. A properly designed database kernel, for example, is at least 100k LoC of low-level C/C++ and that is before you actually write the server application that sits on top of it. I've designed and written kernels for both database engines and network engines, it is not trivial.
So why would you want to go through all this effort instead of writing to the standard POSIX APIs directly? Because performance and scalability. For example, a well-designed disk buffer and scheduler for a database kernel can easily triple the I/O throughput possible with a highly tuned POSIX implementation. A lot of locking and blocking, both explicit and implicit in the OS, become unnecessary. Certain kinds of distributed system problems become easier to solve because you can adaptively schedule data flows at a fine-grained level. One of the reasons Oracle and DB2 scale so well and get such high throughput is that they are built on highly optimized userspace kernels.
Virtualization actually degrades the performance of high-performance server systems in part because the hypervisor acts as a primitive operating system underneath the operating system the server kernel can see. This does not impact non-kernel based servers as much.
Writing an excellent server kernel requires a high level of technical ability and quite a bit of low-level code since you have to reimplement most of the operating system resource management services most developers take for granted. Few pieces of open source server software are built on userspace kernels and the ones that do like PostgreSQL are (currently) only partial kernels that still rely on the OS to do significant things a full kernel would reimplement. A properly designed database kernel, for example, is at least 100k LoC of low-level C/C++ and that is before you actually write the server application that sits on top of it. I've designed and written kernels for both database engines and network engines, it is not trivial.
So why would you want to go through all this effort instead of writing to the standard POSIX APIs directly? Because performance and scalability. For example, a well-designed disk buffer and scheduler for a database kernel can easily triple the I/O throughput possible with a highly tuned POSIX implementation. A lot of locking and blocking, both explicit and implicit in the OS, become unnecessary. Certain kinds of distributed system problems become easier to solve because you can adaptively schedule data flows at a fine-grained level. One of the reasons Oracle and DB2 scale so well and get such high throughput is that they are built on highly optimized userspace kernels.
Virtualization actually degrades the performance of high-performance server systems in part because the hypervisor acts as a primitive operating system underneath the operating system the server kernel can see. This does not impact non-kernel based servers as much.