Every few months, I've had remote OpenBSD servers die mysteriously, without any visible console messages, and still responding to pings, although higher level networking (http, ssh) are unresponsive. The crontab I setup to dump output from top (or systat) also stops. But before the kernel failure load was low and swap was unused.

After rebooting and checking dmesg, I see before the most recent reboot:

uvm_mapent_alloc: out of static map entries

Doing some research, it seems like it could be a problem with kmem_map fragmentation, possibly from long-running processes (like apache, which is being used more heavily now). The problem appears to have been fixed in recent releases of OpenBSD which dynamically increase kmem, but upgrading the server right now isn't feasible.

So the solution is to compile a custom kernel with an increased MAX_KMAPENT added to GENERIC config:

option NKMEMPAGES_MAX=81920
option NKMEMPAGES=81920
option MAX_KMAPENT=8192

To check current nkmempages:

sysctl vm.nkmempages
vm.nkmempages=16384

From man options:

option NKMEMPAGES=value
     option NKMEMPAGES_MAX=value
     option NKMEMPAGES_MIN=value

MAX_KMAPENT isn't listed as an option in the man page but I see it's defined:

uvm/uvm_map.h:#define  MAX_KMAPENT     1024 

Likewise:

arch/i386/include/param.h:#define      NKMEMPAGES_MIN_DEFAULT  ((8 * 1024 * 1024) >> PAGE_SHIFT)
arch/i386/include/param.h:#define      NKMEMPAGES_MAX_DEFAULT  ((64 * 1024 * 1024) >> PAGE_SHIFT)

So compiled a new kernel, installed, and rebooted. Let's see if I revisit this post in a few more months.

If the machine crashes again hopefully watchdog with kern.watchdog.period sysctl'd to 32 will reboot it.