Yesterday around 1am, our server ground to a crawl. This doesn't happen often, but I'm trying to get to the bottom of it.
There is no unusual traffic volume, no unusual processes running, just all of the sudden the server started killing fcgid processes.
[Thu Aug 02 01:17:32 2012] [warn] mod_fcgid: process 26460 graceful kill fail, sending SIGKILL
... for as many fcgid processes as we have...
I suspect there was some swap activity, although I'm not sure if it was due to killed processes being swapped in to die, or if it was because some process ramped up memory usage faster than my process watching scripts can see them.
The oom-killer wasn't triggered (at least it's not logged), so I think this was Apache for some reason restarting the processes.
Any thoughts about where to troubleshoot next? If I were to set fcgi parameters for PHP, where in Virtualmin should I set them?