Something caused server memory spike and killed mysqld

I would appreciate assistance in troubleshooting an issue I recently experienced, and in helping me to take corrective action if something like this repeats itself.

Feb 23 23:38:07 panel kernel: [2111874.944884] [ 9266] 105 9266 333056 37839 131 0 0 mysqld Feb 23 23:38:07 panel kernel: [2111874.946724] Out of memory: Kill process 9266 (mysqld) score 24 or sacrifice child Feb 23 23:38:07 panel kernel: [2111874.947144] Killed process 9266 (mysqld) total-vm:1332224kB, anon-rss:151356kB, file-rss:0kB Feb 23 23:38:07 panel kernel: [2111875.144904] [ 642] 0 642 1084 40 6 0 0 mysqld_safe

I've gone through all the logs available in Webmin (Webmin --> system --> system logs) and I have not been able to find anything that will assist me in identifying the cause of the issue. Any advice on how to isolate the issue is appreciated.

A secondary request is for advice is in achieving the following when something like this takes place in the future.

1) Is there a setting in webmin that will check that certain specified services are running (like a cron job to check every hour) and if not the sepcifieid services are not running A) notify me B) attempt to start specified service. Does something like this exist and if not, can anyone assist me with such a script? I think the services that should be checked are Apache (Nginx in due course), Mysql, BIND, dovecot and Postfix, maybe SSH.

I received a notification from Webmin that MySQL service is not running but I only saw it in the morning; my server was without mysql all night. I use Virtualmin primarily for hosting Wordpress sites so it's a crucial service for my sites to run.

Status: 
Active

Comments

Howdy -- you may want to take a peek at Webmin -> Others -> System and Server Status.

You can use that to setup various forms of monitoring, and/or running a given command if a problem is detected.

Now, as far as the issue you saw -- it may not be possible to determine exactly what happened, but we can certainly do some looking around.

Let's start here -- what is the output of this command:

free -m

Also, if you SSH in as root and run the top command and hit M, it will show the processes using the most RAM.

Mostafa's picture
Submitted by Mostafa on Fri, 02/24/2017 - 14:51

I would recommend installing smem by apt-get or yum and then using smem -tpk