A bit too often!

Does it look like this, the server is inaccessible precisely at these times and it is often at least a few times per day.

Anyone have an idea what causes this?

See the attached picture!

Status: 
Active

Comments

Howdy -- it looks like your server is seeing a pretty high load in that screenshot. What you may want to do is review the running processes to see what's causing such a high load.

One way to get an idea of what's occurring is to run "top".

If you're not sure what the issue is, run the following commands, and let us know what the output is:

uptime
free -m
ps auxwf
mailq | tail -1
netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head -20
LarsReimers's picture
Submitted by LarsReimers on Tue, 11/18/2014 - 13:29

Right now, when all is normal it shows this:

[root@h83-209-52-2 ~]# uptime
20:31:37 up 1 day, 9:07, 1 user, load average: 0.58, 0.44, 1.25
[root@h83-209-52-2 ~]# free -m
total used free shared buffers cached
Mem: 3831 792 3038 25 72 202
-/+ buffers/cache: 518 3313
Swap: 15999 245 15754


[root@h83-209-52-2 ~]# ps auxwf
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2 0.0 0.0 0 0 ? S Nov17 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Nov17 0:01 \_ [migration/ 0]
root 4 0.0 0.0 0 0 ? S Nov17 0:05 \_ [ksoftirqd/ 0]
root 5 0.0 0.0 0 0 ? S Nov17 0:00 \_ [stopper/0]
root 6 0.0 0.0 0 0 ? S Nov17 0:00 \_ [watchdog/0 ]
root 7 0.0 0.0 0 0 ? S Nov17 0:03 \_ [migration/ 1]
root 8 0.0 0.0 0 0 ? S Nov17 0:00 \_ [stopper/1]
root 9 0.0 0.0 0 0 ? S Nov17 0:05 \_ [ksoftirqd/ 1]
root 10 0.0 0.0 0 0 ? S Nov17 0:00 \_ [watchdog/1 ]
root 11 0.0 0.0 0 0 ? S Nov17 0:06 \_ [events/0]
root 12 0.0 0.0 0 0 ? S Nov17 0:10 \_ [events/1]
root 13 0.0 0.0 0 0 ? S Nov17 0:00 \_ [cgroup]
root 14 0.0 0.0 0 0 ? S Nov17 0:00 \_ [khelper]
root 15 0.0 0.0 0 0 ? S Nov17 0:00 \_ [netns]
root 16 0.0 0.0 0 0 ? S Nov17 0:00 \_ [async/mgr]
root 17 0.0 0.0 0 0 ? S Nov17 0:00 \_ [pm]
root 18 0.0 0.0 0 0 ? S Nov17 0:00 \_ [sync_super s]
root 19 0.0 0.0 0 0 ? S Nov17 0:00 \_ [bdi-defaul t]
root 20 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kintegrity d/0]
root 21 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kintegrity d/1]
root 22 0.0 0.0 0 0 ? S Nov17 0:37 \_ [kblockd/0]
root 23 0.0 0.0 0 0 ? S Nov17 0:37 \_ [kblockd/1]
root 24 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kacpid]
root 25 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kacpi_noti fy]
root 26 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kacpi_hotp lug]
root 27 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ata_aux]
root 28 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ata_sff/0]
root 29 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ata_sff/1]
root 30 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ksuspend_u sbd]
root 31 0.0 0.0 0 0 ? S Nov17 0:00 \_ [khubd]
root 32 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kseriod]
root 33 0.0 0.0 0 0 ? S Nov17 0:00 \_ [md/0]
root 34 0.0 0.0 0 0 ? S Nov17 0:00 \_ [md/1]
root 35 0.0 0.0 0 0 ? S Nov17 0:00 \_ [md_misc/0]
root 36 0.0 0.0 0 0 ? S Nov17 0:00 \_ [md_misc/1]
root 37 0.0 0.0 0 0 ? S Nov17 0:00 \_ [linkwatch]
root 38 0.0 0.0 0 0 ? S Nov17 0:00 \_ [khungtaskd ]
root 39 0.4 0.0 0 0 ? S Nov17 9:38 \_ [kswapd0]
root 40 0.0 0.0 0 0 ? SN Nov17 0:00 \_ [ksmd]
root 41 0.0 0.0 0 0 ? SN Nov17 0:04 \_ [khugepaged ]
root 42 0.0 0.0 0 0 ? S Nov17 0:00 \_ [aio/0]
root 43 0.0 0.0 0 0 ? S Nov17 0:00 \_ [aio/1]
root 44 0.0 0.0 0 0 ? S Nov17 0:00 \_ [crypto/0]
root 45 0.0 0.0 0 0 ? S Nov17 0:00 \_ [crypto/1]
root 53 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kthrotld/0 ]
root 54 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kthrotld/1 ]
root 57 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kpsmoused]
root 58 0.0 0.0 0 0 ? S Nov17 0:00 \_ [usbhid_res umer]
root 59 0.0 0.0 0 0 ? S Nov17 0:00 \_ [deferwq]
root 90 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kdmremove]
root 91 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kstriped]
root 121 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ttm_swap]
root 182 0.0 0.0 0 0 ? S Nov17 0:00 \_ [scsi_eh_0]
root 183 0.0 0.0 0 0 ? S Nov17 0:00 \_ [scsi_eh_1]
root 184 0.0 0.0 0 0 ? S Nov17 0:00 \_ [scsi_eh_2]
root 185 0.0 0.0 0 0 ? S Nov17 0:00 \_ [scsi_eh_3]
root 186 0.0 0.0 0 0 ? S Nov17 0:00 \_ [scsi_eh_4]
root 187 0.0 0.0 0 0 ? S Nov17 0:00 \_ [scsi_eh_5]
root 317 0.0 0.0 0 0 ? S Nov17 0:04 \_ [jbd2/sdb2- 8]
root 318 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ext4-dio-u nwrit]
root 435 0.0 0.0 0 0 ? S Nov17 0:02 \_ [edac-polle r]
root 486 0.0 0.0 0 0 ? S Nov17 0:05 \_ [flush-8:0]
root 489 0.0 0.0 0 0 ? S Nov17 0:01 \_ [flush-8:16 ]
root 491 0.0 0.0 0 0 ? S Nov17 0:03 \_ [flush-8:32 ]
root 750 0.0 0.0 0 0 ? S Nov17 0:00 \_ [jbd2/sda1- 8]
root 751 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ext4-dio-u nwrit]
root 752 0.0 0.0 0 0 ? D Nov17 0:02 \_ [jbd2/sdc2- 8]
root 753 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ext4-dio-u nwrit]
root 754 0.0 0.0 0 0 ? S Nov17 0:09 \_ [jbd2/sda5- 8]
root 755 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ext4-dio-u nwrit]
root 923 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kauditd]
root 1305 0.0 0.0 0 0 ? S Nov17 0:53 \_ [kondemand/ 0]
root 1306 0.0 0.0 0 0 ? S Nov17 0:36 \_ [kondemand/ 1]
root 1 0.0 0.0 19364 308 ? Ss Nov17 0:00 /sbin/init
root 404 0.0 0.0 10784 392 ? S
[root@h83-209-52-2 ~]# mailq | tail -1
Mail queue is empty
[root@h83-209-52-2 ~]# netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head -20
1 127.0.0.1
5 91.225.248.132
2 83.209.55.222
68
1 servers)
1 Address
[root@h83-209-52-2 ~]#

Okay, that all looks pretty normal. Next time you run into that issue, try running all the above commands again, and we can compare it to the output you have there when things are working properly.

LarsReimers's picture
Submitted by LarsReimers on Tue, 11/18/2014 - 13:52

So true, I want to do this!

LarsReimers's picture
Submitted by LarsReimers on Wed, 11/19/2014 - 11:27

This is what it looks like when it maxes

[root@h83-209-52-2 ~]# uptime
18:21:32 up 2 days, 6:57, 1 user, load average: 9.57, 8.65, 8.33
[root@h83-209-52-2 ~]# free -m
total used free shared buffers cached
Mem: 3831 3707 123 31 12 115
-/+ buffers/cache: 3580 251
Swap: 15999 12198 3801
[root@h83-209-52-2 ~]# ps auxwf
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2 0.0 0.0 0 0 ? S Nov17 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Nov17 0:02 \_ [migration/0]
root 4 0.0 0.0 0 0 ? S Nov17 0:09 \_ [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S Nov17 0:00 \_ [stopper/0]
root 6 0.0 0.0 0 0 ? S Nov17 0:00 \_ [watchdog/0]
root 7 0.0 0.0 0 0 ? S Nov17 0:04 \_ [migration/1]
root 8 0.0 0.0 0 0 ? S Nov17 0:00 \_ [stopper/1]
root 9 0.0 0.0 0 0 ? S Nov17 0:10 \_ [ksoftirqd/1]
root 10 0.0 0.0 0 0 ? S Nov17 0:00 \_ [watchdog/1]
root 11 0.0 0.0 0 0 ? S Nov17 0:12 \_ [events/0]
root 12 0.0 0.0 0 0 ? S Nov17 0:16 \_ [events/1]
root 13 0.0 0.0 0 0 ? S Nov17 0:00 \_ [cgroup]
root 14 0.0 0.0 0 0 ? S Nov17 0:00 \_ [khelper]
root 15 0.0 0.0 0 0 ? S Nov17 0:00 \_ [netns]
root 16 0.0 0.0 0 0 ? S Nov17 0:00 \_ [async/mgr]
root 17 0.0 0.0 0 0 ? S Nov17 0:00 \_ [pm]
root 18 0.0 0.0 0 0 ? S Nov17 0:00 \_ [sync_supers]
root 19 0.0 0.0 0 0 ? S Nov17 0:00 \_ [bdi-default]
root 20 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kintegrityd/0]
root 21 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kintegrityd/1]
root 22 0.0 0.0 0 0 ? S Nov17 1:16 \_ [kblockd/0]
root 23 0.0 0.0 0 0 ? S Nov17 1:17 \_ [kblockd/1]
root 24 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kacpid]
root 25 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kacpi_notify]
root 26 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kacpi_hotplug]
root 27 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ata_aux]
root 28 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ata_sff/0]
root 29 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ata_sff/1]
root 30 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ksuspend_usbd]
root 31 0.0 0.0 0 0 ? S Nov17 0:00 \_ [khubd]
root 32 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kseriod]
root 33 0.0 0.0 0 0 ? S Nov17 0:00 \_ [md/0]
root 34 0.0 0.0 0 0 ? S Nov17 0:00 \_ [md/1]
root 35 0.0 0.0 0 0 ? S Nov17 0:00 \_ [md_misc/0]
root 36 0.0 0.0 0 0 ? S Nov17 0:00 \_ [md_misc/1]
root 37 0.0 0.0 0 0 ? S Nov17 0:00 \_ [linkwatch]
root 38 0.0 0.0 0 0 ? S Nov17 0:00 \_ [khungtaskd]
root 39 0.6 0.0 0 0 ? D Nov17 20:22 \_ [kswapd0]
root 40 0.0 0.0 0 0 ? SN Nov17 0:00 \_ [ksmd]
root 41 0.0 0.0 0 0 ? SN Nov17 0:07 \_ [khugepaged]
root 42 0.0 0.0 0 0 ? S Nov17 0:00 \_ [aio/0]
root 43 0.0 0.0 0 0 ? S Nov17 0:00 \_ [aio/1]
root 44 0.0 0.0 0 0 ? S Nov17 0:00 \_ [crypto/0]
root 45 0.0 0.0 0 0 ? S Nov17 0:00 \_ [crypto/1]
root 53 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kthrotld/0]
root 54 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kthrotld/1]
root 57 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kpsmoused]
root 58 0.0 0.0 0 0 ? S Nov17 0:00 \_ [usbhid_resumer]
root 59 0.0 0.0 0 0 ? S Nov17 0:00 \_ [deferwq]
root 90 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kdmremove]
root 91 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kstriped]
root 121 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ttm_swap]
root 182 0.0 0.0 0 0 ? S Nov17 0:00 \_ [scsi_eh_0]
root 183 0.0 0.0 0 0 ? S Nov17 0:00 \_ [scsi_eh_1]
root 184 0.0 0.0 0 0 ? S Nov17 0:00 \_ [scsi_eh_2]
root 185 0.0 0.0 0 0 ? S Nov17 0:00 \_ [scsi_eh_3]
root 186 0.0 0.0 0 0 ? S Nov17 0:00 \_ [scsi_eh_4]
root 187 0.0 0.0 0 0 ? S Nov17 0:00 \_ [scsi_eh_5]
root 317 0.0 0.0 0 0 ? S Nov17 0:06 \_ [jbd2/sdb2-8]
root 318 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ext4-dio-unwrit]
root 435 0.0 0.0 0 0 ? S Nov17 0:03 \_ [edac-poller]
root 486 0.0 0.0 0 0 ? S Nov17 0:13 \_ [flush-8:0]
root 489 0.0 0.0 0 0 ? S Nov17 0:02 \_ [flush-8:16]
root 491 0.0 0.0 0 0 ? S Nov17 0:05 \_ [flush-8:32]
root 750 0.0 0.0 0 0 ? S Nov17 0:00 \_ [jbd2/sda1-8]
root 751 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ext4-dio-unwrit]
root 752 0.0 0.0 0 0 ? S Nov17 0:03 \_ [jbd2/sdc2-8]
root 753 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ext4-dio-unwrit]
root 754 0.0 0.0 0 0 ? S Nov17 0:16 \_ [jbd2/sda5-8]
root 755 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ext4-dio-unwrit]
root 923 0.0 0.0 0 0 ? S Nov17 0:01 \_ [kauditd]
root 1305 0.0 0.0 0 0 ? S Nov17 1:21 \_ [kondemand/0]
root 1306 0.0 0.0 0 0 ? S Nov17 0:53 \_ [kondemand/1]
root 1 0.0 0.0 19364 84 ? Ss Nov17 0:01 /sbin/init
root 404 0.0 0.0 10784 324 ? Sroot@pts/0
root 4487 0.0 0.0 108340 1748 pts/0 Ss 18:20 0:00 \_ -bash
root 4544 0.0 0.0 110344 1132 pts/0 R+ 18:21 0:00 \_ ps auxwf
root 1704 0.0 0.0 108204 8 ? S Nov17 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var
mysql 1836 1.3 0.9 1183288 35864 ? Sl Nov17 45:16 \_ /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --u
root 1863 0.0 0.0 19268 336 ? Ss Nov17 0:02 /usr/sbin/dovecot
dovecot 1864 0.0 0.0 12988 360 ? S Nov17 0:00 \_ dovecot/anvil
root 1866 0.0 0.0 13112 476 ? S Nov17 0:00 \_ dovecot/log
root 1878 0.0 0.0 68976 8 ? Ss Nov17 0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -r
root 1879 0.0 0.0 68976 1020 ? S Nov17 0:00 \_ /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -r
root 1880 0.0 0.0 68976 28 ? S Nov17 0:00 \_ /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -r
root 1881 0.0 0.0 68976 8 ? S Nov17 0:00 \_ /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -r
root 1882 0.0 0.0 68976 1020 ? S Nov17 0:00 \_ /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -r
grmilter 1892 0.0 0.0 261484 572 ? Ssl Nov17 0:05 /usr/sbin/milter-greylist -u grmilter -P /var/run/milter-greylist.p
root 1999 0.0 0.0 81328 256 ? Ss Nov17 0:01 /usr/libexec/postfix/master
postfix 2011 0.0 0.0 81596 468 ? S Nov17 0:00 \_ qmgr -l -t fifo -u
postfix 27264 0.0 0.0 81408 444 ? S 16:58 0:00 \_ pickup -l -t fifo -u
postfix 3929 0.0 0.0 81700 3364 ? S 18:15 0:00 \_ local -t unix
root 3932 0.0 0.0 8828 808 ? Ss 18:15 0:00 \_ /usr/bin/procmail-wrapper -o -a h83-209-52-2.dynamic.se.all
root 3933 0.0 0.0 8828 160 ? S 18:15 0:00 \_ /usr/bin/procmail-wrapper -o -a h83-209-52-2.dynamic.se
root 3934 0.0 0.0 30376 1956 ? S 18:15 0:00 \_ /usr/bin/perl /usr/libexec/webmin/virtual-server/lo
root 2036 0.0 0.0 114676 8 ? Ss Nov17 0:00 /usr/sbin/abrtd
520 2056 0.0 0.0 411108 252 ? Sl Nov17 0:42 /usr/local/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf
root 2072 0.0 0.0 117332 356 ? Ss Nov17 0:04 crond
root 2086 0.0 0.0 21576 4 ? Ss Nov17 0:00 /usr/sbin/atd
root 2101 0.0 0.0 64704 144 ? Ss Nov17 0:00 /usr/sbin/certmonger -S -p /var/run/certmonger.pid
root 2167 0.0 0.0 189368 740 ? Ss Nov17 0:00 /usr/libexec/webmin/virtual-server/lookup-domain-daemon.pl
root 2175 0.0 0.0 87580 920 ? Ss Nov17 0:02 /usr/bin/perl /usr/libexec/usermin/miniserv.pl /etc/usermin/miniser
root 2183 0.0 0.0 140204 2864 ? Ss Nov17 0:12 /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/miniserv.
root 4301 0.1 0.6 145904 25324 ? S 18:19 0:00 \_ /usr/libexec/webmin/webmincron/webmincron.pl
root 4306 0.4 1.0 160104 42144 ? S 18:19 0:00 | \_ /usr/libexec/webmin/virtual-server/collectinfo.pl
root 4333 0.0 0.0 11336 1180 ? S 18:19 0:00 | \_ sh -c rpm -q -a --queryformat "%{NAME}\n%{VERSION}-%{RE
root 4334 1.2 0.2 80084 11500 ? D 18:19 0:01 | \_ rpm -q -a --queryformat %{NAME}\n%{VERSION}-%{RELEA
root 4436 0.0 0.1 140316 5516 ? S 18:20 0:00 \_ /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/minis
root 4438 0.0 0.1 140316 5512 ? S 18:20 0:00 \_ /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/minis
root 4439 0.0 0.1 140316 5520 ? S 18:20 0:00 \_ /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/minis
root 4440 0.0 0.1 140316 5520 ? S 18:20 0:00 \_ /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/minis
root 4488 0.0 0.1 140316 5516 ? S 18:20 0:00 \_ /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/minis
root 2192 0.0 0.0 4064 8 tty1 Ss+ Nov17 0:00 /sbin/mingetty /dev/tty1
root 2194 0.0 0.0 4064 8 tty2 Ss+ Nov17 0:00 /sbin/mingetty /dev/tty2
root 2196 0.0 0.0 4064 8 tty3 Ss+ Nov17 0:00 /sbin/mingetty /dev/tty3
root 2198 0.0 0.0 4064 8 tty4 Ss+ Nov17 0:00 /sbin/mingetty /dev/tty4
root 2200 0.0 0.0 4064 8 tty5 Ss+ Nov17 0:00 /sbin/mingetty /dev/tty5
root 2202 0.0 0.0 4064 8 tty6 Ss+ Nov17 0:00 /sbin/mingetty /dev/tty6
nobody 28916 0.0 0.0 55736 388 ? Ss Nov17 0:02 proftpd: (accepting connections)
root 30504 0.0 0.0 440596 320 ? Ss Nov17 0:05 /usr/sbin/httpd
apache 30506 0.0 0.0 251964 248 ? S Nov17 0:02 \_ /usr/sbin/httpd
504 30696 0.0 0.4 356528 17544 ? D Nov17 1:25 | \_ /usr/bin/php-cgi
502 30746 0.0 0.1 267076 5148 ? S Nov17 0:49 | \_ /usr/bin/php-cgi
507 31863 0.0 0.2 357796 8208 ? S Nov17 0:43 | \_ /usr/bin/php-cgi
504 4757 0.0 0.4 350500 15948 ? S Nov18 0:44 | \_ /usr/bin/php-cgi
514 1795 0.3 0.3 205020 12064 ? S Nov18 5:03 | \_ /opt/rh/php54/root/usr/bin/php-cgi
502 9559 0.0 0.2 269740 8708 ? S 00:55 0:06 | \_ /usr/bin/php-cgi
502 22566 0.0 0.2 258564 8092 ? S 16:25 0:00 | \_ /usr/bin/php-cgi
514 23388 0.2 0.4 200184 17380 ? S 16:32 0:18 | \_ /opt/rh/php54/root/usr/bin/php-cgi
507 29596 0.0 0.2 357808 11176 ? D 17:16 0:01 | \_ /usr/bin/php-cgi
514 32023 0.1 0.4 197280 18552 ? S 17:37 0:04 | \_ /opt/rh/php54/root/usr/bin/php-cgi
apache 1585 0.0 0.0 455028 2936 ? S Nov18 0:11 \_ /usr/sbin/httpd
apache 1679 0.0 0.0 441248 920 ? S Nov18 0:11 \_ /usr/sbin/httpd
520 21554 3.4 24.3 5271644 954924 ? D 16:18 4:16 | \_ /usr/bin/php-cgi
apache 1806 0.0 0.0 441240 944 ? S Nov18 0:11 \_ /usr/sbin/httpd
520 23211 3.8 29.4 5271644 1153624 ? D 16:31 4:12 | \_ /usr/bin/php-cgi
apache 1809 0.0 0.0 442740 2888 ? S Nov18 0:12 \_ /usr/sbin/httpd
apache 6872 0.0 0.0 442888 2928 ? S Nov18 0:11 \_ /usr/sbin/httpd
apache 6885 0.0 0.0 445620 948 ? S Nov18 0:11 \_ /usr/sbin/httpd
520 22403 3.6 18.6 5271644 731976 ? D 16:24 4:15 | \_ /usr/bin/php-cgi
apache 6613 0.0 0.0 442740 2836 ? S 00:35 0:07 \_ /usr/sbin/httpd
apache 6731 0.0 0.0 442948 2936 ? S 00:36 0:06 \_ /usr/sbin/httpd
apache 7417 0.0 0.0 442408 2876 ? S 00:40 0:06 \_ /usr/sbin/httpd
apache 8132 0.0 0.0 441900 2900 ? S 00:46 0:06 \_ /usr/sbin/httpd
apache 9923 0.0 0.0 442332 2884 ? S 00:58 0:06 \_ /usr/sbin/httpd
apache 21623 0.0 0.0 442672 2900 ? S 02:11 0:06 \_ /usr/sbin/httpd
apache 23331 0.0 0.0 440860 2860 ? S 16:32 0:00 \_ /usr/sbin/httpd
apache 23342 0.0 0.0 440864 3056 ? S 16:32 0:00 \_ /usr/sbin/httpd
apache 23386 0.0 0.0 440864 3108 ? S 16:32 0:00 \_ /usr/sbin/httpd
apache 27538 0.0 0.0 440860 3056 ? S 17:00 0:00 \_ /usr/sbin/httpd
apache 27540 0.0 0.0 440860 2988 ? S 17:00 0:00 \_ /usr/sbin/httpd
apache 27541 0.0 0.0 440728 3032 ? S 17:00 0:00 \_ /usr/sbin/httpd
apache 27543 0.0 0.0 440728 2828 ? S 17:00 0:00 \_ /usr/sbin/httpd
apache 27544 0.0 0.0 440868 2888 ? S 17:00 0:00 \_ /usr/sbin/httpd
apache 27545 0.0 0.0 440860 2828 ? S 17:00 0:00 \_ /usr/sbin/httpd
apache 27546 0.0 0.0 440728 3064 ? S 17:00 0:00 \_ /usr/sbin/httpd
apache 30742 0.0 0.0 440728 2944 ? S 17:24 0:00 \_ /usr/sbin/httpd
[root@h83-209-52-2 ~]# mailq | tail -1
-- 1 Kbytes in 1 Request.
[root@h83-209-52-2 ~]# netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head -20
2 127.0.0.1
6 83.209.55.222
5 91.225.248.132
5
1 servers)
1 Address
[root@h83-209-52-2 ~]# uptime
18:22:28 up 2 days, 6:58, 1 user, load average: 7.88, 8.32, 8.23
[root@h83-209-52-2 ~]# free -m
total used free shared buffers cached
Mem: 3831 3742 88 36 9 99
-/+ buffers/cache: 3633 198
Swap: 15999 12221 3778
[root@h83-209-52-2 ~]# ps auxwf
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2 0.0 0.0 0 0 ? S Nov17 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Nov17 0:02 \_ [migration/0]
root 4 0.0 0.0 0 0 ? S Nov17 0:09 \_ [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S Nov17 0:00 \_ [stopper/0]
root 6 0.0 0.0 0 0 ? S Nov17 0:00 \_ [watchdog/0]
root 7 0.0 0.0 0 0 ? S Nov17 0:04 \_ [migration/1]
root 8 0.0 0.0 0 0 ? S Nov17 0:00 \_ [stopper/1]
root 9 0.0 0.0 0 0 ? S Nov17 0:10 \_ [ksoftirqd/1]
root 10 0.0 0.0 0 0 ? S Nov17 0:00 \_ [watchdog/1]
root 11 0.0 0.0 0 0 ? S Nov17 0:12 \_ [events/0]
root 12 0.0 0.0 0 0 ? S Nov17 0:16 \_ [events/1]
root 13 0.0 0.0 0 0 ? S Nov17 0:00 \_ [cgroup]
root 14 0.0 0.0 0 0 ? S Nov17 0:00 \_ [khelper]
root 15 0.0 0.0 0 0 ? S Nov17 0:00 \_ [netns]
root 16 0.0 0.0 0 0 ? S Nov17 0:00 \_ [async/mgr]
root 17 0.0 0.0 0 0 ? S Nov17 0:00 \_ [pm]
root 18 0.0 0.0 0 0 ? S Nov17 0:00 \_ [sync_supers]
root 19 0.0 0.0 0 0 ? S Nov17 0:00 \_ [bdi-default]
root 20 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kintegrityd/0]
root 21 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kintegrityd/1]
root 22 0.0 0.0 0 0 ? S Nov17 1:16 \_ [kblockd/0]
root 23 0.0 0.0 0 0 ? S Nov17 1:17 \_ [kblockd/1]
root 24 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kacpid]
root 25 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kacpi_notify]
root 26 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kacpi_hotplug]
root 27 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ata_aux]
root 28 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ata_sff/0]
root 29 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ata_sff/1]
root 30 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ksuspend_usbd]
root 31 0.0 0.0 0 0 ? S Nov17 0:00 \_ [khubd]
root 32 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kseriod]
root 33 0.0 0.0 0 0 ? S Nov17 0:00 \_ [md/0]
root 34 0.0 0.0 0 0 ? S Nov17 0:00 \_ [md/1]
root 35 0.0 0.0 0 0 ? S Nov17 0:00 \_ [md_misc/0]
root 36 0.0 0.0 0 0 ? S Nov17 0:00 \_ [md_misc/1]
root 37 0.0 0.0 0 0 ? S Nov17 0:00 \_ [linkwatch]
root 38 0.0 0.0 0 0 ? S Nov17 0:00 \_ [khungtaskd]
root 39 0.6 0.0 0 0 ? D Nov17 20:23 \_ [kswapd0]
root 40 0.0 0.0 0 0 ? SN Nov17 0:00 \_ [ksmd]
root 41 0.0 0.0 0 0 ? SN Nov17 0:07 \_ [khugepaged]
root 42 0.0 0.0 0 0 ? S Nov17 0:00 \_ [aio/0]
root 43 0.0 0.0 0 0 ? S Nov17 0:00 \_ [aio/1]
root 44 0.0 0.0 0 0 ? S Nov17 0:00 \_ [crypto/0]
root 45 0.0 0.0 0 0 ? S Nov17 0:00 \_ [crypto/1]
root 53 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kthrotld/0]
root 54 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kthrotld/1]
root 57 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kpsmoused]
root 58 0.0 0.0 0 0 ? S Nov17 0:00 \_ [usbhid_resumer]
root 59 0.0 0.0 0 0 ? S Nov17 0:00 \_ [deferwq]
root 90 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kdmremove]
root 91 0.0 0.0 0 0 ? S Nov17 0:00 \_ [kstriped]
root 121 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ttm_swap]
root 182 0.0 0.0 0 0 ? S Nov17 0:00 \_ [scsi_eh_0]
root 183 0.0 0.0 0 0 ? S Nov17 0:00 \_ [scsi_eh_1]
root 184 0.0 0.0 0 0 ? S Nov17 0:00 \_ [scsi_eh_2]
root 185 0.0 0.0 0 0 ? S Nov17 0:00 \_ [scsi_eh_3]
root 186 0.0 0.0 0 0 ? S Nov17 0:00 \_ [scsi_eh_4]
root 187 0.0 0.0 0 0 ? S Nov17 0:00 \_ [scsi_eh_5]
root 317 0.0 0.0 0 0 ? S Nov17 0:06 \_ [jbd2/sdb2-8]
root 318 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ext4-dio-unwrit]
root 435 0.0 0.0 0 0 ? S Nov17 0:03 \_ [edac-poller]
root 486 0.0 0.0 0 0 ? S Nov17 0:13 \_ [flush-8:0]
root 489 0.0 0.0 0 0 ? S Nov17 0:02 \_ [flush-8:16]
root 491 0.0 0.0 0 0 ? S Nov17 0:05 \_ [flush-8:32]
root 750 0.0 0.0 0 0 ? S Nov17 0:00 \_ [jbd2/sda1-8]
root 751 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ext4-dio-unwrit]
root 752 0.0 0.0 0 0 ? S Nov17 0:03 \_ [jbd2/sdc2-8]
root 753 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ext4-dio-unwrit]
root 754 0.0 0.0 0 0 ? S Nov17 0:16 \_ [jbd2/sda5-8]
root 755 0.0 0.0 0 0 ? S Nov17 0:00 \_ [ext4-dio-unwrit]
root 923 0.0 0.0 0 0 ? S Nov17 0:01 \_ [kauditd]
root 1305 0.0 0.0 0 0 ? S Nov17 1:21 \_ [kondemand/0]
root 1306 0.0 0.0 0 0 ? S Nov17 0:53 \_ [kondemand/1]
root 1 0.0 0.0 19364 84 ? Ss Nov17 0:01 /sbin/init
root 404 0.0 0.0 10784 388 ? Sroot@pts/0
root 4487 0.0 0.0 108340 1700 pts/0 Ss 18:20 0:00 \_ -bash
root 4779 0.0 0.0 110344 1132 pts/0 R+ 18:22 0:00 \_ ps auxwf
root 1704 0.0 0.0 108204 8 ? S Nov17 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var
mysql 1836 1.3 0.9 1183288 35668 ? Sl Nov17 45:16 \_ /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --u
root 1863 0.0 0.0 19268 332 ? Ss Nov17 0:02 /usr/sbin/dovecot
dovecot 1864 0.0 0.0 12988 344 ? S Nov17 0:00 \_ dovecot/anvil
root 1866 0.0 0.0 13112 464 ? S Nov17 0:00 \_ dovecot/log
root 1878 0.0 0.0 68976 8 ? Ss Nov17 0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -r
root 1879 0.0 0.0 68976 1020 ? S Nov17 0:00 \_ /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -r
root 1880 0.0 0.0 68976 28 ? S Nov17 0:00 \_ /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -r
root 1881 0.0 0.0 68976 8 ? S Nov17 0:00 \_ /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -r
root 1882 0.0 0.0 68976 1020 ? S Nov17 0:00 \_ /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -r
grmilter 1892 0.0 0.0 261484 572 ? Ssl Nov17 0:05 /usr/sbin/milter-greylist -u grmilter -P /var/run/milter-greylist.p
root 1999 0.0 0.0 81328 256 ? Ss Nov17 0:01 /usr/libexec/postfix/master
postfix 2011 0.0 0.0 81596 468 ? S Nov17 0:00 \_ qmgr -l -t fifo -u
postfix 27264 0.0 0.0 81408 444 ? S 16:58 0:00 \_ pickup -l -t fifo -u
postfix 3929 0.0 0.0 81700 3364 ? S 18:15 0:00 \_ local -t unix
root 3932 0.0 0.0 8828 808 ? Ss 18:15 0:00 | \_ /usr/bin/procmail-wrapper -o -a h83-209-52-2.dynamic.se.all
root 3933 0.0 0.0 8828 160 ? S 18:15 0:00 | \_ /usr/bin/procmail-wrapper -o -a h83-209-52-2.dynamic.se
root 3934 0.0 0.0 30376 1956 ? S 18:15 0:00 | \_ /usr/bin/perl /usr/libexec/webmin/virtual-server/lo
postfix 4549 0.0 0.0 81412 3796 ? S 18:21 0:00 \_ showq -t unix -u
root 2036 0.0 0.0 114676 8 ? Ss Nov17 0:00 /usr/sbin/abrtd
520 2056 0.0 0.0 411108 252 ? Sl Nov17 0:42 /usr/local/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf
root 2072 0.0 0.0 117332 356 ? Ss Nov17 0:04 crond
root 2086 0.0 0.0 21576 4 ? Ss Nov17 0:00 /usr/sbin/atd
root 2101 0.0 0.0 64704 84 ? Ss Nov17 0:00 /usr/sbin/certmonger -S -p /var/run/certmonger.pid
root 2167 0.0 0.0 189368 740 ? Ss Nov17 0:00 /usr/libexec/webmin/virtual-server/lookup-domain-daemon.pl
root 2175 0.0 0.0 87580 920 ? Ss Nov17 0:02 /usr/bin/perl /usr/libexec/usermin/miniserv.pl /etc/usermin/miniser
root 2183 0.0 0.0 140204 2860 ? Ss Nov17 0:12 /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/miniserv.
root 4301 0.0 0.6 145904 25320 ? S 18:19 0:00 \_ /usr/libexec/webmin/webmincron/webmincron.pl
root 4306 1.6 2.6 223044 105124 ? D 18:19 0:03 | \_ /usr/libexec/webmin/virtual-server/collectinfo.pl
root 4436 0.0 0.1 140316 5496 ? S 18:20 0:00 \_ /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/minis
root 4438 0.0 0.1 140316 5492 ? S 18:20 0:00 \_ /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/minis
root 4439 0.0 0.1 140316 5500 ? S 18:20 0:00 \_ /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/minis
root 4440 0.0 0.1 140316 5500 ? S 18:20 0:00 \_ /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/minis
root 4488 0.0 0.1 140316 5496 ? S 18:20 0:00 \_ /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/minis
root 4711 1.3 0.5 145340 21688 ? D 18:22 0:00 \_ /usr/libexec/webmin/webmincron/webmincron.pl
root 2192 0.0 0.0 4064 8 tty1 Ss+ Nov17 0:00 /sbin/mingetty /dev/tty1
root 2194 0.0 0.0 4064 8 tty2 Ss+ Nov17 0:00 /sbin/mingetty /dev/tty2
root 2196 0.0 0.0 4064 8 tty3 Ss+ Nov17 0:00 /sbin/mingetty /dev/tty3
root 2198 0.0 0.0 4064 8 tty4 Ss+ Nov17 0:00 /sbin/mingetty /dev/tty4
root 2200 0.0 0.0 4064 8 tty5 Ss+ Nov17 0:00 /sbin/mingetty /dev/tty5
root 2202 0.0 0.0 4064 8 tty6 Ss+ Nov17 0:00 /sbin/mingetty /dev/tty6
nobody 28916 0.0 0.0 55736 388 ? Ss Nov17 0:02 proftpd: (accepting connections)
root 30504 0.0 0.0 440596 320 ? Ss Nov17 0:05 /usr/sbin/httpd
apache 30506 0.0 0.0 251964 248 ? S Nov17 0:02 \_ /usr/sbin/httpd
504 30696 0.0 0.4 356528 17392 ? S Nov17 1:25 | \_ /usr/bin/php-cgi
502 30746 0.0 0.1 267076 5124 ? S Nov17 0:49 | \_ /usr/bin/php-cgi
507 31863 0.0 0.1 357796 6496 ? S Nov17 0:43 | \_ /usr/bin/php-cgi
504 4757 0.0 0.4 350500 15948 ? S Nov18 0:44 | \_ /usr/bin/php-cgi
514 1795 0.3 0.3 205020 12048 ? S Nov18 5:03 | \_ /opt/rh/php54/root/usr/bin/php-cgi
502 9559 0.0 0.2 269740 8692 ? S 00:55 0:06 | \_ /usr/bin/php-cgi
502 22566 0.0 0.2 258564 8088 ? S 16:25 0:00 | \_ /usr/bin/php-cgi
514 23388 0.2 0.4 200184 17292 ? S 16:32 0:18 | \_ /opt/rh/php54/root/usr/bin/php-cgi
507 29596 0.0 0.3 357800 13128 ? S 17:16 0:01 | \_ /usr/bin/php-cgi
514 32023 0.1 0.2 188832 10180 ? S 17:37 0:04 | \_ /opt/rh/php54/root/usr/bin/php-cgi
apache 1585 0.0 0.0 455028 2940 ? S Nov18 0:11 \_ /usr/sbin/httpd
apache 1679 0.0 0.0 441248 920 ? S Nov18 0:11 \_ /usr/sbin/httpd
520 21554 3.4 24.3 5271644 955124 ? D 16:18 4:17 | \_ /usr/bin/php-cgi
apache 1806 0.0 0.0 441240 944 ? S Nov18 0:11 \_ /usr/sbin/httpd
520 23211 3.8 29.0 5271644 1141484 ? D 16:31 4:12 | \_ /usr/bin/php-cgi
apache 1809 0.0 0.0 442740 2888 ? S Nov18 0:12 \_ /usr/sbin/httpd
apache 6872 0.0 0.0 442888 2928 ? S Nov18 0:11 \_ /usr/sbin/httpd
apache 6885 0.0 0.0 445620 948 ? S Nov18 0:11 \_ /usr/sbin/httpd
520 22403 3.5 18.2 5271644 717836 ? D 16:24 4:15 | \_ /usr/bin/php-cgi
apache 6613 0.0 0.0 442740 2820 ? S 00:35 0:07 \_ /usr/sbin/httpd
apache 6731 0.0 0.0 442948 2936 ? S 00:36 0:06 \_ /usr/sbin/httpd
apache 7417 0.0 0.0 442408 2884 ? S 00:40 0:06 \_ /usr/sbin/httpd
apache 8132 0.0 0.0 441900 2896 ? S 00:46 0:06 \_ /usr/sbin/httpd
apache 9923 0.0 0.0 442332 2888 ? S 00:58 0:06 \_ /usr/sbin/httpd
apache 21623 0.0 0.0 442672 2904 ? S 02:11 0:06 \_ /usr/sbin/httpd
apache 23331 0.0 0.0 440860 2860 ? S 16:32 0:00 \_ /usr/sbin/httpd
apache 23342 0.0 0.0 440864 3056 ? S 16:32 0:00 \_ /usr/sbin/httpd
apache 23386 0.0 0.0 440864 3120 ? S 16:32 0:00 \_ /usr/sbin/httpd
apache 27538 0.0 0.0 440860 3204 ? S 17:00 0:00 \_ /usr/sbin/httpd
apache 27540 0.0 0.0 440860 2988 ? S 17:00 0:00 \_ /usr/sbin/httpd
apache 27541 0.0 0.0 440728 3032 ? S 17:00 0:00 \_ /usr/sbin/httpd
apache 27543 0.0 0.0 440728 2828 ? S 17:00 0:00 \_ /usr/sbin/httpd
apache 27544 0.0 0.0 440868 2888 ? S 17:00 0:00 \_ /usr/sbin/httpd
520 4774 4.6 0.5 288724 20392 ? D 18:22 0:00 | \_ /usr/bin/php-cgi
apache 27545 0.0 0.0 440860 2828 ? S 17:00 0:00 \_ /usr/sbin/httpd
apache 27546 0.0 0.0 440728 3064 ? S 17:00 0:00 \_ /usr/sbin/httpd
apache 30742 0.0 0.0 440728 2944 ? S 17:24 0:00 \_ /usr/sbin/httpd
[root@h83-209-52-2 ~]# mailq | tail -1

-- 1 Kbytes in 1 Request.
[root@h83-209-52-2 ~]# netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head -20
2 127.0.0.1
6 83.209.55.222
5 91.225.248.132
6
1 servers)
1 Address
[root@h83-209-52-2 ~]#

It looks like the culprit is the PHP app being run by the user with the ID "520".

There are several PHP processes being run as that user which are using a large amount of resources.

You can determine which user that is by running this command:

grep :520 /etc/passwd

LarsReimers's picture
Submitted by LarsReimers on Wed, 11/19/2014 - 11:30

Hmm!

[root@h83-209-52-2 ~]# grep :520 /etc/passwdgrep :520 /etc/passwd grep: /etc/passwdgrep: Filen eller katalogen finns inte grep: :520: Filen eller katalogen finns inte /etc/passwd:rallycrosstube:x:520:519:HD Ready Video for rallycross fans - Watch, Share and Enjoy rallycross video and photo.:/home/rallycrosstube:/bin/bash [root@h83-209-52-2 ~]#

Okay, it looks like that is the user "rallycrosstube".

I unfortunately don't know what the cause is -- determining that will take some knowledge of the PHP app that's being run from within that user's account.

What I can offer is that it's taking up a very large amount of resources.

You may want to review that users site, and make sure it's fully up to date, including all the plugins it's using, in case that issue is a bug that has since been fixed.

LarsReimers's picture
Submitted by LarsReimers on Wed, 11/19/2014 - 11:51

Ok, it's my own video sharing website www.rallycrosstube.com.

Thanks, now I know more or less in which direction I should look.

Thanks for the help.

// Lars