zombie processes including httpd, php-cgi, and miniserv.pl

7 posts / 0 new
Last post
#1 Tue, 01/25/2011 - 11:39
arosborough

zombie processes including httpd, php-cgi, and miniserv.pl

I have installed Virtualmin on Centos 5.5 and kept it up to date. I have written a script to keep track of zombie processes, and lately I have been noticing miniserv.pl go defunct every 5 minutes or so.

Is this normal? What approach should I take to resolve this issue? What about the web server and php-cgi?

I tried changing the settings for fast cgi, but it hasn't helped:

<IfModule !mod_fastcgi.c>
    AddHandler fcgid-script fcg fcgi fpl
    IPCConnectTimeout 20
    DefaultMaxClassProcessCount 16
    MaxProcessCount 100
    MaxRequestsPerProcess 500
</IfModule>

Here is the zombie.log:

Fri Jan 21 16:33:01 EST 2011
Z     3919 20621 [httpd] <defunct>
Sat Jan 22 04:27:39 EST 2011
Z    20616 19491 [php-cgi] <defunct>
Sat Jan 22 22:01:01 EST 2011
Z     3919 29739 [httpd] <defunct>
Sun Jan 23 04:03:02 EST 2011
Z     3919 20616 [httpd] <defunct>
Tue Jan 25 08:50:01 EST 2011
Z     7002 32243 [php-cgi] <defunct>
Tue Jan 25 11:30:02 EST 2011
Z     4101  4163 [miniserv.pl] <defunct>
Tue Jan 25 11:31:02 EST 2011
Z     4101  4162 [miniserv.pl] <defunct>
Tue Jan 25 11:37:01 EST 2011
Z     4101  4723 [miniserv.pl] <defunct>
Tue Jan 25 11:44:01 EST 2011
Z     4101  5093 [miniserv.pl] <defunct>
Tue Jan 25 11:51:02 EST 2011
Z     4101  5506 [miniserv.pl] <defunct>
Tue Jan 25 11:59:01 EST 2011
Z     4101  6006 [miniserv.pl] <defunct>
Tue Jan 25 12:06:01 EST 2011
/var/log/zombies.log

Thanks for your support

Tue, 01/25/2011 - 12:45
andreychek

Hmm, that's all pretty unusual. That sounds like you're seeing parent processes mysteriously dieing off.

Are you by chance on a VPS, or some sort of memory constrained system? I'd be curious if you're by chance seeing some sort of issue where, due to memory constraints, the kernel is killing processes to keep your server online.

If that's the case, you should see messages regarding that in the dmesg output, or in the logs files in /var/log.

-Eric

Tue, 01/25/2011 - 13:11 (Reply to #2)
arosborough

This is a virtual server running with 2GB dedicated memory and 4GB total available.

Here is what top says about the resources:

top - 13:16:30 up  1:48,  1 user,  load average: 2.67, 1.19, 0.58
Tasks: 161 total,   7 running, 153 sleeping,   0 stopped,   1 zombie
Cpu(s): 45.2%us,  7.5%sy,  0.0%ni, 44.8%id,  0.0%wa,  0.3%hi,  2.2%si,  0.0%st
Mem:   4044472k total,   861940k used,  3182532k free,    55296k buffers
Swap:  4095992k total,        0k used,  4095992k free,   324636k cached

I haven't been able to find anything in /var/log/messages or /var/log/dmesg that would indicate the processes are being killed. But the zombies do go away on their own.

I found these errors in the http error_log which seem to relate to the defunct httpd and php-cgi process:

[Tue Jan 25 13:38:35 2011] [notice] mod_fcgid: process /home/username/public_html/index.php(11519) exit(idle timeout), terminated by calling exit(), return code: 0
[Tue Jan 25 13:38:35 2011] [notice] mod_fcgid: process /home/username/public_html/index.php(10200) exit(idle timeout), terminated by calling exit(), return code: 0
[Tue Jan 25 13:38:35 2011] [notice] mod_fcgid: process /home/username/public_html/index.php(11353) exit(idle timeout), terminated by calling exit(), return code: 0
[Tue Jan 25 13:38:35 2011] [notice] mod_fcgid: process /home/username/public_html/index.php(11354) exit(idle timeout), terminated by calling exit(), return code: 0
[Tue Jan 25 13:38:35 2011] [notice] mod_fcgid: process /home/username/public_html/index.php(10196) exit(idle timeout), terminated by calling exit(), return code: 0
[Tue Jan 25 14:00:41 2011] [warn] mod_fcgid: process 11372 graceful kill fail, sending SIGKILL
[Tue Jan 25 14:00:47 2011] [notice] mod_fcgid: process /home/username/public_html/index.php(11372) exit(communication error), get stop signal 9

I'm still not sure what is wrong with miniserv.pl

are there supposed to be two processes running at the same time?

# ps aux | grep miniserv | grep -v grep
root      4096  0.0  0.3  77760 14624 ?        Ss   11:29   0:00 /usr/bin/perl /usr/libexec/usermin/miniserv.pl /etc/usermin/miniserv.conf
root      4101  0.0  1.0 113512 44388 ?        Ss   11:29   0:01 /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/miniserv.conf

pid 4101 is the one that is having zombie children.

Tue, 01/25/2011 - 13:43
andreychek

Howdy,

Okay, so something else weird is going on :-)

You certainly seem to have plenty of RAM.

As far as the miniserv.pl processes -- take a close look at the path to each of those. One is for Usermin, the other is for Webmin. So that much is correct, you should indeed have two running.

In your case, it looks like it's the Webmin process where you're seeing the zombie children.

But that again leads to the question... why are the parent processes for 2-3 different daemons dieing off. And for that, I'm not entirely certain.

The best I can offer is to closely study the logs, and try to figure out what the daemon was doing at the time, and who asked it to do that.

For example, all the Webmin actions would be listed in /var/webmin/miniserv.log. You may be able to look at when the process was started, and determine from the logs what may have happened.

For Apache -- I'm curious if you'd run into less problems if you were to use CGI rather than FCGID. Some folks run into occasional issues with it... while those are different from what you're seeing, it may still be worth trying CGI to see if some of your problems go away. You can switch the PHP Execution mode by going into Server Configuration -> Website Options.

Just some thoughts, I'm honestly not sure why you're seeing all that though.

-Eric

Tue, 01/25/2011 - 13:49 (Reply to #4)
arosborough

Thanks for your help Eric,

The miniserv.log doesn't contain any entries at the same time as the zombie process. I will try switch from fast-cgi to CGI and see what happens.

Andrew

Sun, 01/30/2011 - 11:43
arosborough

The httpd and php-cgi defunct processes have disappeared since I changed CGI modes. However I still keep getting defunct webmin processes for webmin 1.530 every 5 to 10 minutes on two servers (identical virtual machines). Should I file some sort of bug report?

I've attached my script if anyone would like to check their machine for zombie processes

It requires this line in /etc/crontab to run every minute.

*/1 * * * * root /root/zombies --log
Sun, 01/30/2011 - 11:58
andreychek

Howdy,

Well, what you're seeing is pretty atypical... while I don't know why exactly, what you're seeing is likely specific to your setup.

I say that since it seems to happen to multiple daemons on your server, as well as because I can't reproduce that on any other systems :-)

The two Virtual Machines you're seeing that on -- do they both run on the same host?

I'd be curious if you'd consider an experiment :-)

If you install something like Virtualbox on your desktop, and in it, install the same Linux distro that you're two other Virtual Machines are using. Give it a similar amount of RAM that your two Virtual Machines have.

And then, install Virtualmin onto your setup in Virtualbox, and put your Zombie hunter on there.

In that case, are you seeing any zombie processes?

If you can find a way to reproduce this issue outside of the Virtual Machine's your other two servers are running on, you may have a case for calling this a bug (or at least, it warrants looking deeper into it). However, I have a suspicion that it's somehow related to the configuration of those two Virtual Machines (or perhaps their underlying host).

-Eric