Hi Virtualmen!
I'm experiencing an odd problem of late where php-cgi processes seem to just hang out indefinitely and build up to very large numbers.
Check out the dates in the log (I killed all the processes 2 days ago, this is just the orphans hanging out since then)
user1 830 25232 0 Apr20 ? 00:00:59 /usr/bin/php-cgi
user1 3781 25232 0 Apr20 ? 00:00:01 /usr/bin/php-cgi
user1 3783 25232 0 Apr20 ? 00:00:03 /usr/bin/php-cgi
510 5790 25232 0 Apr20 ? 00:00:34 /usr/bin/php-cgi
510 5792 25232 0 Apr20 ? 00:00:24 /usr/bin/php-cgi
510 5793 25232 0 Apr20 ? 00:00:31 /usr/bin/php-cgi
user2 6209 25232 0 10:59 ? 00:00:00 /bin/bash php5.fcgi
user2 6210 6209 0 10:59 ? 00:00:01 /usr/bin/php-cgi
530 7691 25232 0 Apr21 ? 00:00:00 /bin/bash php5.fcgi
530 7692 7691 0 Apr21 ? 00:00:02 /usr/bin/php-cgi
user3 8545 25232 0 Apr20 ? 00:00:41 /usr/bin/php-cgi
504 10865 25232 0 Apr21 ? 00:00:00 /bin/bash php5.fcgi
504 10866 10865 0 Apr21 ? 00:00:00 /usr/bin/php-cgi
504 10867 10866 0 Apr21 ? 00:00:02 /usr/bin/php-cgi
504 10868 10866 0 Apr21 ? 00:00:02 /usr/bin/php-cgi
504 10869 10866 0 Apr21 ? 00:00:01 /usr/bin/php-cgi
504 10870 10866 0 Apr21 ? 00:00:01 /usr/bin/php-cgi
user2 11270 25232 0 12:27 ? 00:00:00 /bin/bash php5.fcgi
user2 11271 11270 0 12:27 ? 00:00:02 /usr/bin/php-cgi
user2 14304 25232 0 13:13 ? 00:00:00 /bin/bash php5.fcgi
user2 14305 14304 0 13:13 ? 00:00:00 /usr/bin/php-cgi
528 16454 25232 0 Apr20 ? 00:00:00 /usr/bin/php-cgi
root 18949 11881 0 13:59 pts/0 00:00:00 grep php
user4 19944 25232 0 Apr21 ? 00:00:46 /usr/bin/php-cgi
user5 20083 25232 0 Apr20 ? 00:00:00 /usr/bin/php-cgi
user4 24486 25232 0 07:08 ? 00:00:09 /usr/bin/php-cgi
user4 24487 25232 0 07:08 ? 00:00:20 /usr/bin/php-cgi
user4 24499 25232 0 07:08 ? 00:01:47 /usr/bin/php-cgi
user4 24501 25232 0 07:08 ? 00:01:34 /usr/bin/php-cgi
530 24682 25232 0 Apr21 ? 00:00:00 /bin/bash php5.fcgi
530 24683 24682 0 Apr21 ? 00:00:50 /usr/bin/php-cgi
user3 25754 25232 0 Apr20 ? 00:00:43 /usr/bin/php-cgi
user1 25756 25232 0 Apr20 ? 00:01:10 /usr/bin/php-cgi
user6 26241 25232 0 Apr20 ? 00:01:05 /usr/bin/php-cgi
user2 26307 25232 0 Apr20 ? 00:00:00 /bin/bash php5.fcgi
user2 26308 26307 0 Apr20 ? 00:00:01 /usr/bin/php-cgi
506 26511 25232 0 Apr20 ? 00:00:00 /usr/bin/php-cgi
user7 26807 25232 0 Apr20 ? 00:00:00 /usr/bin/php-cgi
502 27663 25232 0 Apr20 ? 00:00:00 /bin/bash php5.fcgi
502 27664 27663 0 Apr20 ? 00:00:02 /usr/bin/php-cgi
504 27797 25232 0 Apr20 ? 00:00:00 /bin/bash php5.fcgi
504 27798 27797 0 Apr20 ? 00:00:00 /usr/bin/php-cgi
504 27799 27798 0 Apr20 ? 00:00:04 /usr/bin/php-cgi
504 27800 27798 0 Apr20 ? 00:00:04 /usr/bin/php-cgi
504 27801 27798 0 Apr20 ? 00:00:03 /usr/bin/php-cgi
504 27802 27798 0 Apr20 ? 00:00:03 /usr/bin/php-cgi
502 28343 25232 0 Apr20 ? 00:00:00 /bin/bash php5.fcgi
502 28344 28343 0 Apr20 ? 00:00:05 /usr/bin/php-cgi
user5 28375 25232 0 Apr20 ? 00:00:11 /usr/bin/php-cgi
user6 29592 25232 0 Apr20 ? 00:00:00 /usr/bin/php-cgi
user6 29964 25232 0 Apr20 ? 00:00:00 /usr/bin/php-cgi
506 30863 25232 0 Apr20 ? 00:00:01 /usr/bin/php-cgi
user2 32028 25232 0 Apr20 ? 00:00:00 /bin/bash php5.fcgi
user2 32029 32028 0 Apr20 ? 00:00:45 /usr/bin/php-cgi
Any thoughts on what's going wrong and where I should start troubleshooting?
Well, that is quite a few... in smaller numbers, I'd suggest they're just using fcgid. The point of fcgid is to cache your PHP code in memory so that it can be served up faster. The default isn't to have that many running processes though.
Are you using fcgid? You can determine that in Server Configuration -> Website Options.
-Eric
yes, all sites are running fcgid