The system was upgraded from long-working Debian/Etch + VirtualMin installation using these steps: http://www.virtualmin.com/documentation/id,upgrading_debian_etch_to_lenn...
After successful upgrade the system is unstable with following symptoms:
- dmesg is full of segfaults of php5-cgi, two kinds of errors appear:
php5-cgi: segfault at 00002b47e6aa7edb rip 00002aaaaaac31af rsp 00000000407ffc90 error 4 php5-cgi: segfault at 00002b868b0a5ed0 rip 00002b868b0a5ed0 rsp 0000000040800128 error 14
with ration of about 1:5 (67 of the former message, 303 of the latter).
- after some time one of Apache2 processes hangs and has +- 100% CPU usage (varies like 97%-103% - this is dualcore system).
The apache that hangs is not the "root" apache, but rather the one spawning fastcgi processes:
16908 ? Ss 0:06 /usr/sbin/apache2 -k start
17153 ? R 7:01 \_ /usr/sbin/apache2 -k start # <<== this one
17169 ? Z 0:01 \_ [php5-cgi] <defunct>
17180 ? Z 0:01 \_ [php5-cgi] <defunct>
17245 ? Z 0:05 \_ [php4-cgi] <defunct>
17259 ? Z 0:00 \_ [php5-cgi] <defunct>
17260 ? Z 0:01 \_ [php5-cgi] <defunct>
When strace -p is used to attach this process, it does nothing, so I tried using gdb, to trackdown the error and it seems like infinite loop in mod_fcgid.so:
#0 0x00007fa83ac0bf1f in pm_main () from /usr/lib/apache2/modules/mod_fcgid.so
#1 0x00007fa83ac0ee22 in procmgr_post_config () from /usr/lib/apache2/modules/mod_fcgid.so
#2 0x00007fa83ac0ddff in ?? () from /usr/lib/apache2/modules/mod_fcgid.so
#3 0x0000000000438cf4 in ap_run_post_config ()
#4 0x0000000000425bbc in main ()
In case you need further information, please do not hesitate to contact me.
Dominik Pantůček, CTO Trustica s.r.o.