Is there any issue with system resources in the current (3.17) version of VirtualminPro? Over the last couple weeks performance has decayed and anything webserver associated nearly shuts down the entire box.

When I check the system resources I'm using over 800Meg of 1 GIg of RAM. Pretty steep for a dual PIII 933M. There seems to be a cascade of writelog failures that I assume create more process, leading to a ton of serverdown monitor emails.

The only thing coincident with this is using the AWBS CRON job, that process seems to really load up resources but I'm not running it as an active cron.

Any ideas out there?

Hey Dan,

Hard to say without something more concrete to go on. I'm unaware of any issues, but that's not to say something can't be going wrong on your system.

When you say "I'm using over 800Meg of 1 GIg of RAM", it's impossible to know where that's all going without seeing the details. How about sending along the output of this command:

top -b -n 1

This will show me the details of what exactly is using memory, whether it's in active use or swapped out, etc. We can then figure out which processes we need to start looking at for a cuase and solution to your problems.

Wow!! That's pretty stark.

> top -b -n 1


14:19:15 up 48 min, 0 users, load average: 0.56, 0.18, 1.28
103 processes: 96 sleeping, 6 running, 1 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 43.0% 0.0% 56.0% 0.0% 0.4% 0.0% 0.4%
cpu00 42.2% 0.0% 56.9% 0.0% 0.8% 0.0% 0.0%
cpu01 43.9% 0.0% 55.2% 0.0% 0.0% 0.0% 0.8%
Mem: 1025372k av, 948496k used, 76876k free, 0k shrd, 11528k buff
746540k actv, 124544k in_d, 12540k in_c
Swap: 2040244k av, 178028k used, 1862216k free 47548k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
2357 root 24 0 0 0 0 Z 10.1 0.0 0:00 1 miniserv.pl[
2359 root 24 0 74508 47M 1680 R 7.7 4.7 0:00 0 miniserv.pl
1275 root 24 0 73432 2796 616 S 6.1 0.2 0:07 1 miniserv.pl
2360 root 25 0 74512 35M 1684 R 5.7 3.5 0:00 0 miniserv.pl
2361 root 25 0 74512 32M 1684 R 4.8 3.2 0:00 0 miniserv.pl
2351 root 24 0 1000 1000 760 R 0.8 0.0 0:00 1 top
2363 root 25 0 74472 4608 1644 R 0.8 0.4 0:00 0 miniserv.pl
2362 root 25 0 74484 4644 1656 S 0.4 0.4 0:00 1 miniserv.pl
2364 root 24 0 74328 4052 1504 R 0.4 0.3 0:00 1 miniserv.pl
1 root 15 0 104 64 44 S 0.0 0.0 0:04 0 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
3 root RT 0 0 0 0 SW 0.0 0.0 0:00 1 migration/1
4 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd
5 root 39 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
6 root 34 19 0 0 0 SWN 0.0 0.0 0:00 1 ksoftirqd/1
9 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
7 root 15 0 0 0 0 SW 0.0 0.0 0:09 1 kswapd
8 root 15 0 0 0 0 SW 0.0 0.0 0:11 1 kscand
10 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kupdated
11 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
15 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
409 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
761 root 15 0 216 180 132 S 0.0 0.0 0:00 0 syslogd
765 root 15 0 64 4 0 S 0.0 0.0 0:00 0 klogd
775 root 15 0 244 236 180 S 0.0 0.0 0:00 1 irqbalance
792 rpc 18 0 76 4 0 S 0.0 0.0 0:00 1 portmap
811 rpcuser 25 0 80 4 0 S 0.0 0.0 0:00 1 rpc.statd
822 root 15 0 176 156 116 S 0.0 0.0 0:00 1 mdadm
914 root 15 0 260 88 52 S 0.0 0.0 0:00 1 dovecot
924 root 15 0 428 112 40 S 0.0 0.0 0:00 0 dovecot-auth
925 dovecot 15 0 292 80 40 S 0.0 0.0 0:00 0 imap-login
926 dovecot 15 0 292 80 40 S 0.0 0.0 0:00 1 imap-login
927 dovecot 15 0 288 80 40 S 0.0 0.0 0:00 1 imap-login
928 dovecot 15 0 292 80 40 S 0.0 0.0 0:00 1 pop3-login
929 dovecot 15 0 292 80 40 S 0.0 0.0 0:00 0 pop3-login
930 dovecot 15 0 292 80 40 S 0.0 0.0 0:00 0 pop3-login
933 root 15 0 692 320 228 S 0.0 0.0 0:00 0 cupsd
944 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
957 named 15 0 1896 1380 888 S 0.0 0.1 0:00 0 named
958 named 15 0 1896 1380 888 S 0.0 0.1 0:00 1 named
959 named 15 0 1896 1380 888 S 0.0 0.1 0:00 0 named
960 named 15 0 1896 1380 888 S 0.0 0.1 0:00 1 named
961 named 15 0 1896 1380 888 S 0.0 0.1 0:00 0 named
971 root 25 0 256 4 0 S 0.0 0.0 0:00 0 sshd
985 root 24 0 120 4 0 S 0.0 0.0 0:00 1 xinetd
994 clamav 25 0 12728 4 0 S 0.0 0.0 0:00 0 clamd
1006 root 25 0 128 4 0 S 0.0 0.0 0:00 0 safe_mysqld
1035 mysql 15 0 1552 1208 840 S 0.0 0.1 0:00 1 mysqld
1036 mysql 15 0 1552 1208 840 S 0.0 0.1 0:00 0 mysqld
1092 root 15 0 452 212 140 S 0.0 0.0 0:00 1 master
1096 postfix 15 0 432 184 124 S 0.0 0.0 0:00 0 pickup
1097 postfix 15 0 832 504 376 S 0.0 0.0 0:00 1 nqmgr
1104 nobody 15 0 656 200 124 S 0.0 0.0 0:00 0 proftpd
1114 root 25 0 52 4 0 S 0.0 0.0 0:00 0 gpm
1170 root 15 0 484 212 112 S 0.0 0.0 0:00 1 crond
1213 xfs 15 0 2240 60 32 S 0.0 0.0 0:00 1 xfs
1222 daemon 15 0 200 172 124 S 0.0 0.0 0:00 0 atd
1231 root 25 0 104 4 0 S 0.0 0.0 0:00 1 saslauthd
1235 root 25 0 104 4 0 S 0.0 0.0 0:00 1 saslauthd
1236 root 25 0 104 4 0 S 0.0 0.0 0:00 1 saslauthd
1237 root 25 0 104 4 0 S 0.0 0.0 0:00 1 saslauthd
1243 root 25 0 104 4 0 S 0.0 0.0 0:00 1 saslauthd
1256 root 15 0 4100 852 424 S 0.0 0.0 0:00 0 miniserv.pl
1283 root 19 0 56 4 0 S 0.0 0.0 0:00 1 mingetty
1284 root 19 0 56 4 0 S 0.0 0.0 0:00 0 mingetty
1285 root 19 0 56 4 0 S 0.0 0.0 0:00 1 mingetty
1286 root 19 0 52 4 0 S 0.0 0.0 0:00 0 mingetty
1287 root 19 0 48 4 0 S 0.0 0.0 0:00 0 mingetty
1288 root 19 0 52 4 0 S 0.0 0.0 0:00 1 mingetty
1939 root 15 0 4256 2036 840 S 0.0 0.1 0:00 1 httpd
1942 root 25 0 36212 29M 64 S 0.0 2.9 0:04 0 writelogs.pl
1944 root 24 0 36212 29M 64 S 0.0 2.9 0:04 0 writelogs.pl
1945 root 25 0 36212 28M 64 S 0.0 2.8 0:04 0 writelogs.pl
1946 root 25 0 36404 30M 256 S 0.0 3.0 0:04 0 writelogs.pl
1948 root 25 0 36360 31M 208 S 0.0 3.1 0:04 0 writelogs.pl
1949 root 19 0 37288 32M 1140 S 0.0 3.2 0:04 1 writelogs.pl
1950 root 25 0 36288 32M 140 S 0.0 3.2 0:04 0 writelogs.pl
1951 root 15 0 36580 31M 428 S 0.0 3.1 0:04 1 writelogs.pl
1952 root 25 0 36212 31M 64 S 0.0 3.1 0:04 1 writelogs.pl
1953 root 15 0 36620 33M 472 S 0.0 3.3 0:04 0 writelogs.pl
1954 root 25 0 36436 32M 288 S 0.0 3.2 0:04 0 writelogs.pl
1955 root 25 0 36444 31M 292 S 0.0 3.1 0:04 1 writelogs.pl
1956 root 24 0 36356 32M 208 S 0.0 3.2 0:04 0 writelogs.pl
1957 root 15 0 36348 32M 196 S 0.0 3.2 0:04 1 writelogs.pl
1958 root 15 0 36420 32M 268 S 0.0 3.2 0:04 1 writelogs.pl
1959 root 15 0 36852 32M 700 S 0.0 3.2 0:04 1 writelogs.pl
1960 root 25 0 36632 33M 484 S 0.0 3.3 0:04 1 writelogs.pl
1961 root 15 0 36488 34M 336 S 0.0 3.4 0:04 1 writelogs.pl
1962 root 15 0 36448 34M 296 S 0.0 3.4 0:04 1 writelogs.pl
1965 root 25 0 36604 34M 460 S 0.0 3.4 0:04 1 writelogs.pl
1966 root 15 0 36612 33M 464 S 0.0 3.3 0:04 1 writelogs.pl
1967 root 25 0 36452 35M 300 S 0.0 3.5 0:04 1 writelogs.pl
1982 apache 15 0 8176 6620 2988 S 0.0 0.6 0:00 1 httpd
1989 apache 15 0 8088 6884 4192 S 0.0 0.6 0:00 1 httpd
1990 apache 15 0 8176 6976 4292 S 0.0 0.6 0:00 0 httpd
1991 apache 15 0 8036 6768 4160 S 0.0 0.6 0:00 1 httpd
1999 apache 15 0 7140 5904 3504 S 0.0 0.5 0:00 0 httpd
2000 apache 15 0 7736 6544 3820 S 0.0 0.6 0:00 1 httpd
2001 apache 15 0 8064 6820 4168 S 0.0 0.6 0:00 0 httpd
2005 apache 15 0 7208 5988 3580 S 0.0 0.5 0:00 0 httpd
2346 root 24 0 78616 13M 2092 S 0.0 1.3 0:00 1 miniserv.pl
2349 root 25 0 956 956 844 S 0.0 0.0 0:00 1 sh
2350 root 25 0 884 884 716 S 0.0 0.0 0:00 0 su

Well I see it all in the write logs

Well that makes sense, AWBS required 32M set up in PHP.ini

Hey Dan,

Actually, I don't think the writelogs.pl is a problem, though I could be wrong. It is expected to have one writelogs process per domain. The process sizes you're looking at include all of the shared libraries--they all share the same libraries, and so the total usage of all of the writelogs.pl processes is probably pretty close to the size of one process.

There do appear to be more miniserv processes than I would expect, though. It shouldn't increase memory usage by a huge amount (again, lots of shared libs reflected in the process size), but at idle, there will be two miniserv processes (one for Webmin and one for Usermin)...when in use, or when one of the crons that relies on miniserv is running, there would be an extra. I think it would take several simultaneous logged in Webmin/Usermin users to lead to as many miniservs as you have.

Try restarting Webmin and Usermin and see if they go away, temporarily. Could be something broken in the System and Server Status monitors that's leading to hung miniserv processes (maybe...I think that's how such problems exhibit themselves).

After restarting webmin and usermin:
***********************************************

> top -b -n 1


15:01:42 up 1:31, 0 users, load average: 0.47, 0.61, 1.21
100 processes: 98 sleeping, 2 running, 0 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 40.3% 0.0% 40.3% 0.0% 0.9% 0.0% 18.2%
cpu00 40.3% 0.0% 31.7% 0.0% 0.9% 0.0% 26.9%
cpu01 40.3% 0.0% 49.0% 0.0% 0.9% 0.0% 9.6%
Mem: 1025372k av, 943344k used, 82028k free, 0k shrd, 10656k buff
717268k actv, 148060k in_d, 14132k in_c
Swap: 2040244k av, 97692k used, 1942552k free 34816k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
3989 root 20 0 75496 73M 2720 S 13.9 7.3 0:04 1 perl
4191 root 21 0 75688 73M 2888 R 4.7 7.3 0:00 0 perl
4194 root 22 0 75692 73M 2884 S 1.4 7.3 0:00 1 perl
4185 root 24 0 992 992 760 R 0.9 0.0 0:00 1 top
4193 root 22 0 75692 73M 2884 S 0.4 7.3 0:00 1 perl
4195 root 20 0 75692 73M 2884 S 0.4 7.3 0:00 1 perl
1 root 15 0 116 80 56 S 0.0 0.0 0:04 1 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
3 root RT 0 0 0 0 SW 0.0 0.0 0:00 1 migration/1
4 root 15 0 0 0 0 SW 0.0 0.0 0:00 1 keventd
5 root 39 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
6 root 34 19 0 0 0 SWN 0.0 0.0 0:00 1 ksoftirqd/1
9 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
7 root 15 0 0 0 0 SW 0.0 0.0 0:10 1 kswapd
8 root 15 0 0 0 0 SW 0.0 0.0 0:15 1 kscand
10 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kupdated
11 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
15 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
409 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
761 root 15 0 256 232 172 S 0.0 0.0 0:00 0 syslogd
765 root 15 0 64 4 0 S 0.0 0.0 0:00 0 klogd
775 root 15 0 244 236 180 S 0.0 0.0 0:00 1 irqbalance
792 rpc 18 0 76 4 0 S 0.0 0.0 0:00 1 portmap
811 rpcuser 25 0 80 4 0 S 0.0 0.0 0:00 1 rpc.statd
822 root 15 0 176 156 116 S 0.0 0.0 0:00 0 mdadm
914 root 15 0 260 88 52 S 0.0 0.0 0:01 0 dovecot
924 root 15 0 428 112 40 S 0.0 0.0 0:00 1 dovecot-auth
925 dovecot 15 0 292 80 40 S 0.0 0.0 0:00 1 imap-login
926 dovecot 15 0 292 80 40 S 0.0 0.0 0:00 1 imap-login
927 dovecot 15 0 288 80 40 S 0.0 0.0 0:00 1 imap-login
928 dovecot 15 0 292 80 40 S 0.0 0.0 0:00 0 pop3-login
929 dovecot 15 0 292 80 40 S 0.0 0.0 0:00 1 pop3-login
930 dovecot 15 0 292 80 40 S 0.0 0.0 0:00 1 pop3-login
933 root 15 0 692 320 228 S 0.0 0.0 0:00 0 cupsd
944 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
957 named 15 0 1792 1180 784 S 0.0 0.1 0:00 0 named
958 named 15 0 1792 1180 784 S 0.0 0.1 0:00 0 named
959 named 15 0 1792 1180 784 S 0.0 0.1 0:00 1 named
960 named 15 0 1792 1180 784 S 0.0 0.1 0:00 1 named
961 named 15 0 1792 1180 784 S 0.0 0.1 0:00 0 named
971 root 25 0 256 4 0 S 0.0 0.0 0:00 0 sshd
985 root 24 0 120 4 0 S 0.0 0.0 0:00 1 xinetd
994 clamav 25 0 12728 4 0 S 0.0 0.0 0:00 0 clamd
1006 root 25 0 128 4 0 S 0.0 0.0 0:00 0 safe_mysqld
1035 mysql 15 0 1948 1756 1196 S 0.0 0.1 0:00 1 mysqld
1036 mysql 15 0 1948 1756 1196 S 0.0 0.1 0:00 1 mysqld
1092 root 15 0 452 216 140 S 0.0 0.0 0:00 0 master
1096 postfix 15 0 396 144 88 S 0.0 0.0 0:00 0 pickup
1097 postfix 15 0 828 488 372 S 0.0 0.0 0:00 0 nqmgr
1104 nobody 15 0 656 200 124 S 0.0 0.0 0:00 0 proftpd
1114 root 25 0 52 4 0 S 0.0 0.0 0:00 0 gpm
1170 root 15 0 484 192 112 S 0.0 0.0 0:00 0 crond
1213 xfs 15 0 2240 60 32 S 0.0 0.0 0:00 1 xfs
1222 daemon 15 0 200 172 124 S 0.0 0.0 0:00 0 atd
1231 root 25 0 104 4 0 S 0.0 0.0 0:00 1 saslauthd
1235 root 25 0 104 4 0 S 0.0 0.0 0:00 1 saslauthd
1236 root 25 0 104 4 0 S 0.0 0.0 0:00 1 saslauthd
1237 root 25 0 104 4 0 S 0.0 0.0 0:00 1 saslauthd
1243 root 25 0 104 4 0 S 0.0 0.0 0:00 1 saslauthd
1283 root 19 0 56 4 0 S 0.0 0.0 0:00 1 mingetty
1284 root 19 0 56 4 0 S 0.0 0.0 0:00 0 mingetty
1285 root 19 0 56 4 0 S 0.0 0.0 0:00 1 mingetty
1286 root 19 0 52 4 0 S 0.0 0.0 0:00 0 mingetty
1287 root 19 0 48 4 0 S 0.0 0.0 0:00 0 mingetty
1288 root 19 0 52 4 0 S 0.0 0.0 0:00 1 mingetty
2944 root 15 0 7324 5460 3916 S 0.0 0.5 0:00 1 httpd
2945 root 25 0 36588 28M 436 S 0.0 2.8 0:04 0 writelogs.pl
2946 root 24 0 36584 30M 436 S 0.0 3.0 0:04 1 writelogs.pl
2949 root 25 0 36588 29M 436 S 0.0 2.9 0:04 1 writelogs.pl
2950 root 25 0 36584 30M 436 S 0.0 3.0 0:04 0 writelogs.pl
2951 root 25 0 36588 30M 436 S 0.0 2.9 0:04 0 writelogs.pl
2952 root 25 0 36584 32M 436 S 0.0 3.2 0:04 0 writelogs.pl
2953 root 25 0 36584 31M 436 S 0.0 3.1 0:04 1 writelogs.pl
2954 root 25 0 36588 30M 436 S 0.0 3.0 0:04 1 writelogs.pl
2955 root 24 0 36584 31M 436 S 0.0 3.1 0:04 0 writelogs.pl
2956 root 25 0 36584 33M 436 S 0.0 3.3 0:04 0 writelogs.pl
2957 root 25 0 36584 32M 436 S 0.0 3.2 0:04 1 writelogs.pl
2958 root 25 0 36588 32M 436 S 0.0 3.2 0:04 1 writelogs.pl
2959 root 25 0 36584 33M 436 S 0.0 3.2 0:04 0 writelogs.pl
2960 root 15 0 36780 34M 628 S 0.0 3.4 0:04 0 writelogs.pl
2963 root 15 0 36780 33M 628 S 0.0 3.3 0:04 1 writelogs.pl
2964 root 15 0 36780 34M 628 S 0.0 3.4 0:04 1 writelogs.pl
2965 root 15 0 36788 34M 636 S 0.0 3.4 0:04 1 writelogs.pl
2966 root 15 0 36776 35M 628 S 0.0 3.5 0:04 0 writelogs.pl
2976 root 15 0 36780 34M 628 S 0.0 3.4 0:04 1 writelogs.pl
2977 root 25 0 36584 34M 436 S 0.0 3.4 0:04 1 writelogs.pl
2978 root 15 0 36780 34M 628 S 0.0 3.4 0:04 1 writelogs.pl
2979 root 25 0 36588 34M 436 S 0.0 3.4 0:04 1 writelogs.pl
2992 apache 15 0 9132 8168 5252 S 0.0 0.7 0:00 0 httpd
2993 apache 15 0 9332 8368 5424 S 0.0 0.8 0:00 0 httpd
2994 apache 15 0 9332 8380 5432 S 0.0 0.8 0:00 0 httpd
2995 apache 15 0 8684 7716 5052 S 0.0 0.7 0:00 0 httpd
2996 apache 16 0 10404 9436 5292 S 0.0 0.9 0:01 1 httpd
2997 apache 15 0 8640 7692 5040 S 0.0 0.7 0:00 1 httpd
2998 apache 15 0 9280 8320 5420 S 0.0 0.8 0:00 0 httpd
2999 apache 15 0 8200 6944 4660 S 0.0 0.6 0:00 1 httpd
3542 root 15 0 6296 6296 2620 S 0.0 0.6 0:00 0 perl
4180 root 22 0 79484 77M 2892 S 0.0 7.7 0:00 0 perl
4183 root 24 0 956 956 844 S 0.0 0.0 0:00 0 sh
4184 root 25 0 880 880 716 S 0.0 0.0 0:00 0 su



With the primary concern:
Mem: 1025372k av, 943344k used, 82028k free, 0k shrd, 10656k buff

Joe,

I had a similar problem a while back. writelogs.pl processes where hogging resources and causing the system to slow to a crawl. I no longer use writelogs.pl and the problem has gone away, although i may end up with issues if users start deleting directories.

Chris

Am I safe to assume that 1Gig RAM is too minimal nowadays to run a hosting server? I only have 12 virtual servers hosted on this.

Hey Dan,

1GB is probably a bit tight for a dozen sites, but it may very well be that writelogs.pl is doing something nasty. It's not immediately apparent what, but it could happen. It's not running away (at least not obviously so), as the process size is about 30 MB in all cases on your box, with the majority of that being shared libs. If one or more were running away we'd see high and growing memory usage from one or more of them. There's just so little in writelogs.pl (have a look, it's only 32 lines, including comments and whitespace...about 25 lines of actual code), it's hard to see where a problem could sneak in, though each one does take some memory and on a box with low memory to start with it could make the difference.

Try turning off the writelogs feature and see if it makes a big difference. If it does, I'll see about converting the perl writelogs into a C process with a somewhat smaller memory footprint. Perl can be a wee bit heavy these days.