Over the weekend the load average on my server leaped from an average of 0.5 to 15, and got pinned there. The culprit was miniserv.pl. A dozen instances of this were running at 100% cpu usage. I changed the user password and killed all their instances of miniserv.pl, and life went back to normal for the server. Even during the peak of the problem, the server was still able to deliver email and web content without any noticable slowness.
It turns out that the user left an instance of webmail running on a computer the kid was playing on, and then tried to launch multiple usermin windows from another computer, which were not letting him in. Most of these were running at almost 100% cpu capacity. He was on a MacIntosh logging in with Safari. Any reason why these were running at such a high CPU utilization? Such a comedy of errors, I was on the server at the same time killing his windows, thinking I was under a DOS attack, thinking what a silly way for someone to try tying up my server!
Well, there's also certain actions that can take place within Webmin that are CPU intensive. Looking through logs I think is one of them, amongst others.
You might consider looking at the logs in /var/webmin/ and /var/usermin/ to see what was going on around the time the load spiked.
As far as brute force attacks go -- you might want to take a peek in Webmin -> Webmin -> Webmin Configuration -> Authorization. In there, you can have it block hosts with more than N failed login attempts for some timeframe.
I think that's on by default, but you could increase the time the host is blocked for added security.
-Eric
well, it happened again. I can make it happen any time by simply logging into the client account and clicking on the spam folder in usermin. The browser then hangs, and the process miniserv.pl is running at 100% cpu utilization. I found many instances running, caused by the client trying to get to his spam folder to check for false positives. From a 'top' run on my server, before I killed the processes:
top - 09:17:17 up 39 days, 14:28, 1 user, load average: 8.58, 8.34, 8.19
Tasks: 154 total, 11 running, 143 sleeping, 0 stopped, 0 zombie
Cpu(s): 95.9%us, 3.2%sy, 1.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 5019864k total, 4794216k used, 225648k free, 390572k buffers
Swap: 489940k total, 916k used, 489024k free, 3589884k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11633 blauchuc 20 0 17684 14m 2348 R 28 0.3 201:35.03 miniserv.pl
19771 blauchuc 20 0 17556 14m 2348 R 28 0.3 267:48.03 miniserv.pl
18935 blauchuc 20 0 17548 14m 2368 R 27 0.3 270:04.74 miniserv.pl
25573 blauchuc 20 0 17560 14m 2348 R 26 0.3 177:13.33 miniserv.pl
8859 blauchuc 20 0 17592 14m 2344 R 26 0.3 460:10.17 miniserv.pl
11620 blauchuc 20 0 17580 14m 2372 R 26 0.3 201:38.67 miniserv.pl
6921 blauchuc 20 0 17540 14m 2372 R 26 0.3 467:35.37 miniserv.pl
26522 m3rginge 20 0 17588 14m 2316 R 8 0.3 0:00.25 miniserv.pl
26141 munin 30 10 8016 6084 2416 R 3 0.1 0:03.39 munin-graph
26519 administ 20 0 2436 1156 856 R 0 0.0 0:00.04 top
26524 m3rginge 20 0 2784 1380 1016 S 0 0.0 0:00.01 imap
1 root 20 0 2844 1692 548 S 0 0.0 0:15.53 init
Can you shed some wisdom on what is going on here?
Thanks,
Don
How large is the spam folder in question? Also, which format is it in .. a single mbox file, or a Maildir directory?
''
results from du:
du -h /home/blauappraisal.com/homes/blauchuck/Maildir 32K /home/blauappraisal.com/homes/blauchuck/Maildir/new 4.0K /home/blauappraisal.com/homes/blauchuck/Maildir/tmp 4.0K /home/blauappraisal.com/homes/blauchuck/Maildir/.Drafts/new 4.0K /home/blauappraisal.com/homes/blauchuck/Maildir/.Drafts/tmp 4.0K /home/blauappraisal.com/homes/blauchuck/Maildir/.Drafts/cur 44K /home/blauappraisal.com/homes/blauchuck/Maildir/.Drafts 12K /home/blauappraisal.com/homes/blauchuck/Maildir/.spam/new 4.0K /home/blauappraisal.com/homes/blauchuck/Maildir/.spam/tmp 224K /home/blauappraisal.com/homes/blauchuck/Maildir/.spam/cur 360K /home/blauappraisal.com/homes/blauchuck/Maildir/.spam 4.0K /home/blauappraisal.com/homes/blauchuck/Maildir/.Sent/new 4.0K /home/blauappraisal.com/homes/blauchuck/Maildir/.Sent/tmp 79M /home/blauappraisal.com/homes/blauchuck/Maildir/.Sent/cur 80M /home/blauappraisal.com/homes/blauchuck/Maildir/.Sent 30M /home/blauappraisal.com/homes/blauchuck/Maildir/cur 4.0K /home/blauappraisal.com/homes/blauchuck/Maildir/.Trash/new 4.0K /home/blauappraisal.com/homes/blauchuck/Maildir/.Trash/tmp 4.0K /home/blauappraisal.com/homes/blauchuck/Maildir/.Trash/cur 36K /home/blauappraisal.com/homes/blauchuck/Maildir/.Trash 111M /home/blauappraisal.com/homes/blauchuck/Maildir
He has a lot of sent items that were migrated along with his email account from the old server. I will be using your interface to put a 'delete after so many days' on the sent items folder. The spam was not migrated, of course.
So the spam folder is pretty small, and shouldn't cause Webmin to consume 100% CPU while opening it.
One quick fix would be to just delete this spam folder..
''
Jamie, The reason I have not deleted it already is in case you needed anything for analysis. There seems to be a spam out there that can give your code a problem. If my end users get bombed with this spam, I could have a potential disaster on my hands if several of them log into usermin and try to check their spam folder and many shells get launched, each hogging 100% CPU. My system would bog down to a halt eventually. Have you ever seen anything like this before, or should I just delete the folder and stick my head in the sand, hoping that it does not happen again. BTW Jaime, virtualmin RULES! Don't think I'm not grateful. When we upgraded to the paid version, and we saw how many script installers there were, the staff here went nuts over it.
Thanks for keeping that spam file - this may be a Virtualmin bug triggered by mail parsing.
If you re-name the folder to something else and then try to open it, does the same thing happen?
''
Jaime, I saved a copy of the spam folder for you if you need it, deleted it from the user account, reinitialized it with the gtube string, and logged into the usermin interface as the user and checked the spam folder. Everything worked just fine. - Don
If you could email me a tar file of the spam folder at jcameron@virtualmin.com , I could take a look and see if there is something in one of the messages that is making Usermin hang ..
''
HI all,
I had similar slowness issue. Been puzzling for a few day.
What I changed that improve the situation is --
I changed the usermin preference (login to usermin at http://localhost:2000/) and in Configuration category | Show unread messages count Yes Only for IMAP No <-- I set this to No.
I had one folder that is about 3GB -- which causes the cpu + io wait to go off the roof all the time.
your situation may be different, but this is what fixes mine :)
cheers.