bw.pl slow down server

17 posts / 0 new
Last post
#1 Wed, 04/15/2009 - 08:00
PaulVM

bw.pl slow down server

Tried on: CentOS 5.1/5.2/5.3 (basically installed from somethinng like http://www.howtoforge.org/perfect-server-centos-5.2-x86_64 with personalizations) Webmin 1.4.x --> 1.4.70 Virtualmin GPL 3.58 --> 3.67

I have some vhosts configured. Total traffic is some GB/day (from 5 to 30). Every hour I have the cron that start the scheduled: 0 * * * * /etc/webmin/bandwidth/rotate.pl 0 * * * * /etc/webmin/virtual-server/bw.pl

I noticied that bw.pl is very cpu-intensive (around 100% in a dual core). And that it require many minutes (yes minutes), to finish execution. More traffic (GB, log), more minutes (from 2 to 20).

Executing it from command line give me no more info. A part annoyng perl warning that I dont' think relevant (reported below, appreciated hints to eliminate ...). My fault? What can I check? Can I schedule to run only 1-2 time/day (nightly)?

Regards, P.

/etc/webmin/virtual-server/bw.pl

perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LANGUAGE = (unset), LC_ALL = (unset), LANG = "eng_US.UTF-8" are supported and installed on your system. perl: warning: Falling back to the standard locale ("C"). perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LANGUAGE = (unset), LC_ALL = (unset), LANG = "eng_US.UTF-8" are supported and installed on your system. perl: warning: Falling back to the standard locale ("C").

Wed, 04/15/2009 - 08:10
andreychek

Yeah, bw.pl reads through all the logs and calculates current bandwidth usage. If it doesn't matter to you that bandwidth usage is calculated once an hour, then it's perfectly fine to change when bw.pl runs. That means if a Virtual Server uses more bandwidth then they are allocated, Virtualmin won't notice until the next time bw.pl runs.

I did that on my server though -- I have it run in the morning, just after business hours, and at midnight.
-Eric

Wed, 04/15/2009 - 08:35 (Reply to #2)
PaulVM

Ok, this can be a workaround, but my doubt still is: is this slowness normal or I have some misconfigurations around my servers?

And, the:
0 * * * * /etc/webmin/bandwidth/rotate.pl

Can also be rescheduled?

Regards, P.

Thu, 04/16/2009 - 02:32 (Reply to #3)
PaulVM

I don't care about bandwidth usage, but I like reports (to see situation).
My idea was that if this is the "standard situation" (not a my misconfiguration), the bw.pl have some problem if its performance are so poor. I compare it with awstats that do a similar work (same files parsed and more processing done), but is about 10 times faster. And it is also perl code ... ;-)

I am not a perl-coder, and I can only thank Jamie & c. for the work they do. Consider this as a hint ;-)

Regards, P.

Thu, 04/16/2009 - 11:17 (Reply to #4)
Joe
Joe's picture

<div class='quote'>I compare it with awstats that do a similar work (same files parsed and more processing done)</div>

Incorrect. If you only wanted web usage you wouldn't need bw.pl at all, you could use awstats or webalizer. awstats only parses web logs. bw.pl parses the <i>much</i> larger iptables network packet log. awstats parses one domains access_log. Know how many packets are sent and received in the average web pageview? Hundreds. How about an ssh session? Every character you type is a packet. So, bw.pl is processing hundreds or thousands of times the lines of log data of awstats.

Could it be faster? As with all complex code that hasn't seen heavy optimization work, probably. But, it's a lot of data to be processed, no matter what. It will always require a lot of work from the system. We've started doing profiling recently, using Devel::NYTProf, so if there are any stupidities in the code that are taking longer than they should they will show up. Now that the new module loading code is rolled out (which improved performance and memory usage of Webmin+Virtualmin dramatically), we can focus on other things...the cronjobs, collectinfo.pl in particular but bw.pl too, will get some attention.

The smarter optimization is probably going to be to figure out a way to accurately get network data with protocol usage and such without having to log it line-per-packet via iptables. The standard network tools on Linux keep packet counts for free, but not packet sizes or protocols, so it's not useful data in this context. I'm unaware of any method to gather that data without packet logging (either via iptables or a sniffer like wireshark). Another possibility might be to have bw.pl running all the time, processing constantly at a low rate (using, say 1% or .5% of the CPU and a little bit of RAM). No big surges of activity...but you'd always have it running (which seems to tick people off for some reason; I'm not sure why long-running processes, even with niceness of 19, annoy people, but they do).

--

Check out the forum guidelines!

Thu, 04/16/2009 - 13:09 (Reply to #5)
PaulVM

<div class='quote'>Incorrect. If you only wanted web usage you wouldn't need bw.pl at all, you could use awstats or webalizer. awstats only parses web logs. bw.pl parses the much larger iptables network packet log. awstats parses one domains access_log. Know how many packets are sent and received in the average web pageview? ...</div>

I am a newbie in virtualmin, but not so newbie in servers admin ;-)
This boxes where I am testing virtualmin, are simply LAMP server (in a DMZ). Only http and ftp protocol published. SSH only for server administration. Mail limited to httpd post and service messages (few hundred/day)
No or few iptables rules. So, iptables cannot be the cause of the low performance of bw.pl ;-)
Really, until now, I don't know that webmin/virtualmin can be used for trafic/protocol analizer. Which module, option?

Initially I have considered reverse resolution of httpd log as a cause of slowness (typical in weblog analizers), but I haven't found any place where this can be done.

So I have considered a my mistake in some configuration.

Regards, P.

Thu, 04/16/2009 - 13:50 (Reply to #6)
Joe
Joe's picture

Oops. I'm talking about the Webmin Bandwidth monitoring module, which uses packet logging.

Virtualmin's bandwidth monitoring is just adding up all of the logs from web, mail, FTP, etc.. I'll look into it.

--

Check out the forum guidelines!

Fri, 04/24/2009 - 13:34 (Reply to #7)
PaulVM

Any progress? ;-)
Am I the only that have this &quot;problem&quot;?

Regards, P.

Wed, 05/06/2009 - 13:19 (Reply to #8)
Joe
Joe's picture

<div class='quote'>Am I the only that have this &quot;problem&quot;?</div>

You might be. Are all the logs being rotated properly? If cron isn't running or something is breaking the rotations, it could be processing huge files.

I haven't spent any time on bw.pl yet, as I'm still working on getting the new website launched,

--

Check out the forum guidelines!

Thu, 05/07/2009 - 04:22 (Reply to #9)
PaulVM

Thanks for replay.
We can resume this topic when less busy.
At the moment i only report my experiences:

Logs are rotated monthly (low traffic servers).
I noticied degradation from beginning to end of month, but as previously reported, awstats that parses the same files is almost 10 times faster than bw.pl (awstats installed stand-alone, not managed by virtualmin).
In the near future I try using a full virtualmin installed server (now we use virtualmin as simple module of webmin).

Regards, P.

Thu, 05/07/2009 - 07:18 (Reply to #10)
Joe
Joe's picture

Ah, OK, so the default in Virtualmin is to rotate weekly (I'm pretty sure). So, you are seeing dramatically more processing than most folks.

But, as I mentioned, we have begun profiling with NYTProf. If there are serious performance issues, they will be resolved within the next couple of revisions. We're in the midst of launching the new website and our cloud computing management product, which are both bigger pain points for a larger bunch of users (almost everybody in the case of the current website--you can't help but run into problems using it for more than a few minutes).

--

Check out the forum guidelines!

Wed, 04/15/2009 - 09:55
Joe
Joe's picture

As Eric mentioned bw.pl is very resource intensive, if you have a lot of network traffic. There's no way to avoid it, since it's processing network packet logs, which are <i>huge</i>. You can certainly run it less often--though keep in mind that the less often you run it, the larger the logs will be, and so it will take even longer to process. But, it should be running at a low priority (a high niceness level), and so it shouldn't seriously adversely impact service.

If you don't care about bandwidth usage reports and restricting users bandwidth, you could also just turn it off.

--

Check out the forum guidelines!

Wed, 05/06/2009 - 05:55
PaulVM

Still no news?

Thanks, P.

Tue, 06/23/2009 - 05:20
RedKnot

I'm also experiencing high loads when bw.pl is running.

Only bandwidth for http and ftp is shown in the graphs. I did configure the module to also scan /var/log/maillog (postfix log file). Shouldn't the bandwidth monitor also scan imap and pop3 log files?

Thanks, Martijn

Thu, 06/25/2009 - 09:19 (Reply to #14)
andreychek

Yeah, email should certainly be included in that (and by default, it's normally able to detect the location of the mail log).

Also, on most systems, imap and pop3 information goes into the maillog, though I don't know if that's used to further calculate stats or not.

-Eric

Thu, 06/25/2009 - 10:17 (Reply to #15)
RedKnot

Oke then there seems to be a problem? The maillog file does include the size of the messages:

Jun 25 04:19:32 stevie.youngguns.nl postfix/qmgr[23660]: [ID 197553 mail.info] A9DB02CC00: from=xxxxxx@xxxxxxxxxxxx.xxx, size=29772, nrcpt=1 (queue active)

If imap and pop3 is calculated then it should be possible to have a seperate logfile for it. I have a dovecot.log file for it.

Martijn

Sat, 08/21/2010 - 07:31
kenlyle

Trying to reduce load...I've turned off Apache logging, and awstats processing. Looking at killing bw.pl as well, or at least upping my log rotation, per Joe's tip.