We have a Virtualmin setup with 400 websites(domains). When weekly logrotate initiates, the server freezes for half an hour. Heavy I/O and non responsive services.
Why this happens: The built-in logrotate module of Virtualmin creates one logrotate configuration file for every domain in /etc/logrotate.d/. This script tells the logrotate script to rotate the error and access log for that particular domain, then make a graceful restart of httpd wait 5 seconds, than continue. As result when logrotate initiates we have 400 graceful restarts of Apache, one every 5 seconds. This not only makes Apache non accessible but also create huge I/O eventually blocking all other services too.
Here are other people mentioning the same:
What I did to resolve that is to completely disable the built-in logrotate Virtualmin module, remove all the per-domain logrotate configuration files and create a single logrotate configuration file to rotate all the domains' logs at once:
/sbin/service httpd reload > /dev/null 2>/dev/null || true
This solution works as expected - we have only one graceful restart after all the domains logs are rotated.
I have few questions though:
- Why you choose to create per-domain logrotate configuration at first place? There are a lot of arguments against that:
- All the domain log files go in /var/log/virtualmin it is very easy to create one wildcard logrotate configuration for all of them - There are no per-domain logrotate settings, there are though per plan settings (you can customize logrotate directives per plan), but it does not make much sense, basically all the domains logrotate config files are the same, except the filename - Even though before the graceful restart Apache keeps filling the old log file and this might take few seconds to complete for all domains, this is not a big deal if using delaycompress, which keeps the first archived file uncomressed. Basically Apache fill confitue writing to the old file for few more seconds, but no messages will be lost.
Will I run into some incompatibility/restrictions if I keep the logrotate Virtualmin module disabled? Is it only making these per-domain logrotate configurations or it is also responsible for other operations too? IS something depending on the logrotate module?
If you wish to keep per domain config files you can remove the graceful restart from the per domain configs, and add it to the main config after all of them. this will achieve the same. Unfortunately the interface does not allow to create custom logrotate directives if there is no postrotate directive (presuming graceful restart is mandatory there).
Let me know what you think!