You can configure Usermin's spamassasin module to use LDAP as a data store for preferences, and virtualmin can store users in ldap. You could completely disable its spamassasin integration, and use your approach, which should work fine.

Let us know if you run into any problems though.

This setup handles mail many times faster than the procmail setup on Virtualmin Pro

I very seriously doubt it. Maybe "a few percent faster", but many times faster means you can't be running spamassassin or clamav, as that's where we all spend about 98% of our mail processing time. You can disable spamassassin and clamav in Virtualmin's procmail configuration, if you just want to deliver mail really freakin' fast. Arguing amavisd-new vs. procmail is like arguing over whether the red Ferrari is faster than the identically specced blue Ferrari. ;-)

In their daemonized form, spamassassin and clamav under procmail are going to be roughly identical in performance to calling them from amavisd-new.

So, you're almost certainly not gaining any performance at all by using amavisd-new over procmail. The slow parts are the spam and AV scanning, and lookup-domain.pl (which isn't all that slow, as it has a daemon of its own). LDAP lookups are possibly a little faster than lookup-domain.pl, depending on a few factors. But, if performance is the only benefit you see in sticking with your current configuration, I'll assure you that it's not worth the trouble. If there are other reasons, of course, I'm all for folks developing tools to manage it! (And at least you aren't trying to use MySQL for mail users, which is just silly. LDAP has some good and valid uses in a mail system.) ;-)

Also, if you're having trouble getting switched over to the daemonized form of SpamAssassin and ClamAV under Virtualmin Professional's procmail configuration...let us know. We'll be happy to help.

BTW-Virtualmin GPL 3.58, coming out probably Sunday or Monday, is getting the spam and AV configuration tools from Virtualmin Professional. So, hopefully, the proliferation of different (and often complex to migrate from) methods of processing email will converge down to one procmail solution. And the larger user base (GPL has tens of thousands of installations vs. 1500 or so Virtualmin Professional installations) should help us refine it so that it answers a wider range of user requirements.

Hmmm ... yes many times faster. I test with sending 100 emails to the Virtualmin Pro box (with attachments) and it takes about 3.5 minutes to process them all while the current amavisd server processes them in under 30 seconds. That is also with a normal mail load since it is a live mail server while the Virtualmin Pro is in testing and setup.

I am positive that the performance problem is with procmail and spamassassin/clamscan but I don't have them in daemonized form. I am new to procmail so I will take a look and see how to do that since it may solve my problem and I would rather just use the tools available rather than write new ones. Can you point to any quick doc on modifying procmail to use a daemonized form of spamassassin and clamav?

Hmmm ... yes many times faster. I test with sending 100 emails to the Virtualmin Pro box (with attachments) and it takes about 3.5 minutes to process them all while the current amavisd server processes them in under 30 seconds.

Yes, the Virtualmin procmail rules are running spamassassin and clamav are running non-daemonized on your system--they have to start up for each message. (And clamav takes forever to start.)

You just need to switch to clamdscan and spamc. You don't have to know anything about procmail--pretend like procmail doesn't even exist for this exercise--just start the services and tell Virtualmin that's the way you want to process mail. ;-)

Do the following few steps and run your benchmark again:

Start the daemons

Start the spamassassin service on your system. Depending on your OS this can probably be done easily in the Webmin:System:Bootup and Shutdown module. Just find the spamassassin or spamd service and start it, and set it to start on boot.

If you're running Debian/Ubuntu, enable clamav-daemon while you're there. If you're on a Red Hat based system using our clamav packages, the service won't exist yet so continue on to the next step.

Tell Virtualmin to use them

Back in the Virtualmin menu, open Email Messages:Spam and Virus Scanning

If you're on a Red Hat based system, click the "Enable ClamAV Server" button at the bottom of the page. This will create a clamd.virtualmin service and start it. Debian/Ubuntu users continue on to the next step, doing nothing here.

Set "SpamAssassin client program" to "spamc (client for spamd server"

Set "Virus scanning program" to "Server scanner (clamdscan)"

Save it.

Done.


Now you should find that Virtualmin's procmail configuration takes about the same amount of time as your amavisd-new configuration.

I'll try to get this stuff turned on, by default, in the next version of the virtualmin-base script...it's a common misconception that procmail is slow, when in fact, procmail is faster than amavisd-new, and, far more importantly, it just doesn't matter because mail delivery is so little of the work required when processing mail--even if procmail were ten times slower than amavisd-new (which it isn't...it's faster) it would still only account for about a second of the mail delivery process. Performance of procmail and amavisd-new are simply not a factor you should consider in your plans, because the performance of either is dwarfed by the expense of processing mail for filtering viruses and spam.

If we want faster mail delivery, we have to figure out ways to make spamassassin and clamav faster (or make them work less...we plan to introduce a policy daemon soon that will bounce obvious spams earlier in the process and more cheaply, so they don't have to go through the more complex tests).

Well, if I had only known that I could do this with a few check boxes, I wouldn't have done most of it on the command line already ;-)

I am running CentOS5.1 but the option to create the clamd settings doesn't seem to show up. I already did this part anyway but I see that clamdscan cannot find the configuration file unless I symlink /etc/clamd.d/virtualmin.conf to /etc/clamd.conf (the option for --config-file is not in the Virtualmin screen).

After reconfiguring and testing the same email is processed in under 30 seconds so this does the trick, thank you.

The pieces are there and moving but it looks like lookup-domain.pl is not finding the users in LDAP. I am getting flooded with SPAM now and looking at the procmail.log, it appears that the user's procmail settings are not being run because they can't be found. I tried running lookup-domain.pl on the command line but it doesn't return anything. I must still be missing something.

procmail: Assigning "TRAP=/etc/webmin/virtual-server/procmail-logger.pl"
procmail: Executing "/etc/webmin/virtual-server/lookup-domain.pl,user-domain.tld"
procmail: Assigning "VIRTUALMIN="
procmail: [16592] Mon Jun 9 11:09:58 2008
procmail: Executing "/usr/bin/test,,!=,"
procmail: [16592] Mon Jun 9 11:09:58 2008
procmail: Non-zero exitcode (1) from "/usr/bin/test"
procmail: No match on "/usr/bin/test != "

I changed the user.domain.tld but the actual user really exists in LDAP.

If you run the command :
/etc/webmin/virtual-server/lookup-domain.pl user-domain.tld </dev/null
does it output a domain ID?

If not, you may need to restart lookup-domain-daemon.pl, with the command :
/etc/init.d/lookup-domain restart

A restart of domain-lookup seems to have fixed this.