Hacked via WordPress

The last 24 hours has been a huge misery. One of our server was compromised, it looks like two WordPress sites were hacked, not sure exactly how. We have been working around the clock. The sites started to create 10's of thousands of email files, though they were not successfully sent. In the end after hours of battling the hack we only got it to stop by completely deleting Dovecot. Now that we have reinstalled it the few legit emails in Postfix spool folder are going nowhere because it can't find a mail transport. Any suggestions on how to get it working again. In the years we have used the control panel this is the worst thing that has ever happened to us. We deleted the two sites and when they are restored they immediately start to create email files again. Unfortunately, while there are WordPress plugins that scan for viruses, they all seem to depend on WordPress running and we cannot afford to let them run, not even for a short time. We wish there was some way to scan for files without WordPress running. What a mess. There are just too many files in a WordPress site to find out where the hack is and we have not really had to before now.

Status: 
Active

Comments

Howdy -- what is the exact error you receive in the email logs when trying to send an email?

Also, if you restart Postfix, do any errors show up in the logs when it's starting back up?

As far as WordPress goes -- our suggestion there is to make sure it's fully up to date, that the plugins are all fully up to date, and you may want to change the passwords for any WordPress users.

To prevent emails from being sent, you can always just disable Postfix. If Postfix isn't running, your server won't send out any emails from the mail queue.

However, you could try scanning your WordPress files with the Linux Malware Detect tool, which can pick up some web-based malware:

https://www.rfxn.com/projects/linux-malware-detect/

Here is the return when trying to restart Postfix, which it appears to do but just won't send any email. Jul 6 18:52:47 raven postfix/master[23347]: warning: /usr/libexec/postfix/smtp: bad command startup -- throttling

We are pretty careful with WordPress and always use WordFence, which in this case did not help. Thanks for the suggestion of the malware scan. It found 31 effect files. Unfortunately, it did not clean anything. So hopefully I might actually figure out how to get it to clean up the files. But at least we know which files are infected. For example, here is a snippet;

{CAV}PHP.Trojan.Uploader : rennerusa/public_html/wp-includes/js/tinymce/skins/lightgray/img/files.php => /usr/local/maldetect/quarantine/files.php.14347 {CAV}Php.Malware.Mailbot-1 : rennerusa/public_html/wp-includes/js/tinymce/skins/user.php => /usr/local/maldetect/quarantine/user.php.2142 {CAV}Php.Trojan.StopPost : rennerusa/public_html/wp-includes/images/smilies/list.php => /usr/local/maldetect/quarantine/list.php.29468 {CAV}Php.Trojan.StopPost : rennerusa/public_html/wp-includes/SimplePie/Cache/list.php => /usr/local/maldetect/quarantine/list.php.27506 {HEX}php.cmdshell.unclassed.357 : rennerusa/public_html/wp-includes/pomo/cache_checkexpresses.php => /usr/local/maldetect/quarantine/cache_checkexpresses.php.14332 {CAV}PHP.Trojan.Uploader : rennerusa/public_html/includes/fresh/freshCore/canvas/fresh/external/themes.php => /usr/local/maldetect/quarantine/themes.php.10110 {HEX}php.cmdshell.unclassed.357 : rennerusa/public_html/wp-content/plugins/E4nKzzZl.php => /usr/local/maldetect/quarantine/E4nKzzZl.php.27014 {HEX}php.cmdshell.unclassed.357 : rennerusa/public_html/wp-content/plugins/Y4aU3y7.php => /usr/local/maldetect/quarantine/Y4aU3y7.php.2311

If you're seeing a "bad command startup" error, it should be generating another error with that showing what the bad command is... are you seeing other errors in the logs around the time Postfix is started?

I use maldet mostly to help me identify malware -- it's not always perfect, so I typically prefer to manually inspect anything it finds, rather than have it attempt to clean it for me.

Unfortunately, there are form mail piling up on the server, customers quite upset at me (I don't blame them) and if I open up some of the files being generated at /var/spool/postfix/defer, they all say //// action=delayed - reason=mail transport unavailable //// I thought Dovecot was the one to relay for Postfix. Since it was uninstalled after a mail relay hack, it no longer is working. I am not even sure where to tell Dovecot where it should be looking to see if there is mail. The defaults don't seem right but what do I know. We have had issues over the years but this is the first time I felt like I was way in over my head. Any suggestions on the meaning of /// mail transport unavailable/// or even to your knowledge what is the default setup with Virtualmin and mail?

Dovecot isn't involved in delivering email, only Postfix is.

The question I asked in comment #3 above is still the key -- if you're seeing a "bad command startup" error, Postfix should be generating another error along with that showing what the bad command is... are you seeing other errors in the logs around the time Postfix is started?

It looks like a configuration error. Now, if I can find out where it specifies the mailbox size. //////////////// Jul 12 03:22:10 raven postfix/local[23878]: fatal: main.cf configuration error: mailbox_size_limit is smaller than message_size_limit Jul 12 03:22:11 raven postfix/master[1098]: warning: process /usr/libexec/postfix/local pid 23878 exit status 1 Jul 12 03:22:11 raven postfix/master[1098]: warning: /usr/libexec/postfix/local: bad command startup -- throttling Jul 12 03:23:11 raven postfix/local[24130]: fatal: main.cf configuration error: mailbox_size_limit is smaller than message_size_limit Jul 12 03:23:12 raven postfix/master[1098]: warning: process /usr/libexec/postfix/local pid 24130 exit status 1 Jul 12 03:23:12 raven postfix/master[1098]: warning: /usr/libexec/postfix/local: bad command startup -- throttling

Agreed, this looks like the error here:

fatal: main.cf configuration error: mailbox_size_limit is smaller than message_size_limit

That is configured in /etc/postfix/main.cf.

I configured it to a value of 10000 and restarted Postfix. Still get the same thing. Back to the drawing board I guess. There must be something else out of wack as well. There is a reference to smtp bad command startup..... ////////////// Jul 14 10:14:01 raven postfix/local[3596]: fatal: main.cf configuration error: mailbox_size_limit is smaller than message_size_limit Jul 14 10:14:01 raven postfix/smtp[3595]: fatal: unexpected command-line argument: When Jul 14 10:14:02 raven postfix/master[3458]: warning: process /usr/libexec/postfix/local pid 3596 exit status 1 Jul 14 10:14:02 raven postfix/master[3458]: warning: /usr/libexec/postfix/local: bad command startup -- throttling Jul 14 10:14:02 raven postfix/master[3458]: warning: process /usr/libexec/postfix/smtp pid 3595 exit status 1 Jul 14 10:14:02 raven postfix/master[3458]: warning: /usr/libexec/postfix/smtp: bad command startup -- throttling Jul 14 10:14:02 raven postfix/qmgr[3460]: warning: private/smtp socket: malformed response

What is the output of this command:

postconf -n

[root@raven ~]# postconf -n alias_database = hash:/etc/aliases alias_maps = hash:/etc/aliases allow_percent_hack = no bounce_size_limit = 5 broken_sasl_auth_clients = yes command_directory = /usr/sbin config_directory = /etc/postfix daemon_directory = /usr/libexec/postfix data_directory = /var/lib/postfix debug_peer_level = 2 default_destination_concurrency_limit = 1 default_destination_recipient_limit = 1 default_process_limit = 1 deliver_lock_attempts = 10 fork_attempts = 1 home_mailbox = Maildir/ html_directory = no inet_interfaces = all inet_protocols = all initial_destination_concurrency = 2 mail_owner = postfix mailbox_command = /usr/bin/procmail-wrapper -o -a $DOMAIN -d $LOGNAME mailbox_size_limit = 10000 mailq_path = /usr/bin/mailq.postfix manpage_directory = /usr/share/man maximal_queue_lifetime = 2d minimal_backoff_time = 900s mydestination = $myhostname, localhost.$mydomain, localhost, raven.securewebs.com newaliases_path = /usr/bin/newaliases.postfix queue_directory = /var/spool/postfix readme_directory = /usr/share/doc/postfix-2.6.6/README_FILES sample_directory = /usr/share/doc/postfix-2.6.6/samples sender_bcc_maps = hash:/etc/postfix/bcc sendmail_path = /usr/sbin/sendmail.postfix setgid_group = postdrop smtpd_recipient_limit = 20 smtpd_recipient_restrictions = permit_mynetworks permit_sasl_authenticated reject_unauth_destination reject_unknown_reverse_client_hostname permit_inet_interfaces unknown_local_recipient_reject_code = 550 virtual_alias_maps = hash:/etc/postfix/virtual

Hmm, as a quick fix, you could try setting this in your main.cf:

mailbox_size_limit = 0

That would set the mailbox size to unlimited, which should fix that error.

Hi: Good idea. Still have 94 messages in the spool. Still the logs say the mail transport is unavailable. Sent test message and the fatal error as seen below in the logfile is unexpected command-line argument: When. Not sure what "When" means yet. Another configuration error?

-------------Jul 14 11:22:35 raven postfix/pickup[15841]: 5BC068235B: uid=567 from= Jul 14 11:22:35 raven postfix/cleanup[17654]: 5BC068235B: message-id=5a690ee6080f4fc3ccc66914984adbd2@securewebs.com Jul 14 11:22:35 raven postfix/qmgr[15842]: 5BC068235B: from=wp@raven.securewebs.com, size=997, nrcpt=1 (queue active) Jul 14 11:22:35 raven postfix/smtp[17656]: fatal: unexpected command-line argument: When Jul 14 11:22:36 raven postfix/qmgr[15842]: warning: private/smtp socket: malformed response Jul 14 11:22:36 raven postfix/master[15840]: warning: process /usr/libexec/postfix/smtp pid 17656 exit status 1 Jul 14 11:22:36 raven postfix/master[15840]: warning: /usr/libexec/postfix/smtp: bad command startup -- throttling Jul 14 11:22:36 raven postfix/error[17657]: 5BC068235B: to=scott@securewebs.com, relay=none, delay=1, delays=0.01/1/0/0.01, dsn=4.3.0, status=deferred (mail transport unavailable)

That is an odd one! Can you attach your /etc/postfix/master.cf file?

Sure. Suggestions very very welcome. Please note I added .zip to the file name so that it would send.

Sorry I don't see it attached... you could always just copy and paste the contents though. Or provide a link to where I can view it elsewhere.

Oh, before I forget -- I did want to mention for future reference, that there's never a reason to uninstall software.

It sucks when spammers break into a site, and hopefully it doesn't happen again... but it if does, you can stop emails from being sent out by stopping Postfix.

You can do that with this command:

/etc/init.d/postfix stop

Email can't enter or exit the email queue if Postfix isn't running.

At that point, you could resolve the problem, and then once it's corrected, you can then restart Postfix.

Removing software can cause quite a lot of problems, including needing to manually redo the configuration for various services that are configured at Virtualmin's installation time.

Sorry, something went way wrong with the format. - Scott

That's okay, I just wrapped it in "code" tags.

That looks like your main.cf file though, rather than the master.cf.

Can you paste in your master.cf file? Thanks!

Latest from logfiles. Can't figure out where to change the bounce number. Good grief.

Jul 14 12:05:03 raven postfix/qmgr[25191]: warning: private/smtp socket: malformed response Jul 14 12:05:03 raven postfix/master[25181]: warning: process /usr/libexec/postfix/smtp pid 25196 exit status 1 Jul 14 12:05:03 raven postfix/master[25181]: warning: /usr/libexec/postfix/smtp: bad command startup -- throttling Jul 14 12:05:03 raven postfix/qmgr[25191]: warning: private/smtp socket: malformed response Jul 14 12:05:03 raven postfix/master[25181]: warning: process /usr/libexec/postfix/smtp pid 25199 exit status 1 Jul 14 12:05:03 raven postfix/bounce[25627]: fatal: invalid bounce_size_limit parameter value 0 < 1 Jul 14 12:05:04 raven postfix/master[25181]: warning: process /usr/libexec/postfix/bounce pid 25627 exit status 1 Jul 14 12:05:04 raven postfix/master[25181]: warning: /usr/libexec/postfix/bounce: bad command startup -- throttling

#
# Postfix master process configuration file.  For details on the format
# of the file, see the master(5) manual page (command: "man 5 master").
#
# Do not forget to execute "postfix reload" after editing this file.
#
# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
smtp inet n - n - - smtpd -o smtpd_sasl_auth_enable=yes
#submission inet n       -       n       -       -       smtpd
#  -o smtpd_tls_security_level=encrypt
#  -o smtpd_sasl_auth_enable=yes
#  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
#  -o milter_macro_daemon_name=ORIGINATING
#smtps     inet  n       -       n       -       -       smtpd
#  -o smtpd_tls_wrappermode=yes
#  -o smtpd_sasl_auth_enable=yes
#  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
#  -o milter_macro_daemon_name=ORIGINATING
#628      inet  n       -       n       -       -       qmqpd
pickup    fifo  n       -       n       60      1       pickup
cleanup   unix  n       -       n       -       0       cleanup
qmgr      fifo  n       -       n       300     1       qmgr
#qmgr     fifo  n       -       n       300     1       oqmgr
tlsmgr    unix  -       -       n       1000?   1       tlsmgr
rewrite   unix  -       -       n       -       -       trivial-rewrite
bounce    unix  -       -       n       -       0       bounce
defer     unix  -       -       n       -       0       bounce
trace     unix  -       -       n       -       0       bounce
verify    unix  -       -       n       -       1       verify
flush     unix  n       -       n       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp unix - - n - - smtp When relaying mail as backup MX, disable fallback_relay to avoid MX loops
relay     unix  -       -       n       -       -       smtp
-o smtp_fallback_relay=
#       -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq     unix  n       -       n       -       -       showq
error     unix  -       -       n       -       -       error
retry     unix  -       -       n       -       -       error
discard   unix  -       -       n       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       n       -       -       lmtp
anvil     unix  -       -       n       -       1       anvil
scache    unix  -       -       n       -       1       scache
#
# ====================================================================
# Interfaces to non-Postfix software. Be sure to examine the manual
# pages of the non-Postfix software to find out what options it wants.
#
# Many of the following services use the Postfix pipe(8) delivery
# agent.  See the pipe(8) man page for information about ${recipient}
# and other message envelope options.
# ====================================================================
#
# maildrop. See the Postfix MAILDROP_README file for details.
# Also specify in main.cf: maildrop_destination_recipient_limit=1
#
#maildrop  unix  -       n       n       -       -       pipe
#  flags=DRhu user=vmail argv=/usr/local/bin/maildrop -d ${recipient}
#
# ====================================================================
#
# The Cyrus deliver program has changed incompatibly, multiple times.
#
#old-cyrus unix  -       n       n       -       -       pipe
#  flags=R user=cyrus argv=/usr/lib/cyrus-imapd/deliver -e -m ${extension} ${user}
#
# ====================================================================
#
# Cyrus 2.1.5 (Amos Gouaux)
# Also specify in main.cf: cyrus_destination_recipient_limit=1
#
#cyrus     unix  -       n       n       -       -       pipe
#  user=cyrus argv=/usr/lib/cyrus-imapd/deliver -e -r ${sender} -m ${extension} ${user}
#
# ====================================================================
#
# See the Postfix UUCP_README file for configuration details.
#
#uucp      unix  -       n       n       -       -       pipe
#  flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
#
# ====================================================================
#
# Other external delivery methods.
#
#ifmail    unix  -       n       n       -       -       pipe
#  flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
#
#bsmtp     unix  -       n       n       -       -       pipe
#  flags=Fq. user=bsmtp argv=/usr/local/sbin/bsmtp -f $sender $nexthop $recipient
#
#scalemail-backend unix -       n       n       -       2       pipe
#  flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store
#  ${nexthop} ${user} ${extension}
#
#mailman   unix  -       n       n       -       -       pipe
#  flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py
#  ${nexthop} ${user}
submission inet n - n - - smtpd -o smtpd_sasl_auth_enable=yes

Ah, I see one problem that jumps out to me -- this line here:

smtp unix - - n - - smtp When relaying mail as backup MX, disable fallback_relay to avoid MX loops

That isn't valid syntax.

Try commenting that line out (the one below the line beginning with "proxywrite" and above the line beginning with "relay").

Once you do that, restart Postfix, and let's see how that works.

OK, I did that restarted and the logfile has as the last entry: Jul 14 12:18:38 raven postfix/bounce[28333]: fatal: invalid bounce_size_limit parameter value 0 < 1 Jul 14 12:18:39 raven postfix/master[28327]: warning: process /usr/libexec/postfix/bounce pid 28333 exit status 1 Jul 14 12:18:39 raven postfix/master[28327]: warning: /usr/libexec/postfix/bounce: bad command startup -- throttling

It looks like this line is setup in your main.cf file:

bounce_size_limit = 0

It looks like it was manually added to the end there. The error you're receiving says that the value "0" is invalid for that parameter.

Try commenting that line out and restart Postfix.

Did that. The log file now says: Jul 14 12:28:30 raven postfix/master[30574]: warning: process /usr/libexec/postfix/bounce pid 30581 exit status 1 Jul 14 12:28:30 raven postfix/master[30574]: warning: /usr/libexec/postfix/bounce: bad command startup -- throttling

Spool is still stuck. Crap. This one is a pain. Sorry about that.

I suppose I could reinstall postfix but the only problem is the 96 messages. Or at least that is the first problem that comes to mind.

Okay, we're making progress!

Are you seeing any other errors when Postfix is restarted?

It looks like there should be an additional error either before or after the ones you mentioned above there... those are the generic errors, there should be a more specific one along with them.

Sorry I may have missed something, what is it with those 96 messages? Are you trying to keep them from going out?

You may be able to get rid of those first by going into Webmin -> Servers -> Postfix -> Mail Queue prior to starting Postfix.

The messages in the spool are legit. They are generated by forms on the websites. Requests for info, contact forms, shopping carts, etc. That is the problem. I can't get postfix to send them. After the last restart I can't even find a nasty fatal error message. But when flushing the spool file every message says the same thing; "no mail transport available". A bounce setting is not right but does not appear to be a fatal error. Would it be possible to replace the main.cf and/or master.cf file with one from another of our servers of about the same age?

While we should be able to determine what's wrong from messages in the log files, it's also possible to copy in a working main.cf and master.cf file.

The one thing you'd want to make sure of when doing that is to update any references of the other server's hostname or domain name to be that of the current server.

Before overwriting anything, I'd suggest making a backup of your current config files.

To my surprise there was nothing to change in either file. No reference to a specific domain or hostname that had to be changed. The second after I copied from a different machine it fixed postfix. Thank God. Thank you so much for holding my hand! Case closed. :-)