Services Are Down (BIND DNS, Postfix Mail, MySQL). Need Urgent Help please...

23 posts / 0 new
Last post
#1 Wed, 11/10/2010 - 16:11
decay

Services Are Down (BIND DNS, Postfix Mail, MySQL). Need Urgent Help please...

Ahead of time, i am a linux noob. So if you want me to do something, please tell me how to do it also please :)

I am in need of urgent help. Unsure whats going on with my server. Last night all my websites and web services went down on my vps, i contacted my host (volumedrive) and they changed the vps over to new hardware etc. So now the vps is on new hardware and im still getting the same issue/error (basically it didnt fix anything).

Look at the attached image (1.jpg - https://www.virtualmin.com/system/files/1_0.JPG), it shows my virtualmin system information page. it shows that 3 of the services are down and i cant seem to turn them back on (they were running fine a day ago). Can anyone help me? I need to know whats causing this (weather it is Virtualmin, Any other server/linux issue, or my hosts fault)...

This is what i currently get:

When i click "refresh/recheck configurations" i get this:

The status of your system is being checked to ensure that all enabled features are available, that the mail server is properly configured, and that quotas are active ..
BIND DNS server is installed, and the system is configured to use it.
The Postfix server does not appear to be installed on your system, or has not yet been set up properly in Webmin's Postfix Configuration module. If you are running a different mail server, select it on the module configuration page.

.. your system is not ready for use by Virtualmin.

if i try to start BIND DNS Server i get:

Failed to start service : Failed to start BIND : Starting named: [FAILED]

if i try to start Postfix Mail Server i get:

Failed to start service : postsuper: fatal: scan_dir_push: open directory defer: Permission denied

if i try to start MySQL Database Server i get:

Failed to start service :
Timeout error occurred trying to start MySQL Daemon.
Starting MySQL:  [FAILED]
Wed, 11/10/2010 - 18:13
Xencored

You could try updating them packages you have 40 lol there Also run the re-check

Wed, 11/10/2010 - 18:18 (Reply to #2)
decay

did that earlier already... didnt change anything... :(

Wed, 11/10/2010 - 18:23
Locutus

I'd also suggest to first take a look at the package manager (whichever comes with CentOS). For one, to perform the updates, and then, to check for broken dependencies or broken packages. Also try manually starting the BIND and MySQL services in SSH.

Vmin complaining about a missing Postfix and unable to start BIND and MySQL sounds like something is rather seriously broken there... What exactly can have happened that caused this? Some system crash or forced reboot or something?

Do you have backups available (of the system or websites)? It's possible that you need to reinstall the system and restore a backup, especially if you're a "Linux noob" and don't really know how to fix stuff. :)

Wed, 11/10/2010 - 18:30
decay

The volumedrive customer support is very poor. They just sent me the following email after about 12 hours of bugging them:

After looking over the log files and trying to get the services to start, we believe the best suggestion would be to create a new account and have everything moved to the new VPS. We apologize that some of the process and files were damaged in the move, but it was necessary. The IP will stay the same for the VPS. If you would like to do this, please advise and we will have the new VPS setup ASAP so you can begin moving the files.

Now i have no prior backups of the linux system or of virtualmin. Any suggestions on how i can transfer the data over?

For one, i cannot make a backup of the virtualmin as it for obvious reasons:

Backup failed : DBI connect failed : Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

what can i do?

Locutus, thanks for the info. but i have no idea how to do the things you told me... :( And i have no idea what originally caused this, they just completely ignore some of my questions, and im getting a bit annoyed at them for it.. :S

Im kinda screwed at the moment as i had few client websites running on the server (and they were using the MySQL databases)...

Wed, 11/10/2010 - 18:50
Locutus

Well, that's surely a pity... But I must say that creating regular backups is one of the first and foremost "must"s for a server administrator (especially since Virtualmin and Webmin have excellent backup features). Things can basically go awfully wrong at any time, as you've seen now...

(Aside from this, I'd quite probably ditch that "hoster" you're using... If they're unable to move your virtual machine to another host without breaking most of your installed services, however they managed to do that, and don't even hold backups of those for you... Not really the service I'd be using. :) )

To save the databases now, it might help to copy away the /var/lib/mysql directory, where MySQL should - at least it does so in Ubuntu - store all its databases. You might be able to copy those files back after the reinstall.

Same goes for the contents of /home, where all vserver files, users and mails should be stored.

And you might copy away /etc, which holds all config files. Warning! Do not simply copy back the contents of /etc after the reinstall, but only use the copy as reference, in case the new installation is missing things or settings.

As for "no idea how to do the things I told you": I'm sorry, but frankly I must say, if you don't know how to login to your server via SSH and restart a service (and also cannot use Google etc. to find out how to do that), I'm afraid I'll have to suggest you should reconsider if operating a Linux-based hosting server is the right thing for you... Virtualmin is a great help for web hosting, and eases mostly all processes involved in that, but it generally does not liberate you from the responsibility to still have at least a basic understanding of the underlying system.

Wed, 11/10/2010 - 18:49
decay

Thanks. Yeah, im very new to this as you could probably guess. I have googled and figured out how to use bind. I tried running Bind:

/etc/init.d/named start
Starting named:                                            [FAILED]

Does this mean anything?

Question: what would happen if i tried to reinstall bind again using ssh? Is that a bad idea?

Wed, 11/10/2010 - 18:52
Locutus

Hehe, not very talkative there. You'll probably find more in /var/log/syslog as to why the startup failed.

Reinstalling the package sure can't hurt, as in it probably won't damage more than is already broken. ;) Unfortunately though I can't help you with operating the CentOS package manager, since I'm using Ubuntu (a Debian derivative) myself, which has a quite different one.

Wed, 11/10/2010 - 19:14
decay

Sigh, reinstalling bind didnt make any difference.

I asked the host to set me up with a new VPS, now i need to reinstall everything and transfer things over.

Is there a easy way to transfer files from 1 VPS to another using ssh if i know the ip addresses of both?

ps. yeah im sooo tempted to move hosts now... siiigh...

Wed, 11/10/2010 - 19:48
Locutus

Yeah, you can use SFTP... Which is like an FTP client, but using the SSH protocol to transfer stuff. There's a text mode client "SFTP" which works much like the Linux FTP, albeit it's not very comfortable. Might use the Midnight Commander instead which can make sn SFTP connection using "Shell Link" in the Left/Right menu. Might have to install the package for that though, which is named mc in Ubuntu.

Wed, 11/10/2010 - 22:19
decay

OK, now this is where im upto...

New VPS has everything installed virtualmin working etc...

But now, i need to migrate the database over from the old one. as Locutus mentioned, i copied over all the files in: /var/lib/mysql to the same directory on the new server.

But that seems to have screwed up a lot of things maybe? This database has a joomla website by the way. Virtualmin says the following:

Database name db_test
Database server type MySQL
Size on disk 1.14 MB
Number of tables 46

seems like there are some content. But if i click manage this database i get: SQL show index from jos_banner failed : Can't find file: './db_test/jos_banner.frm' (errno: 13) This file is actually in the server.

can anyone understand why this is happening?

Oh also, i did make the same database names on virtualmin before i copied files over.

and now if i try to create a new database (db name = bob) i get:

Failed to create database : SQL insert into db (host, db, user, Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv, Drop_priv, Grant_priv, References_priv, Index_priv, Alter_priv, Create_tmp_table_priv, Lock_tables_priv, Create_view_priv, Show_view_priv, Create_routine_priv, Alter_routine_priv, Execute_priv) values ('localhost', 'bob', 'gismlk', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y') failed : Duplicate entry 'gismlk' for key 2
Thu, 11/11/2010 - 04:27
Locutus

Did you also copy all subdirectories from /var/lib/mysql? Did you stop MySQL while copying the files, and restart it afterwards?

Thu, 11/11/2010 - 05:16 (Reply to #12)
decay

yes, yes and yes... :(

now if this method wont work, is there any other way to export a mysql database without mysql running? O.o

siigh, this is hurting my brain lol..

Thu, 11/11/2010 - 07:24
andreychek

now if this method wont work, is there any other way to export a mysql database without mysql running?

Without MySQL running, you'd need to just copy the database files. That's a fast and reliable way of copying MySQL databases around.

Do you see the directory "/var/lib/mysql/db_test" on both servers?

And then in that directory should be the file "jos_banner.frm", does that exist on both servers?

Next, you'd want to verify that the permissions and ownership are the same in both cases. You can see the permissions and owners by running "ls -l /var/lib/mysql/db_test/jos_banner.frm".

-Eric

Thu, 11/11/2010 - 10:41
Locutus

Ownership and permission is a good point, indeed. Although then the error message should have been "permission denied", and not "can't find file". :)

Next step, if the jos_banner.frm file does exist in the old and new database directory, would be to test accessing it directly from the MySQL command line client.

(If the file actually does not exist, it probably got corrupted/deleted during whatever damaged the rest of your server. At which point you'd again need that backup... :-( )

Execute this in SSH:

mysql -p{your-mysql-root-password}

Make sure there's no space between the -p and the password, and it's not your server root password, but the one you entered during the "Virtualmin post-install wizard" as your MySQL root password. Then execute:

show databases; use db_test; show tables; select * from jos_banner limit 5;

If it works okay, that will show you all databases on the server, then all tables in the database "db_test", then the first 5 rows in the "jos_banner" table.

Thu, 11/11/2010 - 14:35
decay

ok, i see the problem here... when i execute those mysql commands it says:

mysql> show tables;
ERROR 1018 (HY000): Can't read dir of './db_test/' (errno: 13)

then i checked the permissions... the permissions of the contents of the both folders (new and old) are both exactly the same. BUT, the permissions of the folder it self is different..

OLD
drwx------ 2 root root     4096 Nov 10 14:03 db_test

NEW
drws--S--- 2 root  my_user2     4096 Nov 11 15:01 db_test

now to figure out how to fix permissions... arg... do you think this is the cause? My understanding of these permission letters are minimal, but i wonder what that S represent... :S

Thu, 11/11/2010 - 15:25
Locutus

Yeah, both of those owner/permission sets on the old and new server look odd. Normally, the database subdirectories (except for "mysql" itself) are owned by mysql:{virtualserveradmingroup} and have permissions like those on your new server. If that differs, it can clearly be the reason. In your case, the MySQL process does not get access rights cause the directory is owned by root.

You should check under which user account the MySQL daemon is running, you can do that with ps aux | grep mysql and check for the line with the mysqld executable. The user (first column) listed there should be the owner of database directories/files.

Ownership can be changed with chown, permissions with chmod.

The "s" bit is called "setuid / setgid" (set user ID, set group ID). They cause that new files created inside the directory get the ownership of the directory, and not (as is normally the case) the ownership of the user creating the file.

Thu, 11/11/2010 - 15:28 (Reply to #17)
decay

are you by anychance able to give me a ls -l of your mysql folder, so i know what its supposed to look like?

ill try doing what you said for the time being :)

Thu, 11/11/2010 - 20:06
decay

YEY! its finally fixed... oh god that took a while...

basically had to reset the owners on the databases and the tables... Then it was throwing an error saying some tables dont exist... then i had to go manually delete these tables and any links they had from memory 1 by 1... hmmmm, and that fixed it... :)

Thank you both for your awesome help... ahhhh, now i can go get some sleep XD...

btw, anyone know a good/cheap backup service i can use easily with virtualmin? O.o (not going through this again... ever...)... lol..

Thu, 11/11/2010 - 20:12
andreychek

btw, anyone know a good/cheap backup service i can use easily with virtualmin?

If you haven't already, you may want to take a look at Virtualmin's built-in backups. You can access them by going into Backup and Restore -> Scheduled Backups.

I'd suggest setting up a backup to run each day.

You can have it store the backup locally, or on a remote FTP or SSH server.

-Eric

Thu, 11/11/2010 - 20:17 (Reply to #20)
decay

ohh yup yup... i know about that, used that to transfer the files over to the new VPS. it made my life sooo much easier :)

my question was more aimed at a remote backup place i can use. Like some hosting company or something that i can use virtualmin to save backups into. Saving locally on the server is bad if the server dies, and saving locally on my machine is annoying because reusing the backup will be annoying with my upload speed (internet sucks over here)...

Just wondering if theres any recommendations on such places thats all. :)

Sat, 11/20/2010 - 23:24 (Reply to #21)
Jerry Hudgins

I've used rsync.net for that for a good while, and it's worked well.

Fri, 11/12/2010 - 05:27
Locutus

Usually, worthwhile hosting companies offer FTP backup space for their customers. :)

I personally would not use a hoster which does not offer that. Hence I haven't really checked yet on the net for alternative backup space, can't say much.

I do three levels of backup on my virtual machine host. 1) VMs store images of system and user data on another dedicated backup storage VM (on the same host). 2) From there, backups are auto-copied to the hoster's backup store. 3) From time to time, I initiate an offsite backup (= copying the images) to a machine at University where I work :).