Server crash, avoid downtime.

12 posts / 0 new
Last post
#1 Tue, 05/25/2010 - 16:52
rogeriobrito

Server crash, avoid downtime.

Hello all,

I've recently a had very bad HD crash on a virtualmin box, and I couldn't recover anything on that HD. I had several custom services like Oracle and others, and about 80 virtualmin virtual servers on that server. I had it all on backup, but it took me about 36 hours to restore everything on another box. Some of my clients were offline all that time. Terrible. :(

Question is, what's the best setup to avoid downtime? To keep it minimum on a server crash? Is it possible to setup a mirror server? Or just a RAID1 on the server? Or any other solution?

Please send your comments

Thank you

  • Rogerio
Tue, 05/25/2010 - 21:58
andreychek

Howdy,

Firstly, I'd absolutely make certain that my primary server were running RAID for disk redundancy of some sort. Hard disks are one of the first parts of a server to fail, so that'll offer a lot of protection.

From there, having a hot spare is a pretty simple setup to have.

You can take a second server, put the same distro on it as your primary server, install Virtualmin -- then have your nightly backups go to your hot spare, and have them automatically imported into Virtualmin. That way, you'll always have a box ready to go that's only a day's worth of data behind your primary box.

-Eric

Thu, 05/27/2010 - 16:18
rogeriobrito

Hi Eric,

Thank you for your comments, I had a setup like your second suggestion, but I didn't automatically import the backups. How do I do that?

Thank you

Thu, 05/27/2010 - 17:45
andreychek

You can use the Virtualmin command line tools to automate importing a backup.

If you run "virtualmin restore-domain" on the command line, you'll get a list of options. You can just point that at the various files in your Virtualmin backups.

You could create a shell script that does the import, and have it run once a day or so from Cron.

-Eric

Fri, 05/28/2010 - 01:34
ronald
ronald's picture

eric, do you have such a shell script available? I'll like to try this too.

Fri, 05/28/2010 - 09:05 (Reply to #5)
andreychek

Hi Ronald,

I don't, but it may actually be a bit simpler than it sounds...

Firstly, you'd have the primary server sending backups to the secondary server. Over SSH, FTP, it doesn't matter so much, they should just end up on the second server somewhere... let's say they're in the directory /backups. That part you can configure from within the Virtualmin Scheduled Backups.

Then, on the secondary server, you need to setup a process to import all the backups. You can do something like that with:

virtualmin restore-domain --source /backup --all-domains --all-features

And then, just have that "virtualmin" command run from cron once a day or so, and you should be in good shape!

-Eric

Fri, 05/28/2010 - 14:34
ronald
ronald's picture

I deleted all domains on the backup server, transferred new backups over ssh and started restoring.
A number of restores went fine, then...
.. a clash was detected : A MySQL user named androxenelu already exists

and I have this for several domains.
I looked in /var/lib/mysql and it doesn't exist.

Where could I find these users so I can delete them? besides the restore should replace this I would say.

Fri, 05/28/2010 - 15:05
andreychek

It sounds like when you removed the domains, although the database was deleted, the MySQL user may not have been properly removed for some reason.

If you go into Webmin -> Servers -> MySQL Database Server -> User Permissions, you should be able to delete any existing users.

However, if you're able to reproduce this problem, you may want to file a bug in the Support Tracker, as that probably implies a problem of some sort :-)

-Eric

Fri, 05/28/2010 - 16:10
ronald
ronald's picture

thanks eric for pointing to the location.
what happened is that I had a script running every two hours that syncs from the production server to the backup server.
I hadn't stopped that script in time so half way my restore it started to copy the data creating havoc.
Im still cleaning up - ugh - this silly stuff takes most fun out of being a sys admin

Fri, 07/09/2010 - 09:32
rogeriobrito

Hello Eric, it is possible to use rsync on the /home directory, and a MySQL Master/Slave configuration to keep the data always up to date on backup server, right? But what about virtualmin configuration? ... Maybe I could do an automatic backup/restore but excluding the /home dir and MySQL databases, and use rsync for /home and MySQL Master/Slave config? Would that work?

Thanks

Rogerio

Fri, 07/09/2010 - 09:32
andreychek

Howdy,

Yup! That would work just fine.

For a setup like that, you could do these 3 things:

  1. Use rsync to backup /home

  2. Use MySQL replication for backing up MySQL

  3. Configure Virtualmin's backups to only generate backup data for the various config files associated with your domains -- which would get DNS information, email accounts, Apache config files, and such. You could have that generated automatically, and copied to the backup server. You could even configure a cron job to automatically import it using the Virtualmin command line tools (you can see a list of options by running "virtualmin" on the command line).

    -Eric

Mon, 07/12/2010 - 19:11
rogeriobrito

Great Eric, thanks a lot,

I'll try to set it up and post the results here in a few days.

[]s Rogerio

Topic locked