Moving to a new server, backup and restore all virtual servers

33 posts / 0 new
Last post
#1 Wed, 04/18/2012 - 19:05
rogeriobrito

Moving to a new server, backup and restore all virtual servers

Hello all,

I'm moving to a new server, better hardware and software, from Centos 5.8, php 5.2.17, MySQL 5.0 to Redhat 6.2, PHP 5.3.10 and MySQL 5.5.22, with double RAM. I have about 80 virtual servers. How do I backup and restore all virtual servers at once?

I was planning to setup a rsync to sync the home folder prior to migration, so when migrating I would backup everything except home dir. The backup restore will be much faster this way. Is this possible?

Thank you

  • Rogerio
Wed, 04/18/2012 - 22:16
andreychek

Howdy,

Although this doesn't cover migrating home directories the way you mentioned, I thought I'd link to the migration documentation in case you find it helpful:

http://www.virtualmin.com/documentation/system/migrate

You can backup all domains at once, or restore all domains at once -- from within Virtualmin -> Backup and Restore. You could also use the command line tools, using "virtualmin backup-domain" and "virtualmin restore-domain".

You can generate a backup that does not contain home directory data by unselecting the "Server's home directory and web pages " feature on the Features and Settings page in the Virtualmin Backups screen.

-Eric

Thu, 04/19/2012 - 05:56
Locutus

Eric explained it perfectly. :)

A little addition from my end: It might be feasible to do the Virtualmin backup+restore before you rsync the home directory contents.

I'm not 100% certain, but that might prevent problems, since Virtualmin will try to create the home directory in any case, even if its contents are not part of the backup, when it re-creates domains as part of the restore process. And if the directory already exists from your rsync run, VM might go bonkers. :)

Since migration is non-destructive on your existing server, you might just give it a try.

Thu, 04/19/2012 - 17:21
rogeriobrito

Thanks for your comments guys,

My full backup takes about 11 hours to complete (to an external HD), and I want to have as little downtime as possible, so I'm planning to do the following:

1 - Install new server and take it to the datacenter

2 - rsync home dir from the old server with the new server (it will take several hours to complete)

3 - when first rsync is done, stop all services on the old server (downtime starts here)

4 - incremental rsync/home again (this will be several minutes)

5 - meanwhile do a full backup of all virtual servers to a single file excluding home dir, and copy backup file to new server.

6 - after incremental rsync is done, change old servers IP and shutdown old server

7 - change the new servers IP, updating it with the old servers correct IP.

8 - restore the full backup on the new server (without the home dir will not take long. Downtime ends here)

9 - restart new server

10 - test everything

If Locutus is right, I could add 6a step, to move /home dir on the new server to something like /home2, and 8a step to move it back to /home.

That is what I plan to do. If I'm not wrong, the downtime will be less then an hour.No DNS or datacenter firewall rules changes are needed. What do you guys think? Please send me your thoughts. ...

I've just tried to do a full backup without the home dir to check how long would it take, but it failed with the message "A new format backup can only be done when the home directory is included". What should I do?

Thank you

  • Rogerio
Thu, 04/19/2012 - 22:15
andreychek

Howdy,

"A new format backup can only be done when the home directory is included"

For that, you can just change the backup format type.

On the backup screen, where it says "Backup format", choose "One file per server (old format)".

-Eric

Thu, 04/19/2012 - 23:47
rogeriobrito

Hi Eric,

I was choosing the One File Per Server, I've changed to Single file and it worked. Full backup of 131 virtual servers on 11 minutes (without the /home dir).

Do you think my steps above will work?

I have some domains that have dedicated IP addresses. Restoring a full backup, will those IP addresses be configured correctly on the new server? What about greylist settings? SSL certificates? Postfix configuration? What should I manually set up before restoring the backup?

Thanks a lot

  • Rogerio
Fri, 04/20/2012 - 02:50
ronald
ronald's picture

However you will migrate, what I did when i had to move to a new server was:
Install the new server, install virtualmin, compared settings with the old server for the stuff I didn't know by heart. Install extra software if needed.
Once the new server was okay, i moved 1 domain to the new server directly over SSH from the backup feature on the old server.

Then I checked the features again to see if everything was working properly and adjust settings if needed. I checked if the site was running properly via "Preview Website".
Once I was convinced, I backed up all domains through the backup feature over SSH directly to the new server and restored them on the new server.

Then I checked all domains and adjust IP addresses if needed, moved over the ssl certificates etc etc. until it was perfect. Only then I switched servers. It takes a while but I figured to have it right the first time and control the process manually.

Fri, 04/20/2012 - 04:14
Locutus

Ronald is right, you can do this migration with barely any downtime actually.

The only things that should change in active domains on a regular basis is database contents, and maybe home directory contents (mostly emails, possibly web files if people/admins upload stuff).

You can install and prepare the new server with Virtualmin like ronald explained, and ship it to the datacenter without taking the old server offline. Then you make Virtualmin backups of the old server (excluding home directories), again without taking it down. Restore those backups to the new server.

Then you do an rsync of all home directories, still without taking the old server down. Rsync can be done on a live system without problems. If files change during the sync, a subsequent run will pick those up.

Then, and also during previous steps, you check if config and restore on the new server are okay.

Only then you take down the old server, and do another Virtualmin backup all of the databases only, and restore those to the new server. Should not take more than a few minutes, since DB contents are usually small compared to home directory.

Then, with the old server down, you do another home directory rsync to pick up files that were changed during the "big sync". No need to fiddle with home directory names like "/home2" there to not confuse VM.

You can even "rehearse" those steps that mean actual downtime, by doing them with the old server still online. The process will work regardless, only with the distinct danger that DB contents or files change during the transfer.

Check if everything is okay on the new server, then redirect nameserver entries to point to the new server. Oh by the way, one day before the migration you should set the TTL of your domains to like a minute, so that DNS propagation delay won't spoil your nice almost-no-downtime scheme. :)

Mon, 04/23/2012 - 13:40
rogeriobrito

Yes, I'll do all the steps using a new IP to check everything worked good before the actual migration. But at the time of the migration I think it is best to use the old server's ip on the new server, without changing the DNS, because if I change the DNS, while the information is propagating I'll have both servers online, and I'll have users on both servers. That can be a problem with ecommerce websites, where users are posting orders on both servers. Also email messages being delivered on both servers, somethings can get lost. Or is there a way to overcome this? Thanks a lot for your comments guys.

  • Rogerio
Mon, 04/23/2012 - 17:11
ronald
ronald's picture

I guess you have a point there. However you may want to close shop for a few hours and migrate at a point with the least customers

Tue, 04/24/2012 - 23:46
rogeriobrito

Yes, of course. I'll do it on late hours, after midnight. I'll update this post with the results of my tests and migration.

Thanks for all the help.

  • Rogerio
Wed, 05/09/2012 - 18:22
rogeriobrito

I'm preparing to move this weekend and from the tests I've done some things will have to be moved manually, like postfix configuration.

Where is the Greylisting configuration file so I can copy it?

Thanks

  • Rogerio
Wed, 05/09/2012 - 22:05
andreychek

Howdy,

The greylisting config is in /etc/postgrey/.

You would first need to enable greylisting though, that can be done in Email Messages -> Email Greylisting.

-Eric

Thu, 05/10/2012 - 23:31 (Reply to #13)
rogeriobrito

It is actually

/etc/postfix

files (postgrey_whitelist_clients, postgrey_whitelist_clients.local, postgrey_whitelist_recipients)

Thank you Eric!

Thu, 05/10/2012 - 09:23
rogeriobrito

Hello guys,

New problem on my tests. I have my new server ready with restored full backup (excluding /home dir), and /home dir copied with rsync. All websites are working good, but the problem is all the email folders on IMAP clients have become unsubscribed!

How can I fix this?

Thank you

Thu, 05/10/2012 - 09:59
andreychek

Where are you Dovecot index files stored?

One way to determine that is with this command:

dovecot -n|grep mail_location

Thu, 05/10/2012 - 10:14 (Reply to #16)
rogeriobrito

Here it is..

[root@linux03 public_html]# dovecot -n|grep mail_location mail_location = maildir:~/Maildir:INDEX=/var/lib/dovecot-virtualmin/index/%u:CONTROL=/var/lib/dovecot-virtualmin/control/%u [root@linux03 public_html]#

Thanks

Thu, 05/10/2012 - 10:37 (Reply to #17)
rogeriobrito

Previous post is for the new server.

For current server it is:

[root@linux03 Maildir]# dovecot -n|grep mail_location mail_location: maildir:~/Maildir [root@linux03 Maildir]#
Thu, 05/10/2012 - 12:36
andreychek

To resolve the issue you're seeing, you could set the mail_location on the new server to use the value from your old server.

Then, copy over the /var/lib/dovecot-virtualmin/ directory (and all the sub-directories), which contains the dovecot index and control files.

-Eric

Thu, 05/10/2012 - 13:40 (Reply to #19)
rogeriobrito

Hi Eric,

Changing the mail_location did it. All folders are ok now.

Do I really have to copy over the /var/lib/dovecot-virtualmin/ directory ? The /home directory will be rsynced again before the migration.

Thank you!

Thu, 05/10/2012 - 13:07
rogeriobrito

The current server has 4Gb RAM.

The new server has 16Gb RAM. I've used the Percona Online tools to create the MySQL configuration file. Is there any other configuration I should update to make better use of the extra memory?

Thanks

  • Rogerio
Tue, 05/22/2012 - 16:35
wocul

I'd just like to thank everybody who contributed to this discussion. This has been really insightful and truly helpful.

Eventually, it'd be great if VirtualMin could provide some more support for such migrations, i.e. using some sort of wizard to streamline the entire workflow (many of the steps involved are largely identical).

In the meantime, it'd be great if some of the info mentioned in this thread, could be added to the mentioned "migration tutorial" here: http://www.virtualmin.com/documentation/system/migrate

Alternatively, I'd suggest to at least add a link to this thread.

I feel it'd be great if VirtualMin/CloudMin could further work towards this goal, i.e. of facilitating migrations between installations (virtual or dedicated). I went through a lot of trouble doing such things with other management panels, and it seems VirtualMin is almost half way there.

With some more automation and customization options, many of these steps could be done in an automated/scripted fashion.

Just think in terms of setting up additional name server entries, preparing migrations etc.

What could be done a little better on the VirtualMin side of things is providing explicit support for migration-related features. This includes for example remote database replication/synchronization, in the case of mySQL this is fairly simple to set up (found instructions on your forum!) and then there are other things that are currently not yet backed up as part of a virtual domain server, such as for example domain-specific cron jobs.

The potential issue related to migrating virtual domains and eMail addresses were already mentioned.

Some scripted method to compare/validate a migrated server would also be great.

It would be great to see a little more consistency eventually, so that migrating a "virtual domain" from one virtualmin installation to another one, could be largely automated using a handful of custom steps.

Thanks again

  • wocul
Tue, 05/22/2012 - 17:23
rogeriobrito

Hello all,

Just to send a final word, the migration was successful and downtime was only about 40 minutes, using the IP from the old server on the new server. Very good! I just did one thing differently from the steps above, I've changed the old server's IP to another IP instead of turning it off, then updated the new server with the old server's IP and I could still access the old server to check and copy some configurations.

A few things to be aware of when migrating like I did from a Centos 5.8 32 bits, PHP 5.2.17, MySQL 5.0 to Centos 6.2 64 Bits, PHP 5.3.12, MySQL 5.5.32.

  • If you don't have SELinux enabled on old server, disabled it on the new server before the first restore test. You can have permission problems if after restoring the backup you decide to disable SeLinux.
  • php.ini modules directory was restored as (extension_dir = "/usr/lib/php/modules") but on the new server the correct path is (extension_dir = "/usr/lib64/php/modules"), so I had to do search/replace text on all php.ini files. Also had to set the timezone on all php.ini files.
  • Mysql 5.5 does not accept long usernames as 5.0 does. So I had to change some long usernames that used to work on my PHP applications, updating them to the short name showed on Virtualmin->Edit Virtual Server (For MySQL database : username). Virtualmin shows this when the Administration username is longer than MySQL accepts.
  • Dovecot mail_location was different on the new server, so I had to update it to match the old server's location.
  • Also, postfix, spamassassin, Postgrey, ProFTPd FTP Server and SSH server custom configurations were copied manually.

Other than that, it was all good, I'm pretty happy with the server up and running.

Congratulations for the Virtualmin guys, it is a great product. And thanks for everyone that helped on this thread.

Cheers,

  • Rogerio
Mon, 06/04/2012 - 10:16
wocul

The postfix/dovecot related issues seem familiar, it would be great if VirtualMin could recognize such mismatches and suggest a modified config for review, so that the corresponding settings and folders can be rsync'ed

And yes, it would be also great if VirtualMin could recognize that a VHOST has matching cron jobs under the same user name, so that these could be also saved/restored accordingly.

Thu, 12/06/2012 - 05:16
fuggi

Also, postfix, spamassassin, Postgrey, ProFTPd FTP Server and SSH server custom configurations were copied manually.
Are those settings (usually found in Webmin, also e.g. Apache's general configuration) not backed up and restored by Virtualmin's backup with the --all-features option?

Regards,
fuggi

Thu, 12/06/2012 - 09:18
Locutus

Custom config for Postfix, and Postgrey whitelists are copied in the Virtualmin backup, the rest is not.

Thu, 12/06/2012 - 10:57 (Reply to #26)
fuggi

Are they at least copied by Webmin's "Backup configurations files" functionality?

Thu, 12/06/2012 - 14:23
Locutus

Never tried that, but I suppose they should. What's the problem exactly? You could always just archive /etc and extract what you need from it on the new server.

Fri, 12/14/2012 - 05:49
fuggi

The problem is that it is neither an easy nor robust procedure if some parts are done automatically but others have do be done manually and the admin has to keep it all in mind.

Fri, 12/14/2012 - 06:54
Locutus

Yes, I have to agree there.

Problem though is that Linux installations are very versatile and can differ greatly. Many different distros, a patchwork of software put together, and everybody sets stuff up their own way. So, coming up with a migration procedure that works for all of them is non-trivial to say the least. :)

So, some manual work is probably always required when doing such a migration.

Sun, 12/29/2013 - 18:30
wocul

this isn't mentioned on the website ( http://www.virtualmin.com/documentation/system/migrate ) - so just for future reference:

to reduce downtime, dynamic webbsites using mySQL (phpBB, wordpress, drupal etc) will be typically set up to do mySQL replication during the migration to a new server, i.e. remote DB mirroring onto the new server - that's when the old server can stop using its own local DB and use the new DB as its master DB, so that regardless of the DNS propagation delay, both servers (old + new) will use the new DB on the new server.

http://www.virtualmin.com/node/18141

http://www.virtualmin.com/node/10137

http://www.virtualmin.com/node/18268

Mon, 12/30/2013 - 17:42
Chris_C

Thanks for this thread. I'm probably going to have to go through this somewhat aggravating sounding manual/automatic migration of virtualmin from one machine to another, since the VPS OS is debian 6 squeeze which will reach end of support in about 5 months, May 2014 probably, requiring a move to debian 7 wheezy, but that'll require moving from the current hardware node which is centos 5 kernel 2.6.18, to a new hardware node since debian 7 wheezy requires the hardware node to be running kernel 2.6.32+ aka centos 6. Argh.

Tue, 12/31/2013 - 13:48
wocul

I just went through the same thing and ended up messing up my whole installation, there are some folks in the "jobs" forum offering support with these sorts of things - if you cannot afford any downtime, I suggest to get in touch with them ...

BTW: I used this tutorial to set up mySQL replication in order to migrate the whole thing with zero downtime: http://techblog.procurios.nl/k/n618/news/view/56429/14863/how-to-migrate...

And this for validating the replicated DB: http://www.tutorialsolve.com/2009/07/23/tutorial-for-mysql-data-validation/ http://www.alexwilliams.ca/blog/2009/10/01/scripted-mysql-replication-co...

Once everything was in place,I only had to change the DB credentials on the old server to use the new/replicated DB instead.