Do I Really Need Cloud Min to Migrate Servers from One Host to Another

We are about to migrate from a dedicated server (CentOS 6.2) at GoGrid (housed in their data center in San Francisco) which has 9 domains/virtual servers managed with VirtualMin/WebMin, to Linode (Ubunty 14.04 LTS) at their Hurricane Electric 2 data center in Fremont.

We plan to put one domain on a separate "linode" instance, another domain on another "linode" instance and third linode "instance (with more HD space) will hold the other 7 domains.

My developer in Brazil (who also uses your products) said that for this set up: three different linodes, where we have no intention of doing more later, that we really do not need CloudMin. I just wanted to confirm that Cloud Min really doesn't "get us anything"

I think the process is still the same for migration, whether we have CloudMin or not, we have to use VirtualMin to back up the site on the machine in SF, then on the new cloud instance in Fremont we "restore" ... Can VirtualMin do this "remote-between-to-different-hosts" job? Or would it help to use CloudMin. There was a suggestion on the forums here that since we are moving from CentOS to Ubuntu, that we might not want to migrate the virtualMin settings for the sites.

We own a 10 domain license, but since you don't support a license across different cloud instances, I'm told I will need to install VirtualMin GPL on the two instances that only have a single domain. Please confirm. Never having done this before I'm a bit at a loss. trying to make it as pain free as possible but a bit concerned about the "upgrade" to Ubuntu and possibly newer versions of PHP an MySQL.

My plan would be to spin up the "big" Linode instance... install our current VirtualMin on there (can I do that when it is also on the original server?) and then let VirtualMin do all the app layer updates and installations.

Any advice on how best to proceed will be much appreciated.

Status: 
Active

Comments

Howdy -- it doesn't sound like Cloudmin would be managing the VPS instances that you'd be managing.

In that case, you can perform your migrations using the documentation here:

https://www.virtualmin.com/documentation/system/migrate

That documentation should cover most of the questions you asked.

For example, it's no problem to change distributions during a migration. And you can use your same Virtualmin Pro license on both servers during a migration.

Feel free to let us know if you have any other questions!

ha... I just found that page before I got your response by email.

but it doesn't work to do a "copy all domains" so yes, questions:

1) two of them are huge... 80GB each (most data in public_html/media) where we have lots of audio and image files) so does the back up command bundle all of the into 1 single giant tar ?

2) I need to put two of the domains each on their own independent "instance" /server.

would we still use the

mkdir /root/backups virtualmin backup-domain --dest /root/backups/ --all-domains --all-features --newformat --all-virtualmin

cmd?

what if we don't want to do "all-domains" at once? There are a lot of timing issues with data being generated daily on the blog of the old site... that will be missing on the new site... so we want to go carefully one by one... and then shut down any updates on the old site until DNS migrates (this seems safer than expecting "incremental" back up to just pick up the most recent in the MySQL databases)

3) What about php.ini if the PHP version is different?

4) can I use my current 10 domain Virtual min license on all three cloud servers? (I will still only admin 9 domains total)

5) Does it make sense to do individual back ups first of the "little" sites (not much data) and test those first?

6) Does your "incremental" back up just examine the changes and then port new changes from the old site to the new site? And, does this include additions to MySQL databases?

1) two of them are huge... 80GB each (most data in public_html/media) where we have lots of audio and image files) so does the back up command bundle all of the into 1 single giant tar ?

It generates backups in one .tar file per domain.

2) I need to put two of the domains each on their own independent "instance" /server.

would we still use the

mkdir /root/backups virtualmin backup-domain --dest /root/backups/ --all-domains --all-features --newformat --all-virtualmin

If you generate the backups as seen in the migration documentation, you can just copy the .tar file for each domain to the server you wish to restore it on.

what if we don't want to do "all-domains" at once?

You just need to generate a backup for the domain, you can do that any way you want, and copy that to your preferred server.

3) What about php.ini if the PHP version is different?

There may indeed be issues with the php.ini file, if the PHP versions have a different syntax. I would suggest testing the migration first to determine what all would need to be corrected.

4) can I use my current 10 domain Virtual min license on all three cloud servers? (I will still only admin 9 domains total)

If you're migrating to multiple servers, you would need a license for each of the servers you're moving to.

5) Does it make sense to do individual back ups first of the "little" sites (not much data) and test those first?

I would definitely suggest testing first, for any domains where you don't wish to see downtime.

6) Does your "incremental" back up just examine the changes and then port new changes from the old site to the new site? And, does this include additions to MySQL databases?

An incremental backup contains only the files that changed since the last full backup.

However, all database content would be included.

Feel free to let us know if you have additional questions!

Andreychek wrote:

"You just need to generate a backup for the domain, you can do that any way you want, and copy that to your preferred server."

"anyway you want" ... I presume in the virtualMin GUI we can use

Backup and Restore --> Backup Virtual Servers (panel dialog) --> select a server --> "Any plan" [leave as default] --> Features to back up (Select All) --> Virtualmin settings to back up (select all)

and we get the same result as the command line?

I'm looking at your docs for cmd line operation -- does this look right? (where, in this example, the single domain would be hinduismtoday.com)

virtualmin backup-domain --dest /root/backups/ --domains hinduismtoday.com --all-features --newformat --all-virtualmin

Did I get that right?

and what about the tar for VM itself?

virtualmin restore-domain --source /root/backups/virtualmin.tar.gz --all-virtualmin

is this generated when we use

[--all-virtualmin] | [--virtualmin config]

the first time? and then for subsequent domain back ups we don't need that?

Also why don't we need " [--as-owner]" if you back up as root... is the owner/group of all the files preserved? and recreated on restore?

??

Backup Virtual Servers

Yes, these migration steps aren't doing anything "special" in generating the backups -- they're just typical backups.

Performing a migration is just a matter of generating a full backup, copying it to your new server, and restoring it.

The instructions above are an example of how to do that, but there's many other legitimate ways to perform a migration.

The command you mentioned above looks good.

The virtualmin.tar.gz file is generated because the "--all-virtualmin" was passed in on the command line when making the backup. Subsequent backups don't need that --all-virtualmin parameter, unless the Virtualmin settings changed.

The "--as-owner" parameter isn't necessary when run from the command line. It just causes the backup process to run as the owner, rather than root. In both cases, the permissions would be preserved in the same way.

One last question and I'm good to go

VirtualMin GPL should work fine on the two servers with one domain each, right? I guess I would need to restore a copy of the Vm tar file on each of the three servers (where there was only one before) VM GPL will restore the back up made from VMPro?

Or maybe we let the VM GPL start fresh on those two?

Yeah if you don't need Virtualmin Pro on two of those servers, I would suggest putting Virtualmin GPL on there, and restoring the backups with it.

It's no problem to restore the Virtualmin Pro backups using Virtualmin GPL -- you just wouldn't have access to Virtualmin Pro features, such as resellers, or new install scripts (but any existing apps installed via an Install Script will continue to work fine).

Alright we made progress, spun up a new "linode" and installed virtual min and tried to do a back up of one small domain:

Faced some issues:

1) on the fresh in stall on ubuntu, the final step to verify VirtualMin config failed due to lack of a Mailman list of some sort. "Virtual Min Not Ready to use" was the report at the bottom.

2) we then went over to the current/old server and ran

[root@sat backups]# virtualmin backup-domain --dest /root/backups/ --domain wailuarivernursery.com --all-features --newformat --all-virtualmin

And we got a failure (see below)

what now?

Starting backup.. Creating backup for virtual server wailuarivernursery.com ..     Copying virtual server configuration ..     .. done

    Backing up Cron jobs ..     .. none defined.

    Copying records in DNS domain ..     .. domain not found!

             Saving Virtualmin configuration ..         .. done

        Saving templates and plans ..         .. done

        Saving resellers ..         .. done

        Saving email templates ..         .. done

        Saving custom fields, links, categories and shells ..         .. done

        Saving custom script installers ..         .. done

        Saving custom styles ..         .. done

        Saving scheduled backups ..         .. done

        Saving FTP directory restrictions ..         .. done

        Saving DKIM settings ..         .. not installed

        Saving greylisting settings ..         .. not installed

        Save mail rate limiting configuration ..         .. not installed

        Saving mail server configuration ..         .. done

    .. done

    Backup failed!

Hmm, are you using Mailman? If not, you may just want to go into System Settings -> Features and Plugins, and there disable the mailman feature.

Regarding the backup that failed -- it looks like it's having trouble finding the DNS records for that domain.

Is it using Virtualmin for DNS? Or is DNS for that domain being hosted externally, at a third party?

DNS IS hosted by a third party

What about the inability verify VirtualMin on the new linode because of missing mailman list (it seems like VM want a list to be created...)

What about the inability verify VirtualMin on the new linode because of missing mailman list (it seems like VM want a list to be created...)

It's possible my answer there blended into the rest of my response in comment #9 -- I'll paste that in again below. Let me know if you have questions on this:

Hmm, are you using Mailman? If not, you may just want to go into System Settings -> Features and Plugins, and there disable the mailman feature.

DNS IS hosted by a third party

If DNS is hosted at a third party, it shouldn't be looking for the DNS zone on your server.

In Edit Virtual Server, is the BIND DNS Domain feature disabled? You shouldn't need that if your server isn't hosting DNS for that domain.

Sorry I was reading peddle to the middle and missed your mailman response.

And, yes we had DNS Bind on for some sites when it is not needed, will uncheck and try again.

andreychek:

I just wanted to say that we succeeded with the new VirtualMin Install on the new host server at linode, backed up several small domains on the old box, ran the scp to push the tar's over to the new server, ran restore one of them late this afternoon and all went well! getting the site in preview...

That site only required Apache and nothing else, so we will see how things go with the more complicated ones, but so far, so good!

Thanks for all your guidance!

Glad to hear it's coming along well, thanks for the heads up!

We have ported five small domains to the new server and they are all on a shared IP, the restore process seems to have gone without a hitch. We are getting new home folders for each new user, and httpd.conf directives all look good with the correct doc roots pointing to the correct doc roots for each user.

BUT...when we try to preview the sites we restored today, they all default to the doc root of the first site that we restored this morning. Just guessing, but it appears to us that virtual min is somehow "stuck" on the doc root for one user/site. We dont know how to fix this.

All the settings that we see are unique for each restored site. But we are seeing the homepage for Domain A even when trying to preview Domain B.

Hmm, what is the output of this command on the server where you're seeing that issue:

grep -i <virtualhost /etc/apache2/sites-enabled/*.conf

root@hostname:/home/wailuanursery/public_html# grep -i <virtualhost /etc/apache2/sites-enabled/*.conf -bash: virtualhost: No such file or directory

Shoot, I missed the quotes in that command... it should look like this:

grep -i '<virtualhost' /etc/apache2/sites-enabled/*.conf

root@hostname:/etc/apache2/sites-enabled# grep -i '<virtualhost' /etc/apache2/sites-enabled/*.conf /etc/apache2/sites-enabled/000-default.conf: /etc/apache2/sites-enabled/0-wiki.hindu.org.conf: /etc/apache2/sites-enabled/0-wiki.hindu.org.conf: /etc/apache2/sites-enabled/eastkauaiwater.org.conf: /etc/apache2/sites-enabled/jnanadana.com.conf: /etc/apache2/sites-enabled/kauaialohaendowment.org.conf: /etc/apache2/sites-enabled/wailuarivernursery.com.conf:

interesting... I see that in the GUI for the Webmin-Apache Server... only the Wailuarivernursery has an actual IP assigned... and that is the one that is showing for all the others that we try to preview. the asterisk is translated as "Any"

also... 0-wiki.hindu.org.conf is wrong... why a "0-" in front of the domain?

FYI these were all backed up on the server in SF (using VirtualMin) scp to the new linode image and then restored there

They all had and asterisk before the port:80 (btw... where is a like to your markdown rules for these posts?)

we got this:

root@hostname:/etc/apache2/sites-enabled# grep -i virtualhost' /etc/apache2/sites-enabled/*.conf /etc/apache2/sites-enabled/000-default.conf:<VirtualHost *:80> /etc/apache2/sites-enabled/0-wiki.hindu.org.conf:<VirtualHost *:80> /etc/apache2/sites-enabled/0-wiki.hindu.org.conf:<VirtualHost 45.56.82.123:443 [2600:3c01::f03c:91ff:fe37:416]:443> /etc/apache2/sites-enabled/eastkauaiwater.org.conf:<VirtualHost *:80> /etc/apache2/sites-enabled/jnanadana.com.conf:<VirtualHost *:80> /etc/apache2/sites-enabled/kauaialohaendowment.org.conf:<VirtualHost *:80> /etc/apache2/sites-enabled/wailuarivernursery.com.conf:<VirtualHost 45.56.82.123:80 [2600:3c01::f03c:91ff:fe37:416]:80> [code]   I went in and manually edited all those *conf files and entered the full IP string from the nursery site and restarted apache... now they all work... but it would help us to understand if we did something wrong upon restoring the sites.

I fixed the conf files manually form with nano... they all preview as expected now. Must have been some quirk in the restore process that we missed.

We're glad to hear you got it working!

We'll do some testing with restores on Ubuntu 14.04 and CentOS 7 to make sure there isn't a problem with the way domains are restored, as both of those distros are using Apache 2.4.

But still I would like to preview sites form the IP so that we can avoid and strange behaviors... I looked up this issue... seems it is not recommend as a long term strategy for security reasons, but for the period of this migration it would be good if I could test sites on a shared IP using

http://44.44.44.192/~user

or something like that... is that possible? in some of these PHP CMS's (XOOPS, DokuWiki and others) they want to set a baseURL... since DNS is not cut over... would like to be able to enter that BaseURL in that form temporarily.

is that possible? put another way:

is the documentation on this still valid as of May 2015:

http://www.virtualmin.com/documentation/web/faq

But this documention is either outdated or unclear as I am unable to locate the "Click on the Automatic Virtual Hosts icon."

[FYI: I understand the security risks and would turn this all off as soon as testing is finished. this is a not a long term strategy for reselling space to let clients see their sites before going live... it's just to get me thru this week.]

Those issues are increased exponentially by allowing CGI/PHP scripts to run in this way, and we strongly discourage use of UserDir with executable applications, as it introduces serious security problems.

docs say:

However, you can use this mode, if you wish.

To switch your Virtualmin server to behaving the way you're used to, you just need to turn it on in Apache. Here's how:

Browse to the Apache Webmin module.

Click on the Default virtual server (just be sure you know what your default Domain is so you can point folks to the right place).

Click on the Automatic Virtual Hosts icon.

Switch the radio button by Automatic virtual host root to the one beside the empty field, and fill in the field with, /home/%0/public_html. Note that if you've configured Virtualmin to create domains with a different path convention (e.g. "/home/a/abc") things get more complicated and you'll need to construct a path using the variables documented here:

http://httpd.apache.org/docs/1.3/mod/mod_vhost_alias.html

Due to the way userdir works, it's not simple to do that.

Instead, our recommendation is to use aliases.

Point a particular domain to your new server, such as example.com -- and then you can give any of the domain names on your server an alias, such as mydomain.example.com.

Aliases can be configured in System Settings -> Server Templates -> Default -> Virtual Server Creation, and there you can set "Automatically create alias domain".

The UserDir section ought to be removed from the documentation, as it's not as easy to setup anymore as it appears in that document. I'll be reviewing that further and either adding to the documentation, or more than likely removing it entirely.

We recommend aliases, as they're easy to configure and they're secure, it just requires a spare domain name (though you could even use a sub-domain if you wanted).

Not sure I understand that very well... let me feed back to you:

in the DNS... we would take an unused domain... say jnanadana.org

point this to the new IP on the new cloud instance... which is shared by 5 different domains.

So then if I want to preview one of the domains that was migrated... I just create an alias domain

"www.jnanadana.org"

the we can use that domain to preview the migrated site?

Did I get that right?

I guess obviously you could only do this one at a time since 5 domains share the same IP (but that's OK...)

OR I will set up 9 subdomains just for test the 9 sites we are migrating

That sounds almost correct, but it can be easy to setup if Virtualmin is configured for that prior to importing your domains.

What you would do is go into System Settings -> Server Templates -> Default -> Virtual Server Creation, and set "Automatically create alias domain" to "jnanadana.org".

Then, when you import or create a domain named, say, example.com -- Virtualmin will create an alias for that domain named example.jnanadana.org.

You can then browse to example.jnanadana.org.

That will work for all domains on your server -- you will have aliases for each domain, looking something like MY-DOMAIN-ALIAS.jnanadana.org.

Does that make more sense?

Yes that make sense.

1) of course the alias domain needs to be pointing to the IP that is propagated globally...right? OR could onejust set that up in his local hosts file on hgis own box and map

45.46.96.123 example.jnanadana.org

2) I've already imported all domains except one... :-( so how do i set up the alias domains now?

I went ahead and tried the "intuitive" route... navigated to a domain the tab under virtual Min

Create Virtual Server --> Click on link for "alias of wailuarivernursery.com"
--> enter "test.wailuarivernursery.com --> uncheck "Create website --> save

now go to my local host file on my own box

enter

45.85.46.128 test.wailuarivernursery.com

then go to my browser and to go for it.

OK so, it works to the point of getting back to the Linode instance but I get:

"Forbidden

You don't have permission to access / on this server."

So... still missing something.

The "Setup website for domain" option would need to be enabled... otherwise, it won't be considered an alias in Apache.

Awesome!

I created

test.domaina.com test.domainb.com

etc. for the domains on the shared IP

and added them to my local hosts file

45.83.123.257 test.domaina.com

45.83.123.257 test.domainb.com

it works!

You have been so patient! Can I help by writing up the documentation for you on this one?

What is the mark up we use here for code? you use standard Markdown? on this forum?

So far so good, migration is going well. Most web sites are up and running on the new linode instance and the setting of an alias and adding that to the host file on my own machine works great for previews

But I have on "stubborn" case... where my.domain.org

returns this on any attempt to preview or look at any file in any directory.

Forbidden You don't have permission to access / on this server.

I restored this domain from a VirtualMin back up, like all the others. But even if I create a simple /home/mydomain/public_html/index.html file with "Hello, My Domain is Up and Running."

and try to preview, either externally using the "test.mydomain.org" method or from the "Preview web site" option inside VirtualMin we get the same error.

Forbidden You don't have permission to access /index.html on this server.

I'm trying to examine carefully all settings in VirtualMin for this domain, and compare them with settings for the domains that do work on the same shared IP... but I can't find any differences. (Obviously there must be some difference) the home/mydomain owneruser and the owner and group for all files match... permissions all look correct...

So why is access forbidden?

hhmm. I'll next look at the HTTPD directives...

I just found one discrepancy... on this one "myDomain.org" in the web site configuration, if I check the "Users and Groups" it shows "run CGI as unix user 1009 group 1004

But the other sites have the same number in both of these fields. Could there be some confusion at the unix user level?

For that particular domain that you're having problems with -- do you see the access attempt in the Apache log files for the domain in $HOME/logs/access_log?

The goal here is to make sure that requests for this domain are going to the correct site.

cd /home/wikihinduorg/logs tail -fn100 access_log

I get a line appended on each attempt to get to the "test.wikihindu.org" alias that I set up (as suggest above) which works for other domains.

24.43.231.222 - - [05/May/2015:21:19:44 -0700] "GET / HTTP/1.1" 403 409 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:37.0) Gecko/20100101 Firefox/37.0"
24.43.231.222 - - [05/May/2015:21:19:52 -0700] "GET / HTTP/1.1" 403 409 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:37.0) Gecko/20100101 Firefox/37.0"
24.43.231.222 - - [05/May/2015:21:20:04 -0700] "GET / HTTP/1.1" 403 409 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:37.0) Gecko/20100101 Firefox/37.0"

Switching to the error_log we see:

[Tue May 05 21:19:44.641090 2015] [authz_core:error] [pid 1865] [client 24.43.231.222:49256] AH01630: client denied by server configuration: /home/wikihinduorg/public_html/
[Tue May 05 21:19:52.312367 2015] [authz_core:error] [pid 12083] [client 24.43.231.222:49257] AH01630: client denied by server configuration: /home/wikihinduorg/public_html/
[Tue May 05 21:20:04.813561 2015] [authz_core:error] [pid 1863] [client 24.43.231.222:49258] AH01630: client denied by server configuration: /home/wikihinduorg/public_html/

which is a totally new thing for me! I looked thru as many settings as I could find in VirtualMin for this domain and I'm not seeing something that would cause this (frankly though, I may still not be looking in the right place)

Interesting...I decided to leave my tail of the error log running...

I'm also seeing another strange error on this virtual server/domain (which was backed up and restored using exact same procedure as all the ones that do work...)

[Tue May 05 21:42:45.219944 2015] [core:error] [pid 12094] [client 178.137.19.143:59202] AH00082: an unknown filter was not added: includes
[Tue May 05 21:42:46.152342 2015] [core:error] [pid 1863] [client 178.137.19.143:54421] AH00082: an unknown filter was not added: includes
[Tue May 05 21:42:46.868985 2015] [core:error] [pid 6572] [client 178.137.19.143:62365] AH00082: an unknown filter was not added: includes