2 questions - 1 topic: Migrate Virtualmin - to LXD?!

7 posts / 0 new
Last post
#1 Mon, 09/23/2019 - 17:17
6tRSqw_e

2 questions - 1 topic: Migrate Virtualmin - to LXD?!

tl;dr: I cannot migrate (because backup + restore is broken) my Virtualmin.
If I knew who to turn to. If I knew, I'd have already bought them/you a beer.

Introduction: I am a veteran Linux, BSD, user, sysadmin, power user. Not to brag, just wasted too many years of my life fixing my xorg config. Get the joke? But, for real. So. I was paying for a hosting because I didn't want to bother with the whole stack of things. I did set up my own host setup at some point, several times in my life, but there is so many things to look out for, I was looking to streamline that. That's where the hosting came in. Long story short, the hosting got hacked. I had enough at that point, and went looking for a suite.

There was ISPConfig, cPanel, some others and Virtualmin. Webmin looked always nice, so Virtualmin felt great out of the box.
Friend told me he was using it for years, he is happy with it, can back it up, restore. I was like, great!

I purchased a license (10) first, then a (50), and now another. So happy three license owner here, yayyy.

So, this is a 2-in-1, but they are related, and I don't think it deserves two topic, as they are related.
I'll try to be as short as possible to avoid posting a 5 page old thread, so here it goes.

  • Basically I used Virtualmin for months now on a virtual machine under Hyper-V.
  • Start was rough, there were a few bugs here and there, I even wrote a document about it, but you know, small things. like milter not working.
  • So, I set it all up, fixed the quirks, I was up and running. I did backups, I did script to backup, all the stuff. Which all turned out to be utterly useless. But hey. REMEMBER KIDS, ALWAYS TEST YOUR BACKUPS!
  • Time went by, and I was paying a bit much for this server. Like a simple VPS could do all this. Meanwhile I also learned how to work with LXC.
  • How great it would be to rent a new virtual server, put LXC/LXD on it, and run Virtualmin in that. Oh boy. Oh, the joy.
  • So I got a new virtual server, started a new Ubuntu 18.04. All superb!
  • Here is where the love story becomes grim. Individual backup files just don't work.. at least for me.
  • So I tried importing the individual smaller files. Nothing. It just didn't import anything. Great!
  • I started doing a full backup. Error. Ruh-roh. What could it be?

DomainExample.com has user "admin".
Uh, okay? I did NOT name the users, or groups, or anything? But sure it has? So what?
Well, turns out, until I fix that, I will NOT get a complete backup until I fix that.

Maybe this is why the individual backup / restore failed too.
What was the problem?
Turns out, Virutalmin has it's own conciousness. And when you add a new virtual server, it can name it three ways. Three is what I saw, there may be others.

Let's call our domain "DomainA.com".
User/group can be: domaina
Path can be: /home/domaina
User/group can be: domaina.com
Path can be: /home/domaina.com
User can be "admin" ! I am not kidding.

  • So I had to rename these. How do you do that? Well, the Change Domain feature does NOT work if you have sub-domains.
  • You can make it work if you disable DNS under each sub-domain, for the parent you rename.
  • But oh joy, the rename will NOT be complete. Once Virtualmin says it's done, it'll simply crash. It will start the eternal slumber.
  • Turns out, the subs were not renamed. I had to use "grep" and "sed", to look for all occurences for "/home/domain/" and replace with "/home/domain.com/".
  • So you think our journey has come to an end? Worry not, there is lot more.
  • At this point, I had the domains renamed, and the old host worked fine. Oh my.
  • So, backup again. Back it all up, then SCP from new host, then try to load it back up.
  • Restore complete. Yay! Oh wait... some sites are missing.... what now... well, this remains a mistery to this day. Some sites just did not import, and there were 0 errors.
  • Oh well, I'll just use the command line. So I did. Full backup again there, move it over. Import from the new site. You writing notes? Backup on the live server, import on website on new one. Good.
  • All the sites. All the stuff. OH BOY. Minus the tunes I did, like fixing milter, but you know, nothing biggie. So I change DNS, check the site... it's not loading. HTTPS won't load.
  • At this point I spent I think 8-10 hours digging without luck. No webmin, apache, or any error. Nothing. Ports all good, lxc forward good, yet it's not working.

At this point... Big Brain Time.
If they are both Ubuntu 18.04. (new and old).. They both use Virtualmin, they both are updated.... can't I just like copy files over and call it a day?
And so I did. This is NOT a backup, it's not what anyone would want to use live but you get by with what you've got. I was like, I'll just export the lxc container, back that up, all good.

Well, this worked surprisingly well! But I have to start /etc/init.d/webmin by hand. Hmm, most interesting.
In fact, everything loads up, there are the sites, everything. But I have to start everything by hand.
There must be some kind of service setting for systemd that I missed or that gets messed upon copying.

Sooooooooo....
What are the two things here?

  • How do I backup and restore so that it actually works?
  • How do you copy a Virtualmin (so to say) over to a new server and make it work, like the services?

Ps.: Yes, each and evry time I re-deployed the LXC container. Such as between two backups, I redeployed it, did a clean virtualmin install, then proceeded. I didn't want to risk contamination.

Mon, 09/23/2019 - 17:17
6tRSqw_e

I realize that this post is still messy - I mean just look at the sheer size of the "short post" and the timeline of events - so if you have any questions or need more details on any part, ask away. I'll try my best to reply ASAP.

Ps.: Once I fix this, I'll try to dig up my "bug list" and the quirks you have to look out for on LXD. It's no biggie, like you have to go into Dovecot settings and set it to all interfaces. But really, that's easy.

Mon, 09/23/2019 - 22:18
noisemarine

This is the recommended way to migrate a Virtualmin server. Are you saying none of this works?

https://www.virtualmin.com/documentation/system/migrate

You say your backup and restore fails, but you don't say what steps you take to perform either.

Tue, 09/24/2019 - 03:22 (Reply to #3)
6tRSqw_e

Yeah that guide, does not work. Simply put. :D
- The stock backup can have conflicts due to the various user/group names. See below.
- The CLI backup/restore feature is broken. I don't mean to offend the devs here by any means, it's just a thing that does not function. It says full backup is done, restore says full restore is complete, no errors, and you are missing half the settings/files/sites. gg.

0) Per site backups are absolutely broken. Try to restore them. Good luck! (It won't work.) 1) As I mentioned in my post/guide, account/group names and paths get created randomly by default. It's a mess.
2) Now renaming a simple site with no sub-domain is easy. Few minutes tops.
3) But who doesn't have sub domains? So that's where the problem comes in...
4) See above. You have to disable DNS feature for each sub. Rename main. Virtualmin crashes. Then you rename by hand with grep and nano/mc/sed/whatever and THEN you can start it up again. 5) Now you have a system with a backup that actually works! YAY!
6) At this point, your backups will still not function, don't worry. The backup will finish, the restore will finish. But.
You still don't have a working Virtualmin. Yayyyyy!
7) do a: sudo virtualmin fix-domain-permissions --all-domains
Now, you still have to.... - You have to edit the /etc/postfix/main.cf file to include the new hostname as local destination.
- You have to change all /etc/apache2/sites-available directives to just be "Any" (ie.: listen on all interface/ports). You can change the IP here as well to the new one, but that's a hassle. Both method works, I went with Any. You simply go to each entry, and make it
- Go into Webmin->Dovecot->All interfaces.

At this point you have web (http, https) and SQL. And Virtualmin. Congrats, you sweated blood and tears. But it's not over..
The emails are still not working properly. I'm still working on this one. Once I have a complete guide I will post it.

As I said in the beginning, this could all annoy me, as this whole thing means noobs will not be able to do a backup and restore at all.
Like it just gives a false sense of security, and practically all Virtualmin installation by itself is a risk. But, do the other panels migrate / restore & backup fine?
I do not know. Maybe they are all broken. So we do with what we have.

As the saying goes, always test your backup. Rent a virtual machine from anyone, or fire up your own. Install a new Ubuntu 18.04 for example, and try to restore. A backup without test is no backup. So they say. And boy, were they right...

Ps.: Not to complain, but another pet peeve is that I have a pro sub but it means absolutely nothing. There is no support. I have a 10-site license that I bought as a demo and I could not cancel it ever since. I opened a ticket months ago, no replies, the license is still there, I'm still paying. Most likely I'll have to get a new credit card/account at some point unless.... lol.

Tue, 09/24/2019 - 03:24
6tRSqw_e

Note: I did not modify the core installation by any means. Not like that would be unsupported (changing a config file here or there). I simply added some Postfix safeguards in the config through the Webmin interface, and I just fixed the milter thing that threw errors by default. IIRC, when I installed Virtualmin (2 months ago?), the email signing was also broken. Yeah it was. I had to change /etc/default/dkim something something to fix it... :/

Tue, 09/24/2019 - 04:03
noisemarine

I'm not sure what you did to your system/s, but I've not had any of those problems with migrating. If there were a problem with Virtualmin, you can bet there'd be a flood of posts about it. I've not used LXD however, only OpenVZ and KVM.

I have had two hiccups that test restores/migrations have found:

  • mysql databases created as the root/admin user were not associated with the virtual server I was using them with, and don't seem to end up in any backup.

  • I have two Virtualmin servers and backup to a remote server. I was backing up to a single directory and the two Virtualmin configs were clobbering each other, even though the virtual server backups were unique. I now backup to separate directories (one for each server).

The upshot is, Virtualmin isn't going to prevent you from doing something stupid. The migration process works, but if you hack and experiment outside of Virtualmin/Webmin, when it breaks you get to keep both pieces.

Sorry I don't have better news for you. Good luck with getting it all straightened out. :)

Tue, 09/24/2019 - 04:07 (Reply to #6)
6tRSqw_e

I know this is not an easy topic, I mean everyone will encounter different quirks. I realize the size of the task that Virtualmin tries to take on.

With all the renames sorted out, things seem fine.
I will check what you posted as well, seems like great tips.
Honestly, there should be like a community wiki, and we could make a "Pitfalls of Backup & Restore, what to look out for" article with all the bits. I mean I'll still post it on my blog and the forums.... in the hopes of folks will find it.

And "people would complain". Eh, who knows. A regular dedi lasts a long time. I mean I just backed up without testing. Trusted the software. Don't do that.