migration from 3.98.gpl to 4.10.gpl fails

10 posts / 0 new
Last post
#1 Wed, 09/17/2014 - 18:09
cjfsyntropy

migration from 3.98.gpl to 4.10.gpl fails

I'm migrating a FreeBSD 8.0 system running webmin 1.620 and virtualmin 3.98.gpl to a fresh Debian Wheezy system running webmin 1.701 and virtualmin 4.10.gpl. There are about 54 domains on the old system.

I installed a fresh Wheezy system. I added my favorite utilities including my own firewall scripts. I added the following to apt/sources.list:

# usermin & webmin
deb http://download.webmin.com/download/repository sarge contrib
 
# procmail-wrapper virtualmin-base
deb http://software.virtualmin.com/gpl/debian/ virtualmin-wheezy main
deb http://software.virtualmin.com/gpl/debian/ virtualmin-universal main

I ran install.sh

I ran out of disk space in /tmp, so I "mv /tmp/.webmin /var/tmp" and in Webmin -> Webmin -> Webmin Configuration -> Advanced Options, and changing "Temporary files directory" to /var/tmp/.webmin

I followed the instructions at http://www.virtualmin.com/documentation/system/migrate

I scp'ied the backups from the FreeBSD system.

I ran virtualmin restore-domain --source /var/migration-backups/virtualmin.tar.gz --all-virtualmin No apparent errors:

Checking for missing features ..
.. all features in backup are supported
 
Checking for errors in backup ..
.. no errors found
 
Starting restore..
Extracting backup archive file ..
.. done
 
Restoring Virtualmin settings ..
    Restoring Virtualmin configuration ..
    .. done
 
    Restoring templates and plans ..
    .. done
 
    Restoring email templates ..
    .. done
 
    Restoring custom fields, links, categories and shells ..
    .. done
 
    Restoring custom script installers ..
    .. done
 
    Restoring scheduled backups and keys ..
    .. done
 
    Restoring FTP directory restrictions ..
    .. done
 
    Restoring DKIM settings ..
    .. not installed
 
    Restoring greylisting settings ..
    .. not installed
 
    Restoring rate limiting settings ..
    Restoring mail server configuration ..
    .. done
 
.. done
 
Applying FTP server configuration ..
.. done
 
Restore completed successfully.

I ran virtualmin restore-domain --source /var/migration-backups/ --all-domains --all-features There is just one error at the end:

Checking for missing features ..
.. all features in backup are supported
 
Checking for errors in backup ..
.. no errors found
 
Starting restore..
Extracting backup archive files ..
.. done
 
Restoring backup for virtual server chinamedicinemall.com ..
    Restoring virtual server password, quota and other details ..
    .. done
 
    Updating administration password and quotas ..
    .. done
 
    Restoring Cron jobs ..
    .. done
 
    Extracting TAR file of home directory ..
    .. done
 
    Setting ownership of home directory ..
    .. done
 
    Restoring Apache virtual host configuration ..
    .. done
 
    Checking restored PHP execution mode ..
    .. mode Apache mod_php OK for this system
 
    Restoring allowed MySQL hosts ..
    .. done
 
    Re-loading MySQL database chinamed ..
    .. done
 
    Restoring Webmin ACL files ..
    .. done
 
    Re-creating mail and FTP users ..
 Error: makemap failed : sh: 1: makemap: not found
 
 Error
 -----
 makemap failed : sh: 1: makemap: not found

In Debian makemap is provided by the sendmail-bin package, but that conflicts with postfix which is required for virtualmin. So I do not know how to resolve that error.

Is that error the reason why only 1 of the 45 domains got migrated (according to the "List Virtual Servers" screen in the GUI)?

Is there a listing of all the places virtualmin stores data so I can audit the restore and determine what got migrated and what is missing?

Why is it not seeing the other 55 files like *.{info,dom,tar.gz} in /var/migration-backups/ ? Should I try to specify each domain one at a time? How?

Wed, 09/17/2014 - 22:46
andreychek

Howdy,

Hmm, it shouldn't actually need makemap on a Postfix system, and I wouldn't recommend attempting to install/use Sendmail, that just tends to make things harder :-)

If you look in System Settings -> Virtualmin Config, what is "Mail server to configure" set to on that first screen there?

If your last server was using Sendmail, and you imported the old Virtualmin settings, it may be trying to configure the email users with Sendmail rather than Postfix. You could always change that option to use Postfix.

It sounds like you're attempting the migration the right way though -- just pointing Virtualmin at a directory full of backups. The .dom and .info files are just metadata, the .tar.gz files are your actual backups.

However, by default, if any error occurs during a restore the restore process will stop and not continue on with the other domains.

-Eric

Tue, 09/23/2014 - 15:13 (Reply to #2)
cjfsyntropy

System Settings -> Virtualmin Config -> "Mail server to configure" solved the problem. This is a sendmail to postfix migration too, unfortunately. And that has created another problem. Although virtualmin restore-domain --source /var/migration-backups/ --all-domains --all-features finished without further errors. I now have a convoluted configuration which I'm not sure how to straighten out.

When I go to Postfix Mail Server -> User Mailboxes, mailboxes appear to be in ~USER/Maildir. But postfix delivers mail to /var/mail/USER in mbox format. What kind of postfix configuration do I need for Virtualmin? I assume the migration got it wrong because the old server was sendmail. I assume the migration, therefore, does not touch my local postfix configuration. So I can hack that to make it work without fear that the migration will change it. Is there anything the migration does that I need to be cautious about?

Does virtualmin prefer Maildir or mbox format? Why did the migration configure Dovecot as Maildir? It also put the user's mail from the old server into ~USER/Maildir. Is Maildir the Virtualmin way to do mail? Will it be easier to migrate postfix to Maildir or Dovecot to mbox?

Finally, this was a test run of the migration. I plan to run the virtualmin restore-domain ... command again for the actual cutover. Is it idempotent? How can I finagle the migration so that mail works after the cutover.

Tue, 09/23/2014 - 22:10
andreychek

Howdy,

Virtualmin configures the various services on your system to use Maildir by default. It does that during the installation of Virtualmin. While you don't have to use that, that's the preferred format.

What format are your current mailboxes in? After performing the migration, are you saying it moved your existing mailboxes over to Maildir?

Virtualmin configures Postfix to use procmail for email delivery... you can verify that email is being delivered to Maildir directories by looking at /etc/procmailrc.

Regarding the restore-domain command -- you can run that as often as you want, though note that doing so doesn't remove the original homedir contents, though it will overwrite files that exist. I believe the "--delete-existing" option should delete the existing homedir contents first.

-Eric

Tue, 09/23/2014 - 22:24 (Reply to #4)
cjfsyntropy

Maildir is fine by me. But currently new mails are delivered to /var/mail/MBOX instead of $HOME/Maildir. My /etc/procmailrc looks like this:

LOGFILE=/var/log/procmail.log
TRAP=/etc/webmin/virtual-server/procmail-logger.pl
DEFAULT=/var/spool/mail/$LOGNAME
ORGMAIL=/var/spool/mail/$LOGNAME
DROPPRIVS=yes

How should it look for a "normal" Virtualmin configuration? How should I use the GUI to choose Maildir in a Virtualmin compatible way?

--delete-existing sounds like exactly what I need: thanks.

Wed, 09/24/2014 - 22:54 (Reply to #5)
cjfsyntropy

When I run virtualmin restore-domain --delete-existing --source /var/migration-backups/ --all-domains --all-features, I get these errors:

Checking for missing features ..
.. all features in backup are supported
 
Checking for errors in backup ..
.. no errors found
 
Starting restore..
Extracting backup archive files ..
.. done
 
Deleting virtual server chinamedicinemall.com ..
    Deleting mail aliases ..
    .. done
 
    Deleting Webmin login ..
    .. done
 
    Deleting MySQL database chinamed ..
    .. done
 
    Deleting MySQL login ..
    .. done
 
    Deleting virtual website ..
    .. done
 
    Deleting home directory ..
    .. done
 
    Deleting administration user ..
    .. done
 
    Deleting administration group ..
    .. done
 
    Deleting server details for chinamedicinemall.com ..
    .. done
 
    Re-loading Webmin ..
    .. done
 
    Applying web server configuration ..
    .. done
 
.. done
 
Re-creating virtual server chinamedicinemall.com ..
    Creating administration group chinamed ..
    .. done
 
    Creating administration user chinamed ..
    .. done
 
    Creating aliases for administration user ..
    .. done
 
    Adding administration user to groups ..
    .. done
 
    Creating home directory ..
    .. done
 
    Creating mailbox for administration user ..
    .. done
 
    Adding new virtual website ..
    .. done
 
    Performing other Apache configuration ..
    .. done
 
    Creating MySQL login ..
    .. done
 
    Creating MySQL database chinamed ..
    .. done
 
    Creating Webmin user ..
Use of uninitialized value in concatenation (.) or string at /usr/share/webmin/acl/acl-lib.pl line 468.
Use of uninitialized value in concatenation (.) or string at /usr/share/webmin/acl/acl-lib.pl line 468.
Use of uninitialized value in concatenation (.) or string at /usr/share/webmin/acl/acl-lib.pl line 654.
Use of uninitialized value in concatenation (.) or string at /usr/share/webmin/acl/acl-lib.pl line 654.
    .. done
 
    Applying web server configuration ..
    .. not running!
 
    Re-loading Webmin ..
    .. done
 
    Saving server details ..
    .. done
 
Restoring backup for virtual server chinamedicinemall.com ..
    Restoring virtual server password, quota and other details ..
    .. done
 
    Updating administration password and quotas ..
    .. done
 
    Restoring Cron jobs ..
    .. done
 
    Extracting TAR file of home directory ..
    .. done
 
    Setting ownership of home directory ..
    .. done
 
    Restoring Apache virtual host configuration ..
    .. done
 
    Checking restored PHP execution mode ..
    .. mode Apache mod_php OK for this system
 
    Restoring allowed MySQL hosts ..
    .. done
 
    Re-loading MySQL database chinamed ..
    .. done
 
    Restoring Webmin ACL files ..
    .. done
 
    Re-creating mail and FTP users ..
    .. done
 
    Re-creating mail aliases ..
    .. done
 
    Restoring mail files ..
    .. done
 
    Moving additional mail folders ..
    .. done
 
    Restoring mail and FTP user Cron jobs ..
    .. done
 
    Updating Webmin user ..
    .. done
 
[It continues like this for about 11 more virtual domains, then it fails on this restore:]
 
Deleting virtual server campbellvoiceover.com ..
    Deleting mail aliases ..
    .. done
 
    Deleting Webmin login ..
    .. done
 
    Deleting MySQL database campbell ..
    .. done
 
    Deleting MySQL login ..
    .. done
 
    Deleting virtual website ..
    .. done
 
    Removing from email domains list ..
    .. done
 
    Deleting administration user ..
    .. done
 
    Deleting administration group ..
    .. done
 
    Deleting server details for campbellvoiceover.com ..
    .. done
 
    Applying web server configuration ..
    .. not running!
 
    Re-loading Webmin ..
    .. done
 
.. done
 
Re-creating virtual server campbellvoiceover.com ..
    Creating administration group campbell ..
    .. done
 
    Creating administration user campbell ..
    .. done
 
    Creating aliases for administration user ..
    .. done
 
    Adding administration user to groups ..
    .. done
 
    Creating home directory ..
    .. done
 
    Creating mailbox for administration user ..
    .. done
 
    Adding to email domains list ..
    .. done
 
    Adding default mail aliases ..
    .. done
 
    Adding new virtual website ..
    .. done
 
    Performing other Apache configuration ..
    .. done
 
    Creating MySQL login ..
    .. done
 
    Creating MySQL database campbell ..
    .. done
 
    Creating Webmin user ..
Use of uninitialized value in concatenation (.) or string at /usr/share/webmin/acl/acl-lib.pl line 468.
Use of uninitialized value in concatenation (.) or string at /usr/share/webmin/acl/acl-lib.pl line 468.
Use of uninitialized value in concatenation (.) or string at /usr/share/webmin/acl/acl-lib.pl line 654.
Use of uninitialized value in concatenation (.) or string at /usr/share/webmin/acl/acl-lib.pl line 654.
    .. done
 
    Applying web server configuration ..
    .. not running!
 
    Re-loading Webmin ..
    .. done
 
    Saving server details ..
    .. done
 
Restoring backup for virtual server campbellvoiceover.com ..
    Restoring virtual server password, quota and other details ..
    .. done
 
    Updating administration password and quotas ..
    .. done
 
    Restoring Cron jobs ..
    .. done
 
    Extracting TAR file of home directory ..
    .. done
 
    Setting ownership of home directory ..
    .. done
 
    Restoring Apache virtual host configuration ..
    .. done
 
    Checking restored PHP execution mode ..
Error: Failed to copy /etc/php5/apache2/php.ini to /home/campbell/etc/php5/php.ini : cp: cannot create regular file `/home/campbell/etc/php5/php.ini': No such file or directory
 
Error
-----
Failed to copy /etc/php5/apache2/php.ini to /home/campbell/etc/php5/php.ini : cp: cannot create regular file `/home/campbell/etc/php5/php.ini': No such file or directory
 
-----

What could have gone wrong? I mv'd /home/campbell out of the way and repeated but it did not work.

I also tried with and without the updated virtualmin.tar.gz* files in /var/migration-backups: I worry about those files changing my postfix mailer again.

Wed, 09/24/2014 - 10:52
andreychek

Howdy,

Hmm, what you're seeing there is a non-default setup, and things shouldn't appear that way on a freshly installed system.

However, we can reconfigure things so that they use Maildir.

What is the output of this command:

postconf -n

Wed, 09/24/2014 - 11:35 (Reply to #7)
cjfsyntropy

It is a freshly installed system. I added the CA and certs for TLS, but otherwise did not change it from a standard Debian install followed by running Virtualmin's install.sh and the migration scripts documented above.

alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases
allow_percent_hack = no
append_dot_mydomain = no
biff = no
broken_sasl_auth_clients = yes
config_directory = /etc/postfix
home_mailbox = Maildir/
inet_interfaces = all
mailbox_command = /usr/bin/procmail-wrapper -o -a $DOMAIN -d $LOGNAME
mailbox_size_limit = 0
mydestination = polka.omnikoz.com, localhost.omnikoz.com, localhost
myhostname = polka.omnikoz.com
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
myorigin = /etc/mailname
readme_directory = no
recipient_delimiter = +
relayhost =
sender_bcc_maps = hash:/etc/postfix/bcc
smtp_tls_CAfile = /etc/lfrr/ca/ca.crt
smtp_tls_cert_file = /etc/lfrr/ca/thishost.crt
smtp_tls_key_file = /etc/lfrr/ca/thishost.key
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_use_tls = yes
smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
smtpd_recipient_restrictions = permit_mynetworks permit_sasl_authenticated reject_unauth_destination check_policy_service inet:127.0.0.1:10023
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
smtpd_tls_CAfile = /etc/lfrr/ca/ca.crt
smtpd_tls_cert_file = /etc/lfrr/ca/thishost.crt
smtpd_tls_key_file = /etc/lfrr/ca/thishost.key
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtpd_use_tls = yes
virtual_alias_maps = hash:/etc/postfix/virtual
Wed, 09/24/2014 - 12:07
andreychek

Okay, your Postfix config looks good, so it's just the procmailrc that needs tweaked.

The /etc/procmailrc file on a new setup should look something like this:

LOGFILE=/var/log/procmail.log
TRAP=/etc/webmin/virtual-server/procmail-logger.pl
:0wi
VIRTUALMIN=|/etc/webmin/virtual-server/lookup-domain.pl $LOGNAME
EXITCODE=$?
:0
* ?/usr/bin/test "$EXITCODE" = "73"
/dev/null
EXITCODE=0
:0
* ?/usr/bin/test "$VIRTUALMIN" != ""
{
INCLUDERC=/etc/webmin/virtual-server/procmail/$VIRTUALMIN
}
ORGMAIL=$HOME/Maildir/
DEFAULT=$HOME/Maildir/
DROPPRIVS=yes
:0
$DEFAULT

Try changing your procmailrc to the above... once you do that, any new emails should be delivered using Maildir.

-Eric

Wed, 09/24/2014 - 19:27 (Reply to #9)
cjfsyntropy

Thanks for your help Eric, but it didn't quite work.

On my system there is no file /etc/webmin/virtual-server/lookup-domain.pl

There is a file /usr/share/webmin/virtual-server/lookup-domain.pl

Is that the correct file? Should I copy it to /etc/webmin/virtual-server/ or just reference the version in /usr/share/webmin/virtual-server/?

Also, on my system there is no directory /etc/webmin/virtual-server/procmail/

Maybe the old server did not have procmail and so the migration failed to create these files? Do I need to generate a site specific procmail file for each of the 54 virtual domains? How would I do that? Is it documented anywhere?

Topic locked