Virtualmin 3.99 upgrade failed on Debian Lenny

After installing suggested update, Virtualmin 3.99 update failed, I think because of broken webmin. At upgrade I noticed that Debian moved their Lenny repositories to archive, as Lenny is EOL. After fixing repositories apt-get update finished fine. But apt-get upgrade (with webmin-virtual-server as only package) failed with error that "Webmin does not appear to be installed on your system.". And installing Webmin with apt-get says that its already installed.

Earlier at initial upgrade, this the error that Webmin generated:

Preparing to replace webmin 1.620 (using .../archives/webmin_1.630_all.deb) ...
Unpacking replacement webmin ...
dpkg: error processing /var/cache/apt/archives/webmin_1.630_all.deb (--unpack):
unable to create `./usr/share/webmin/mysql/help/table_form.nl.html': No space left on device
dpkg-deb: subprocess paste killed by signal (Broken pipe)
grep: /etc/webmin/miniserv.conf: No such file or directory
Webmin install complete. You can now login to http://ns1:/
as root with your root password, or as any user who can use sudo
to run commands as root.
Errors were encountered while processing:
/var/cache/apt/archives/webmin_1.630_all.deb

Also this is the error of Virtualmin:

(Reading database ... 88720 files and directories currently installed.)
Preparing to replace usermin 1.540 (using .../archives/usermin_1.550_all.deb) ...
Unpacking replacement usermin ...
Preparing to replace webmin 1.620 (using .../archives/webmin_1.630_all.deb) ...
Unpacking replacement webmin ...
Preparing to replace webmin-virtual-server 3.99.gpl (using .../webmin-virtual-server_4.00.gpl_all.deb) ...
Webmin does not appear to be installed on your system.
This package cannot be installed unless the Debian version of Webmin
is installed first.
dpkg: error processing /var/cache/apt/archives/webmin-virtual-server_4.00.gpl_all.deb (--unpack):
subprocess pre-installation script returned error exit status 1
grep: /etc/webmin/config: No such file or directory
grep: /etc/webmin/config: No such file or directory
usage: copyconfig.pl <os>[/real-os] <version>[/real-version] <webmin-dir> <config-dir> [module ...] at /usr/share/webmin/copyconfig.pl line 7.
grep: /etc/webmin/webmin.acl: No such file or directory
head: cannot open `/etc/webmin/webmin.acl' for reading: No such file or directory
grep: /etc/webmin/webmin.acl: No such file or directory
grep: /etc/webmin/webmin.acl: No such file or directory
Error: Script was not run with full path (failed to find /usr/share/webmin/run-postinstalls.pl under )

-----
Script was not run with full path (failed to find /usr/share/webmin/run-postinstalls.pl under )
-----
Errors were encountered while processing:
/var/cache/apt/archives/webmin-virtual-server_4.00.gpl_all.deb

I did not want to try to remove/install Webmin, because I don't want to make more trouble. Now my Virtualmin does not work and I also client receive an email with error, but email actually works:

Command died with status 127: "/usr/bin/procmail-wrapper -o -a $DOMAIN -d $LOGNAME'

And also error in procmail log:

From MAILER-DAEMON  Wed Jun 12 11:25:02 2013
Subject: Undelivered Mail Returned to Sender
  Folder: /root/Maildir/new/1371029102.18138_0.ns1                         2685
/bin/sh: /etc/webmin/virtual-server/procmail-logger.pl: No such file or directory

Could you help me how to fix this ?

Regards, Petar

Status: 
Active

Comments

Howdy -- it looks like the initial issue is that your hard drive was out of space. I see this error while upgrading Webmin:

No space left on device

That may have triggered a problem with the Webmin package that is causing the additional issues that you are seeing.

The first question is -- do you have backups of your Virtual Servers and perhaps your filesystem?

Second, what output does this command produce:

ls /etc/webmin/

I was thinking the same, but obviously there is enough space, unless its needed more than 600 MB on root.

Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda2            1.9G  1.2G  592M  67% /
tmpfs                 641M     0  641M   0% /lib/init/rw
udev                   10M  532K  9.5M   6% /dev
tmpfs                 641M     0  641M   0% /dev/shm
/dev/xvda1            183M   43M  131M  25% /boot
/dev/xvdc1            9.9G  5.4G  4.5G  55% /home
/dev/xvdb1            3.0G  634M  2.3G  22% /var
/dev/xvdf1             20G   16G  3.7G  81% /backup

Here is the output of /etc/webmin/:

uninstall.sh  webmin.acl

I have a backup of Virtualmin servers few days ago, when upgrade failed. Also there is Virtualmin configuration backup and file system backup week or two ago

Hmm, what does "df -i" show?

And is this a dedicated server, or VPS? If a VPS, what kind of VPS is it -- Xen, OpenVZ, KVM?

df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/xvda2            122160   99032   23128   82% /
tmpfs                 164006       3  164003    1% /lib/init/rw
udev                  164006     981  163025    1% /dev
tmpfs                 164006       1  164005    1% /dev/shm
/dev/xvda1             48192      40   48152    1% /boot
/dev/xvdc1            655360  222785  432575   34% /home
/dev/xvdb1            196608    9325  187283    5% /var
/dev/xvdf1           1310720    2347 1308373    1% /backup

It's dedicated server with Citrix XenServer and beside this Virtualmin, there is also some Windows machine.

Hrm, it's tough to say why you were getting that error, as it looks like there's plenty of space in there now. It's possible something else was going on at the time which caused space, or perhaps inodes, to temporarily be used up.

However, it appears that the key at the moment is going to be getting everything else back online.

The first step I'd take is to reinstall Webmin, and see if that works properly.

You can do that with this command:

apt-get install --reinstall webmin

Does that install cleanly, without any errors?

If so, the next step would be to do the same for Virtualmin and related modules.

You can see all the Webmin/Virtualmin packages that need reinstalled by running this command:

dpkg -l 'webmin*' | grep ii

After doing that -- are you then able to log into Virtualmin on port 10000? (note that you likely won't see your Virtual Servers yet, that'll be the next step)

Reinstalling Webmin was successful and and I able to login with Webmin. Should I reinstall with same command every module, or maybe is enough to just upgrade with apt-get, because webmin-virtual-server is the only package that needs to be upgraded.

Also is it safe to reinstall the modules, because I need to keep all virtualmin configurations.

Unfortunately, all of the data from /etc/webmin is gone, which is where all those modules store their config information.

In order to begin the process of getting it all back, you'll need to upgrade and/or reinstall all of those packages, which will regenerate a lot of the config data.

So if you can upgrade some of the packages, that's great, but those that you can't upgrade you'd need to reinstall them.

I upgraded webmin-virtual-server and reinstalled all other modules. I can access now Virtualmin, but as you mentioned all configuration is gone. Can I just restore Virtualmin Settings previoulsy created with "Backup and Restore > Scheduled Backups > Features and settings > Virtualmin settings to also backup", to restore only configuration and not touching any of system services and data (MySQL, Apache Websites, Mailboxes ...). If so, is the option "Just re-import server details?" in "Other Restore options" is the thing that I need. I also have backup of Webmin configurations.

There's a few different ways to move forward from here, we'll work with you to find the simplest one.

When you say you have a backup of your Webmin configuration -- what exactly does that backup consist of?

I'm not sure, but I think that there are created with Webmin > Webmin > Backup Configuration Files and all the modules selected from "Modules to backup".

ns1:/# ls /backup/temp/
boot  etc  tmp  var
ns1:/# ls /backup/temp/etc/
apache2  default   group     inetd.conf         localtime       mailname    nsswitch.conf  postgresql   rpc           ssh          webalizer
bind     dovecot   gshadow   init.d             logrotate.conf  mime.types  pam.d          procmailrc   rsyslog.conf  sysctl.conf  webmin
cron.d   fstab     hostname  inittab            logrotate.d     mysql       passwd         proftpd      services      timezone
crontab  ftpusers  hosts     iptables.up.rules  mailcap         network     postfix        resolv.conf  shadow        usermin

content of /etc/webmin (excluded *.acl files)

ns1:/# ls -C /backup/temp/etc/webmin/ | grep -v ".acl"
at                inittab                  postfix      telnet
config            ldap-server              proftpd      usermin
cron              logrotate                qmailadmin   vgetty
fetchmail         mon                      sentry       webalizer
file              mount                    servers      webmin
grub              sshd

And also Virtualmin configurations:

ns1:/# ls /backup/temp/virtualmin/
virtualmin_config           virtualmin_custom_shells    virtualmin_mailserver_dkimkey         virtualmin_mailserver_maincf    virtualmin_templates
virtualmin_custom           virtualmin_email            virtualmin_mailserver_grey            virtualmin_mailserver_mastercf  virtualmin_templates_global
virtualmin_custom_linkcats  virtualmin_mailserver       virtualmin_mailserver_greyclients     virtualmin_scheds               virtualmin_templates_plans
virtualmin_custom_links     virtualmin_mailserver_dkim  virtualmin_mailserver_greyrecipients  virtualmin_scripts              virtualmin_templates_skel_0

I think this are all the necessary configuration.

Thanks! I've asked Jamie for some input on a couple of issues; once we get those issues sorted out, I'll respond here and we'll finish getting your system up and running again.

In the meantime, if your email is bouncing -- you can always temporarily stop Postfix. Mail servers will retry the mail delivery for some time, usually every 20 minutes for a week. And we'll have your system fixed up long before then :-)

Okay, so all of your websites are working, what you're missing is the Virtualmin metadata from /etc/webmin.

What we need to do is extract that from your Virtualmin backups, without making changes to the user data.

Jamie's suggestion of the best way to do that is to use this command:

virtualmin restore-domain --source /path/to/backups/ --all-domains --feature virtualmin --fix

That will restore the metadata from all the backups files located in "/path/to/backups/".

After this command I can see domains in Virtualmin, but some configurations are missing. Where should I start from:

  • Master Admin user is gone, now I'm using root
  • On System Information screen, Post-Installation Wizard is shown
  • Checking Configuration failed with following error:
  • MySQL does not appear to be installed and running on your system, or has not yet been set up properly in Webmin's MySQL Database module. If your system does not use MySQL, it should be disabled in Virtualmin's module configuration page.

    .. your system is not ready for use by Virtualmin.
    During command execution there was and error: /usr/bin/mysqlshow: Access denied for user 'root'@'localhost' (using password: NO)
  • Server Templates and Account Plans are missing
  • On Validate Virtual Servers page, there are few erorrs:
  • Webalizer reporting : Webalizer configuration file /etc/webmin/webalizer/home_domain_logs_access_log.conf was not found
    MySQL database : MySQL database domain does not exist
    Webmin login : Webmin user domain does not exist
  • List of Installed scripts are missing
  • In SpamAssassin Mail Filter page there is error that local configuration file is missing in /etc/webmin/virtual-server/spam/
  • All Scheduled Backups, Virtualmin, Webmin and Filesystem backups are missing and there previous backup logs
  • Also all Webmin users are missing

The command you ran re-populated the metadata for your various Virtual Servers... that will enable us to perform these next steps to finish fixing things up.

The next step -- do you have a backup file named virtualmin.tar.gz?

If so, go ahead and perform a restore of that file... and then let us know when that's finished.

Just restored the virtualmin.tar.gz, re-checked the configuration and it passed. Also I put the password for MySQL modul, and I can see the databases from Virtualmin.

Great! We're almost finished.

Did restoring virtualmin.tar.gz also restore your Webmin/admin users for your various Virtual Servers? Or are those still missing?

Anything else that's missing, we'll need to restore from the Virtualmin backups... going into Backup and Restore -> Restore Backups, pointing it at the backup directory, and restoring the features that are missing.

You won't need to restore databases or home directory data -- but you may need to restore "Virtual server password, description and other details", "Administration user's password, quota and Cron jobs", and "Webmin ACL files".

Feel free to let us know what's missing, and we can help with which features you need to restore.

Everything is restored properly, thank a lot for the help :) I also restored configurations of "Virtualmin Virtual Servers", Webmin Configuration" and "Webmin users" from Webmin backup. Domain validation was showing some Webalizer errors on few domains, but after re-enabling the option, all domain validation completed successfully.

There is one small thing that I need to restore: Webmin > Filesystem Backup Schedules. There were few customized schedules, then I can always create again, but honestly, I'm little lazy after this :)

Thanks again for the help :)

Super, I'm glad all that is working!

We're working on a way to prevent this all from occurring; that should never happen simply because you're low on disk space.

We'll have a fix for that available in the next couple of days... but in the meantime, just to be safe, you may want to make a backup of your /etc/webmin directory. Then, in the off-chance that it occurs again during an update, you can simply replace that directory.