Virtualmin SSH backups started failing.

33 posts / 0 new
Last post
#1 Mon, 11/17/2008 - 09:04
AbitatSupport

Virtualmin SSH backups started failing.

Hi,

We have a daily backup via SSH to a remote server, which used to work just fine. Suddenly, the backup directory's are being created but absolutely nothing is being put inside of them.

What log files should I be checking for errors? I have access to both the virtualmin server and the backup ssh server.

Thanks!

edit On a manual test, several of the virtual servers gave this:

sh: /tmp/.webmin/17112008/gallery.mysite.net.tar.gz: No such file or directory gzip: stdout: Broken pipe tar: -: Cannot write: Broken pipe tar: Error is not recoverable: exiting now

or this:

sh: /tmp/.webmin/17112008/demo.mysite.net.tar.gz: No such file or directory gzip: stdout: Broken pipe tar: -: Wrote only 4096 of 10240 bytes tar: Error is not recoverable: exiting now

In fact, all the tars fail after the first few servers complete successfully, and finally the error below:

Uploading archive to SSH server .. .. upload failed! myuser@x.x.x.x's password: /tmp/.webmin/17112008: No such file or directory Backup failed! See the progress output above for the reason why.

I assume that whatever is causing the original failure, is causing the fails afterward, and eventually, there is no file to upload because of those errors.

But what is causing it originally I guess is what I am wondering.<br><br>Post edited by: AbitatSupport, at: 2008/11/17 09:36

Mon, 11/17/2008 - 09:38
andreychek

Howdy,

Is there enough space in /tmp for the backup? I suspect the error would be different is space were an issue, but I figure I'd ask :-)

Barring that, I might file a support request, I'm not sure what would cause the errors you're getting.
-Eric

Mon, 11/17/2008 - 09:59 (Reply to #2)
AbitatSupport

Well, df says we have 99G left and our sites are certainly not that large.

Hum, okay thanks for the input tho :)

Mon, 11/17/2008 - 10:01 (Reply to #3)
andreychek

Agreed, it definitely doesn't look like a space issue :-)

Yeah, I'd file a request here, Jamie may know the case of that:

http://www.virtualmin.com/bug-tracker/

Mon, 08/24/2009 - 05:38
elsdon

Hey there,

I'm encountering the same issues as above. Has there been any updates to the situation? My /tmp has plenty of space, as does my destination. It's been dying periodically for like the past 3-4 weeks daily.. Never on the same backups. Sometimes when I run them manually, they'll work, but the scheduled/automatic backgrounds seem to fail.

Mon, 08/24/2009 - 08:50 (Reply to #5)
andreychek

Hrm, this particular post was last year, so I don't really recall what came of it :-)

What specific error(s) are you seeing during the backup?

-Eric

Mon, 08/24/2009 - 19:18
elsdon

I'm seeing the exact same error as above.

Basically, after the entire 'site' has been tarballed, it'll try to upload it to remote ftp and it'll pop that error out. So, it's almost as if the tar wasn't created properly and doesn't exist, or something. I haven't been able to localize the error yet.

Mon, 08/24/2009 - 19:22
andreychek

What is the output of these two commands:

  1. mount

  2. df -h

Thanks,

-Eric

Mon, 08/24/2009 - 19:28
elsdon

haha, /tmp has plenty of space, that's the first thing I checked.

.tar.gz can be >4GB now right? It's 2009, I wouldn't imagine that being a limitation still??

It's peculiar, like, not all of the 'site' tarball creations fail.. I'll see a bunch of successful backups created and transferred onto the remote site, just randomly one will fail.

Mon, 08/24/2009 - 19:32
andreychek

Yes, the tar archives can be quite large these days (I don't know about the exact limitations, but someone else was working with a 22GB file just earlier today).

However, it'd still be helpful if you offered the output of those two commands :-)

-Eric

Mon, 08/24/2009 - 20:27
elsdon

sure. It's on a VPS so there's no mount breakdown, just one large filesystem.

[root@la tmp]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/simfs             60G   12G   49G  19% /
[root@la tmp]# mount
/dev/simfs on / type reiserfs (rw,usrquota,grpquota)
/proc on /proc type proc (rw)
/sys on /sys type sysfs (rw)
none on /dev/pts type devpts (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

Total backup size talking like 7.2GB or so (for about 6-7 different 'sites')

This is the error:

Creating TAR file of home directory .. .. TAR failed! sh: /tmp/.webmin/24-08-2009/site.tar.gz: No such file or directory

gzip: stdout: Broken pipe /bin/tar: -: Cannot write: Broken pipe /bin/tar: Error is not recoverable: exiting now

Mon, 08/24/2009 - 22:51
andreychek

Thanks for all the info -- I had sent a message to Jamie to see if he has some thoughts on what's going awry.

-Eric

Mon, 08/24/2009 - 23:05 (Reply to #12)
JamieCameron

Hi guys,

I'd be interested to know exactly what settings you have for these backups that are failing, such as the backup path, format and strftime setting.

Also, do you perhaps have two backups running at the same time to remote destinations?

''

Tue, 08/25/2009 - 19:11
elsdon

I do have two backups that run on the same day.. The site backup runs at midnight every night, and takes about 2 hours to run a full backup.

I have another 'virtualmin' backup that just backs up settings that runs nightly at 2am.

Remote Backup path: /backup/sites/%d-%m-%Y Do strftime-style time substitutions on file or directory name - Yes Transfer each virtual server after it is backed up - Yes

Create destination directory? Yes

Halt the backup immediately [X] Backup level [X]

The backups do have a chance to overrun each other due to the 2 hour interval I've put in. Each time backup runs, does it rm -rf the /tmp directory? I can see that coincedentally running if my site backup is taking longer than usual.

Tue, 08/25/2009 - 23:31 (Reply to #14)
JamieCameron

The two concurrent backups could cause this problem..

Do they both use the same remote backup path?

''

Tue, 01/31/2012 - 04:20
brainbug

Hi!

Are there any news/solutions for this issue?

Suddenly, I'm encountering the same problem. Some servers fail to backup with a message like:

Uploading archive to SSH server my.backupserver.tld .. .. upload failed! /tmp/.webmin/24107-backup/mysite.tld.tar.gz: No such file or directory

All of the 11 sites that fail to backup are "Alias-Servers". I'm not sure if that is just coincidence or not. Other Alias-Servers backup sucessfully. It looks like it's always the same servers that fail to backup. I tried to just delete and recreate the failing servers, but that didn't help.

mount: dev/simfs on / type simfs (rw,relatime,usrquota,grpquota) proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs (rw,relatime) tmpfs on /lib/init/rw type tmpfs (rw,nosuid,relatime,mode=755) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,relatime) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)

df -h /dev/simfs 300G 105G 196G 35% / tmpfs 2,0G 0 2,0G 0% /lib/init/rw tmpfs 2,0G 0 2,0G 0% /dev/shm

Virtualmin: 3.89 gpl

Backup settings: Features to backup: Backup all features Backup destinations: SSH server Additional destination options: Transfer each virtual server after it is backed up Backup format: One file per server Create destination directory?: Yes Action on error: Continue with other features and servers Backup level: Full (all files)

It makes no difference wheter the backup is scheduled or not.

Tue, 02/14/2012 - 01:55
compre

we encounter the same problem.

Error message

Uploading archive to SSH server .. .. upload failed! myuser@x.x.x.x's password: /tmp/.webmin/17112008: No such file or directory /xxxxxxxxx.tar.gz: No such file or directory

Wed, 06/06/2012 - 12:31
glenvistalabs

I am having the exact same problem. It happens consistently, with usually the same 2-3 domains (out of >10 total) failing to remotely back up via SSH in this exact manner. Oddly enough, it happens during every automatic backup, but whenever I tell Virtualmin to perform the backup (manually, apart from its schedule) every domain backs up without any errors.

The directories that don't exist are named like: /tmp/.webmin/[X]-[hostname] Where [hostname] is the name of the server running Virtualmin (Pro) and performing the backups, and [X] is some 4-6 digit number

The user on the remote machine is not privileged (not authorized to perform commands as root via sudo). I intend to keep it this way because Virtualmin stores its password on the machine being backed up, making it extremely insecure. However, the ownership of /tmp/.webmin is root:root, and the permissions are 755 (rwxr-xr-x.), so it shouldn't even be able to write to that directory in the first place (but it's the way that Webmin/Virtualmin automatically set it up, so I'm inclined to think this is the way they should be).

Can anyone explain what is going on here?

Fri, 03/01/2013 - 01:19
dajohn

Same problem here. Is there a solution available?

Fri, 03/01/2013 - 09:24
andreychek

A lot of backup related issues were resolved recently... you may want to make sure you're using the most recent Virtualmin version. Are you using 3.98 there?

-Eric

Fri, 03/01/2013 - 13:28 (Reply to #20)
JamieCameron

Virtualmin version 3.99 will be out in a day or two, and should include a fix for this.

''

Tue, 04/23/2013 - 04:35
Yourname

I have the latest, and nothing got fixed. My error is this for 3 out of the 50 or so domains I have:

Write failed: Broken pipe lost connection

.. completed in 3 minutes, 27 seconds

Thu, 04/25/2013 - 22:59 (Reply to #22)
Yourname

This has now become an everyday occurrence. :(

Fri, 04/26/2013 - 15:15 (Reply to #23)
JamieCameron

Do you have access to the SSH server logs from the remote system? It looks like the connection is being killed, and the logs might explain why.

''

Mon, 04/29/2013 - 05:54 (Reply to #24)
Yourname

Thanks, but it's a whole lotta nothing:

Apr 21 00:54:13 localhost sshd[22882]: Accepted password for backup from 192.168.1.11 port 33354 ssh2
Apr 21 00:54:13 localhost sshd[22882]: pam_unix(sshd:session): session opened for user backup by (uid=0)
Apr 21 00:54:15 localhost sshd[22882]: pam_unix(sshd:session): session closed for user backup
Apr 21 00:54:17 localhost sshd[22902]: Accepted password for backup from 192.168.1.11 port 33355 ssh2
Apr 21 00:54:17 localhost sshd[22902]: pam_unix(sshd:session): session opened for user backup by (uid=0)
Apr 21 00:54:18 localhost sshd[22902]: pam_unix(sshd:session): session closed for user backup
Apr 21 00:54:20 localhost sshd[22922]: Accepted password for backup from 192.168.1.11 port 33356 ssh2
Apr 21 00:54:20 localhost sshd[22922]: pam_unix(sshd:session): session opened for user backup by (uid=0)
Apr 21 00:54:21 localhost sshd[22922]: pam_unix(sshd:session): session closed for user backup
Apr 21 00:54:25 localhost sshd[22942]: Accepted password for backup from 192.168.1.11 port 33357 ssh2
Apr 21 00:54:25 localhost sshd[22942]: pam_unix(sshd:session): session opened for user backup by (uid=0)
Mon, 04/29/2013 - 18:26 (Reply to #25)
JamieCameron

Does the remote user perhaps have a shell like scponly that prevents execution of commands like mkdir ?

''

Tue, 04/30/2013 - 04:50 (Reply to #26)
Yourname

Nope, just a regular bash shell for everyone.

Tue, 04/30/2013 - 22:51 (Reply to #27)
JamieCameron

Since this problem seems to be limited to a few users, it would be useful if I could get remote access to your system to see what is really going wrong. Email me directly at jcameron@virtualmin.com if that is possible.

''

Thu, 04/25/2013 - 01:58
remibruggeman

Hello,

I'm having Virtualmin 3.99 and I am also facing some backup issues. These backup plans are running on 2 servers:

First schedule

  • All virtual servers
  • All features and settings
  • to SSH server File on server: /home/backupname/backup/%Y-%m-%d-%H%M
  • User & Pass set
  • Delete old backups after 35 days
  • Additional Destination options: Do Strftime-style, Transfer each virtual server after it is backed up
  • Backup format: one file per server, create destination directory
  • Action on error: Continue
  • Backup level: FULL
  • E-mail backup report to: my e-mail
  • Scheduled Backup Time: Complex schedule: Every Monday at 03:00

Second backup schedule:

  • All virtual servers
  • All features and settings
  • to SSH server File on server: /home/backupname/backup/INCR/%Y-%m-%d-%H%M
  • User & Pass set
  • Delete old backups after 35 days
  • Additional Destination options: Do Strftime-style, Transfer each virtual server after it is backed up
  • Backup format: one file per server, create destination directory
  • Action on error: Continue
  • Backup level: Incremental
  • E-mail backup report to: my e-mail
  • Scheduled Backup Time: Complex schedule: At cron time 0,30 * * * *

The output of mount command:

[root@turing remi]# mount
/dev/md4 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/md0 on /boot type ext4 (rw)
/dev/md1 on /data type ext4 (rw)
/dev/md2 on /secure type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

Output of df-h:

Filesystem            Size  Used Avail Use% Mounted on
/dev/md4              3.6T  7.1G  3.4T   1% /
tmpfs                  16G  4.0K   16G   1% /dev/shm
/dev/md0              496M   51M  420M  11% /boot
/dev/md1              3.6T  1.3T  2.2T  38% /data
/dev/md2              496M   11M  460M   3% /secure

Error message I get by mail:

   Uploading archive to SSH server XX.XX.XX.XX ..
   .. upload failed! sshbackup@XX.XX.XX.XX's password:
backup.be.tar.gz                                 0%    0     0.0KB/s   --:-- ETAbackup.be.tar.gz                               100%   22KB  21.8KB/s   00:00   
Write failed: Broken pipe
lost connection

or:

   Uploading archive to SSH server XX.XX.XX.XX ..
   .. upload failed! sshbackup@XX.XX.XX.XX's password:
286162_31137_28_backup.pl                       0%    0     0.0KB/s   --:-- ETA286162_31137_28_backup.pl                     100% 6060     5.9KB/s   00:00   
Write failed: Broken pipe
lost connection


.. completed in 2 hours, 0:48 minutes

Let me please also state that both links have at least 100Mbit/s uplinks. (ssh server has 120Mbit/sec, production has 100Mbit/sec)

I hope somebody has any use to this so this might solve a problem.

I do not 'really' have a problem with this since backups are running every 30 min, but as you can see with the last failed one, it took 2:48h to complete. that might create a problem for me ...

Tue, 02/09/2016 - 15:19
soydemadrid

Hi I also have this problem. I'm using GPL Webmin / Virtualmin 1.782

I get an email after 2 days and 10 hours with this error:

gzip: stdout: Broken pipe /bin/tar: -: Wrote only 4096 of 10240 bytes /bin/tar: Error is not recoverable: exiting now

Can anyone please help fix this.

Tue, 02/09/2016 - 15:31
soydemadrid

Just another quick update. I realised that I DO have automatic spin-down enabled on my backup server after 45 mins of non-use.

It's a long shot as I know other users have the same error so this probably isn't the cause, but I'm running another backup now with auto-spindown turned off just to see if this is the cause of the issue. If it is causing the problem is there some kind of "keep-alive" solution that webmin/virtualmin could have when backup up to keep the backup drive mounted and powered up?

Tue, 03/08/2016 - 13:46
soydemadrid

Hi further to this, turning off the spindowns on my backup box hasn't fixed this. I now get this error:

Uploading archive to SSH server 192.168.0.2 .. .. upload failed! root@192.168.0.2's password: /home/webmintemp/426161_12111_39_backup.pl: No such file or directory

.. completed in 2 days, 2 hours, 35:18 minutes

Can anyone please help with a fix?

Mon, 03/14/2016 - 15:06
soydemadrid

I now have the fix as per help from a webmin bug request. If you go to Webmin Configuration, Advanced Options and then set 'Maximum Age of Temporary Files' to more days such as 30 days then the backups now complete as they're not getting cleaned up before they've finished if very big!

Thanks and I hope this helps.

Topic locked