Virtualmin SSH backups started failing.

  • AbitatSupport
  • 05/05/08
  • Offline
Posted: Mon, 2008-11-17 10:04

Hi,

We have a daily backup via SSH to a remote server, which used to work just fine. Suddenly, the backup directory's are being created but absolutely nothing is being put inside of them.

What log files should I be checking for errors? I have access to both the virtualmin server and the backup ssh server.

Thanks!

edit On a manual test, several of the virtual servers gave this:

sh: /tmp/.webmin/17112008/gallery.mysite.net.tar.gz: No such file or directory

gzip: stdout: Broken pipe tar: -: Cannot write: Broken pipe tar: Error is not recoverable: exiting now

or this:

sh: /tmp/.webmin/17112008/demo.mysite.net.tar.gz: No such file or directory

gzip: stdout: Broken pipe tar: -: Wrote only 4096 of 10240 bytes tar: Error is not recoverable: exiting now

In fact, all the tars fail after the first few servers complete successfully, and finally the error below:

Uploading archive to SSH server .. .. upload failed! myuser@x.x.x.x's password: /tmp/.webmin/17112008: No such file or directory

Backup failed! See the progress output above for the reason why.

I assume that whatever is causing the original failure, is causing the fails afterward, and eventually, there is no file to upload because of those errors.

But what is causing it originally I guess is what I am wondering.<br><br>Post edited by: AbitatSupport, at: 2008/11/17 09:36


Re:Virtualmin SSH backups started failing.

  • andreychek
  • 01/05/09
  • Offline
  • Mon, 2008-11-17 10:38

Howdy,

Is there enough space in /tmp for the backup? I suspect the error would be different is space were an issue, but I figure I'd ask :-)

Barring that, I might file a support request, I'm not sure what would cause the errors you're getting. -Eric


Re:Virtualmin SSH backups started failing.

  • AbitatSupport
  • 05/05/08
  • Offline
  • Mon, 2008-11-17 10:59

Well, df says we have 99G left and our sites are certainly not that large.

Hum, okay thanks for the input tho :)


Re:Virtualmin SSH backups started failing.

  • andreychek
  • 01/05/09
  • Offline
  • Mon, 2008-11-17 11:01

Agreed, it definitely doesn't look like a space issue :-)

Yeah, I'd file a request here, Jamie may know the case of that:

http://www.virtualmin.com/bug-tracker/


any updates?

  • elsdon
  • 05/11/09
  • Offline
  • Mon, 2009-08-24 05:38

Hey there,

I'm encountering the same issues as above. Has there been any updates to the situation? My /tmp has plenty of space, as does my destination. It's been dying periodically for like the past 3-4 weeks daily.. Never on the same backups. Sometimes when I run them manually, they'll work, but the scheduled/automatic backgrounds seem to fail.


Hrm, this particular post was

  • andreychek
  • 01/05/09
  • Offline
  • Mon, 2009-08-24 08:50

Hrm, this particular post was last year, so I don't really recall what came of it :-)

What specific error(s) are you seeing during the backup?

-Eric


humm.

  • elsdon
  • 05/11/09
  • Offline
  • Mon, 2009-08-24 19:18

I'm seeing the exact same error as above.

Basically, after the entire 'site' has been tarballed, it'll try to upload it to remote ftp and it'll pop that error out. So, it's almost as if the tar wasn't created properly and doesn't exist, or something. I haven't been able to localize the error yet.


What is the output of these

  • andreychek
  • 01/05/09
  • Offline
  • Mon, 2009-08-24 19:22

What is the output of these two commands:

  1. mount

  2. df -h

Thanks,

-Eric


haha, /tmp has plenty of

  • elsdon
  • 05/11/09
  • Offline
  • Mon, 2009-08-24 19:28

haha, /tmp has plenty of space, that's the first thing I checked.

.tar.gz can be >4GB now right? It's 2009, I wouldn't imagine that being a limitation still??

It's peculiar, like, not all of the 'site' tarball creations fail.. I'll see a bunch of successful backups created and transferred onto the remote site, just randomly one will fail.


Yes, the tar archives can be

  • andreychek
  • 01/05/09
  • Offline
  • Mon, 2009-08-24 19:32

Yes, the tar archives can be quite large these days (I don't know about the exact limitations, but someone else was working with a 22GB file just earlier today).

However, it'd still be helpful if you offered the output of those two commands :-)

-Eric


sure. It's on a VPS so

  • elsdon
  • 05/11/09
  • Offline
  • Mon, 2009-08-24 20:27

sure. It's on a VPS so there's no mount breakdown, just one large filesystem.

[root@la tmp]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/simfs             60G   12G   49G  19% /

[root@la tmp]# mount
/dev/simfs on / type reiserfs (rw,usrquota,grpquota)
/proc on /proc type proc (rw)
/sys on /sys type sysfs (rw)
none on /dev/pts type devpts (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

Total backup size talking like 7.2GB or so (for about 6-7 different 'sites')

This is the error:

Creating TAR file of home directory .. .. TAR failed! sh: /tmp/.webmin/24-08-2009/site.tar.gz: No such file or directory

gzip: stdout: Broken pipe /bin/tar: -: Cannot write: Broken pipe /bin/tar: Error is not recoverable: exiting now


Thanks for all the info -- I

  • andreychek
  • 01/05/09
  • Offline
  • Mon, 2009-08-24 22:51

Thanks for all the info -- I had sent a message to Jamie to see if he has some thoughts on what's going awry.

-Eric


Backup settings

  • JamieCameron
  • 10/23/08
  • Offline
  • Mon, 2009-08-24 23:05

Hi guys,

I'd be interested to know exactly what settings you have for these backups that are failing, such as the backup path, format and strftime setting.

Also, do you perhaps have two backups running at the same time to remote destinations?


Why, yes I do.

  • elsdon
  • 05/11/09
  • Offline
  • Tue, 2009-08-25 19:11

I do have two backups that run on the same day.. The site backup runs at midnight every night, and takes about 2 hours to run a full backup.

I have another 'virtualmin' backup that just backs up settings that runs nightly at 2am.

Remote Backup path: /backup/sites/%d-%m-%Y Do strftime-style time substitutions on file or directory name - Yes Transfer each virtual server after it is backed up - Yes

Create destination directory? Yes

Halt the backup immediately [X] Backup level [X]

The backups do have a chance to overrun each other due to the 2 hour interval I've put in. Each time backup runs, does it rm -rf the /tmp directory? I can see that coincedentally running if my site backup is taking longer than usual.


Concurrent backups

  • JamieCameron
  • 10/23/08
  • Offline
  • Tue, 2009-08-25 23:31

The two concurrent backups could cause this problem..

Do they both use the same remote backup path?


Hi!Are there any

  • brainbug
  • 01/31/12
  • Offline
  • Tue, 2012-01-31 05:20

Hi!

Are there any news/solutions for this issue?

Suddenly, I'm encountering the same problem. Some servers fail to backup with a message like:

Uploading archive to SSH server my.backupserver.tld .. .. upload failed! /tmp/.webmin/24107-backup/mysite.tld.tar.gz: No such file or directory

All of the 11 sites that fail to backup are "Alias-Servers". I'm not sure if that is just coincidence or not. Other Alias-Servers backup sucessfully. It looks like it's always the same servers that fail to backup. I tried to just delete and recreate the failing servers, but that didn't help.

mount: dev/simfs on / type simfs (rw,relatime,usrquota,grpquota) proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs (rw,relatime) tmpfs on /lib/init/rw type tmpfs (rw,nosuid,relatime,mode=755) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,relatime) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)

df -h /dev/simfs 300G 105G 196G 35% / tmpfs 2,0G 0 2,0G 0% /lib/init/rw tmpfs 2,0G 0 2,0G 0% /dev/shm

Virtualmin: 3.89 gpl

Backup settings: Features to backup: Backup all features Backup destinations: SSH server Additional destination options: Transfer each virtual server after it is backed up Backup format: One file per server Create destination directory?: Yes Action on error: Continue with other features and servers Backup level: Full (all files)

It makes no difference wheter the backup is scheduled or not.


the same here

  • compre
  • 07/17/08
  • Offline
  • Tue, 2012-02-14 02:55

we encounter the same problem.

Error message

Uploading archive to SSH server .. .. upload failed! myuser@x.x.x.x's password: /tmp/.webmin/17112008: No such file or directory /xxxxxxxxx.tar.gz: No such file or directory


Any new news on this?

  • glenvistalabs
  • 08/24/10
  • Offline
  • Wed, 2012-06-06 12:31

I am having the exact same problem. It happens consistently, with usually the same 2-3 domains (out of >10 total) failing to remotely back up via SSH in this exact manner. Oddly enough, it happens during every automatic backup, but whenever I tell Virtualmin to perform the backup (manually, apart from its schedule) every domain backs up without any errors.

The directories that don't exist are named like: /tmp/.webmin/[X]-[hostname] Where [hostname] is the name of the server running Virtualmin (Pro) and performing the backups, and [X] is some 4-6 digit number

The user on the remote machine is not privileged (not authorized to perform commands as root via sudo). I intend to keep it this way because Virtualmin stores its password on the machine being backed up, making it extremely insecure. However, the ownership of /tmp/.webmin is root:root, and the permissions are 755 (rwxr-xr-x.), so it shouldn't even be able to write to that directory in the first place (but it's the way that Webmin/Virtualmin automatically set it up, so I'm inclined to think this is the way they should be).

Can anyone explain what is going on here?


Same proble

  • dajohn
  • 03/01/13
  • Offline
  • Fri, 2013-03-01 02:19

Same problem here. Is there a solution available?


A lot of backup related

  • andreychek
  • 01/05/09
  • Offline
  • Fri, 2013-03-01 10:24

A lot of backup related issues were resolved recently... you may want to make sure you're using the most recent Virtualmin version. Are you using 3.98 there?

-Eric


Virtualmin 3.99

  • JamieCameron
  • 10/23/08
  • Offline
  • Fri, 2013-03-01 14:28

Virtualmin version 3.99 will be out in a day or two, and should include a fix for this.


Didn't fix.

  • Yourname
  • 09/11/12
  • Offline
  • Tue, 2013-04-23 04:35

I have the latest, and nothing got fixed. My error is this for 3 out of the 50 or so domains I have:

Write failed: Broken pipe lost connection

.. completed in 3 minutes, 27 seconds


Everyday.

  • Yourname
  • 09/11/12
  • Offline
  • Thu, 2013-04-25 22:59

This has now become an everyday occurrence. :(


Broken pipe

  • JamieCameron
  • 10/23/08
  • Offline
  • Fri, 2013-04-26 15:15

Do you have access to the SSH server logs from the remote system? It looks like the connection is being killed, and the logs might explain why.


Jamie

  • Yourname
  • 09/11/12
  • Offline
  • Mon, 2013-04-29 05:54

Thanks, but it's a whole lotta nothing:

Apr 21 00:54:13 localhost sshd[22882]: Accepted password for backup from 192.168.1.11 port 33354 ssh2
Apr 21 00:54:13 localhost sshd[22882]: pam_unix(sshd:session): session opened for user backup by (uid=0)
Apr 21 00:54:15 localhost sshd[22882]: pam_unix(sshd:session): session closed for user backup
Apr 21 00:54:17 localhost sshd[22902]: Accepted password for backup from 192.168.1.11 port 33355 ssh2
Apr 21 00:54:17 localhost sshd[22902]: pam_unix(sshd:session): session opened for user backup by (uid=0)
Apr 21 00:54:18 localhost sshd[22902]: pam_unix(sshd:session): session closed for user backup
Apr 21 00:54:20 localhost sshd[22922]: Accepted password for backup from 192.168.1.11 port 33356 ssh2
Apr 21 00:54:20 localhost sshd[22922]: pam_unix(sshd:session): session opened for user backup by (uid=0)
Apr 21 00:54:21 localhost sshd[22922]: pam_unix(sshd:session): session closed for user backup
Apr 21 00:54:25 localhost sshd[22942]: Accepted password for backup from 192.168.1.11 port 33357 ssh2
Apr 21 00:54:25 localhost sshd[22942]: pam_unix(sshd:session): session opened for user backup by (uid=0)


scponly

  • JamieCameron
  • 10/23/08
  • Offline
  • Mon, 2013-04-29 18:26

Does the remote user perhaps have a shell like scponly that prevents execution of commands like mkdir ?


Nope, just a regular bash

  • Yourname
  • 09/11/12
  • Offline
  • Tue, 2013-04-30 04:50

Nope, just a regular bash shell for everyone.


Remote access

  • JamieCameron
  • 10/23/08
  • Offline
  • Tue, 2013-04-30 22:51

Since this problem seems to be limited to a few users, it would be useful if I could get remote access to your system to see what is really going wrong. Email me directly at jcameron@virtualmin.com if that is possible.


Hello,I'm having Virtualmin

  • remibruggeman
  • 06/20/07
  • Offline
  • Thu, 2013-04-25 01:58

Hello,

I'm having Virtualmin 3.99 and I am also facing some backup issues. These backup plans are running on 2 servers:


First schedule


  • All virtual servers
  • All features and settings
  • to SSH server File on server: /home/backupname/backup/%Y-%m-%d-%H%M
  • User & Pass set
  • Delete old backups after 35 days
  • Additional Destination options: Do Strftime-style, Transfer each virtual server after it is backed up
  • Backup format: one file per server, create destination directory
  • Action on error: Continue
  • Backup level: FULL
  • E-mail backup report to: my e-mail
  • Scheduled Backup Time: Complex schedule: Every Monday at 03:00

Second backup schedule:


  • All virtual servers
  • All features and settings
  • to SSH server File on server: /home/backupname/backup/INCR/%Y-%m-%d-%H%M
  • User & Pass set
  • Delete old backups after 35 days
  • Additional Destination options: Do Strftime-style, Transfer each virtual server after it is backed up
  • Backup format: one file per server, create destination directory
  • Action on error: Continue
  • Backup level: Incremental
  • E-mail backup report to: my e-mail
  • Scheduled Backup Time: Complex schedule: At cron time 0,30 * * * *

The output of mount command:

[root@turing remi]# mount
/dev/md4 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/md0 on /boot type ext4 (rw)
/dev/md1 on /data type ext4 (rw)
/dev/md2 on /secure type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
Output of df-h:
Filesystem            Size  Used Avail Use% Mounted on
/dev/md4              3.6T  7.1G  3.4T   1% /
tmpfs                  16G  4.0K   16G   1% /dev/shm
/dev/md0              496M   51M  420M  11% /boot
/dev/md1              3.6T  1.3T  2.2T  38% /data
/dev/md2              496M   11M  460M   3% /secure
Error message I get by mail:
   Uploading archive to SSH server XX.XX.XX.XX ..
   .. upload failed! sshbackup@XX.XX.XX.XX's password: 
backup.be.tar.gz                                 0%    0     0.0KB/s   --:-- ETAbackup.be.tar.gz                               100%   22KB  21.8KB/s   00:00    
Write failed: Broken pipe
lost connection
or:
   Uploading archive to SSH server XX.XX.XX.XX ..
   .. upload failed! sshbackup@XX.XX.XX.XX's password: 
286162_31137_28_backup.pl                       0%    0     0.0KB/s   --:-- ETA286162_31137_28_backup.pl                     100% 6060     5.9KB/s   00:00    
Write failed: Broken pipe
lost connection
 
 
.. completed in 2 hours, 0:48 minutes

Let me please also state that both links have at least 100Mbit/s uplinks. (ssh server has 120Mbit/sec, production has 100Mbit/sec)

I hope somebody has any use to this so this might solve a problem.

I do not 'really' have a problem with this since backups are running every 30 min, but as you can see with the last failed one, it took 2:48h to complete. that might create a problem for me ...