Storing to S3 fails

13 posts / 0 new
Last post
#1 Wed, 01/16/2013 - 10:17
bleeker

Storing to S3 fails

I noticed recently that Virtualmin added multi-part upload and mutli-level backup directories. I have 3.94 so I thought I'd try since s3cmd sync is not quite working for me. But Virtualmin fails. It sent 6 of the 14 files generated during the backup each of the last 2 nights. I changed this 2 nights ago so I was expecting results for the 15th and 16th. It also failed to create a backup log file. I see the resulting files, as I mentioned, so it must have run, but under backup logs there is nothing.

I'm backing up to local and then adding a destination to S3. Since some things made it to the S3 bucket then my credentials must be correct. Here is my bucket path: informatics-backup/INF-WEB/virtualmin/%Y-%m-%d See attachment for settings, log results screen, and what ended up in S3.

There should be one file that is 34GB and the rest are no bigger than 8KB for each night. I can see all the files fine on the local store.

Wed, 01/16/2013 - 12:22
andreychek

Howdy,

Well, there's been a lot of backup and S3 fixes over the last few Virtualmin versions.

I don't know for sure if it'll fix the specific issue you're seeing, but it might!

My suggestion would be to upgrade to the most recent Virtualmin version, and see if that resolves your issue.

If not, let us know and we can look into that further.

However, hopefully, one of the many fixes Jamie put into recent Virtualmin versions will fix that!

-Eric

Wed, 01/16/2013 - 13:43 (Reply to #2)
bleeker

I just updated to 3.94 last Friday and the system info page on Virtualmin says there are no more updates. How would I get a newer version?

Wed, 01/16/2013 - 14:29
andreychek

Hmm, well, Virtualmin 3.94 is about 6 months old... which Linux distro is it that you're using?

And how did you perform your Virtualmin install, did you use the install.sh script for that?

-Eric

Wed, 01/16/2013 - 14:45
bleeker

I inherited the system. Looks like they did this, oh, more than a year ago: wget http://software.virtualmin.com/gpl/scripts/install.sh sudo sh install.sh

Ubuntu 10.04.4 LTS Part of sources.list: deb http://us.archive.ubuntu.com/ubuntu/ lucid-updates main restricted universe multiverse deb http://archive.canonical.com/ lucid partner deb http://software.virtualmin.com/gpl/ubuntu/ virtualmin-lucid main deb http://software.virtualmin.com/gpl/ubuntu/ virtualmin-universal main

Wed, 01/16/2013 - 15:21
andreychek

Hmm, that sources.list looks correct at a glance... but I can also confirm that the latest Virtualmin is in that repo.

So something is awry :-)

What happens if you log into the command line as root, and run "apt-get update && apt-get upgrade", are you then prompted to upgrade to the most recent Virtualmin?

-Eric

Wed, 01/16/2013 - 15:45
bleeker

Not sure I can just do that right now. Strict controls on our production server so we don't upgrade something that might break a function. Is there a way to look without doing?

Thu, 01/17/2013 - 10:52
bleeker

I launched a new copy of the production server on Amazon and tried the command:

sudo apt-get update && apt-get upgrade
...
Reading package lists... Done
E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied)
E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?

Did sudo su instead then ran: apt-get update && apt-get upgrade

The following packages will be upgraded:
  libfreetype6 libfreetype6-dev libnspr4-0d libnss3-1d
4 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,643kB of archives.
After this operation, 86.0kB of additional disk space will be used.
Do you want to continue [Y/n]?

So I ran those updates (no Virtualmin) and the Virtualmin system screen says there are no updates to be had and Virtualmin is up to date and at version "Virtualmin version 3.94.gpl GPL".

Thu, 01/17/2013 - 11:16
andreychek

Hmm, can you paste in the output you receive when running "apt-get update" (as root, or with su/sudo)?

Something is awry, but I'm not sure what yet... seeing the apt-get update output may show some clues as to what's going on.

-Eric

Thu, 01/17/2013 - 11:23
bleeker
Hit http://us-east-1.ec2.archive.ubuntu.com lucid Release.gpg
Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ lucid/main Translation-en_US
Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ lucid/universe Translation-en_US
Get:1 http://us-east-1.ec2.archive.ubuntu.com lucid-updates Release.gpg [198B]
Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ lucid-updates/main Translation-en_US
Ign http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ lucid-updates/universe Translation-en_US
Hit http://us-east-1.ec2.archive.ubuntu.com lucid Release
Hit http://us-east-1.ec2.archive.ubuntu.com lucid-updates Release
Get:2 http://security.ubuntu.com lucid-security Release.gpg [198B]
Hit http://ppa.launchpad.net lucid Release.gpg
Hit http://us-east-1.ec2.archive.ubuntu.com lucid/main Packages
Ign http://security.ubuntu.com/ubuntu/ lucid-security/main Translation-en_US
Ign http://ppa.launchpad.net/awstools-dev/awstools/ubuntu/ lucid/main Translation-en_US
Get:3 http://us-east-1.ec2.archive.ubuntu.com lucid/universe Packages [5,430kB]
Ign http://security.ubuntu.com/ubuntu/ lucid-security/universe Translation-en_US
Hit http://ppa.launchpad.net lucid Release
Get:4 http://security.ubuntu.com lucid-security Release [57.3kB]
Hit http://ppa.launchpad.net lucid/main Packages
Get:5 http://us-east-1.ec2.archive.ubuntu.com lucid/main Sources [659kB]
Hit http://us-east-1.ec2.archive.ubuntu.com lucid/universe Sources
Get:6 http://us-east-1.ec2.archive.ubuntu.com lucid-updates/main Packages [663kB]
Hit http://us-east-1.ec2.archive.ubuntu.com lucid-updates/universe Packages
Hit http://us-east-1.ec2.archive.ubuntu.com lucid-updates/main Sources
Get:7 http://us-east-1.ec2.archive.ubuntu.com lucid-updates/universe Sources [106kB]
Get:8 http://security.ubuntu.com lucid-security/main Packages [473kB]
Get:9 http://security.ubuntu.com lucid-security/universe Packages [167kB]
Get:10 http://security.ubuntu.com lucid-security/main Sources [133kB]
Get:11 http://security.ubuntu.com lucid-security/universe Sources [43.8kB]
Fetched 7,732kB in 7s (1,030kB/s)
Reading package lists... Done
Thu, 01/17/2013 - 12:46
andreychek

Hmm, I don't actually see the Virtualmin repository mentioned in any of that.

I know you mentioned above that you saw the Virtualmin repo in your sources.list, but it doesn't appear from the above that it's seeing those lines for the Virtualmin repository.

You may need to post your entire sources.list file, including the Virtualmin part, in order for us to see what's going on with that.

-Eric

Thu, 01/17/2013 - 14:04
bleeker

That may be it. The sources.list file was not correct. We have found that Amazon instances when restored get the sources.list file overwritten for some odd reason. I forgot to go back and overwright the sources.list file with the master copy that we use. The one I included before was that master copy. Anyway, after putting the proper sources.list file in place, Virtualmin indeed reports that there are 3 updates.

webmin-virtual-server Webmin module for 'Virtualmin Virtual Servers (GPL)' New version 3.97.gpl-2 Virtualmin
webmin-virtual-server-mobile Webmin theme 'Virtualmin Mobile Theme' New version 2.5 Virtualmin
webmin-virtualmin-awstats Webmin module for 'AWstats Reporting' New version 4.8 Virtualmin

Installing them brings my test server to 3.97. I assume that is the latest then?

The odd thing to me now is that this proper sources.list file should have been used in our process last Friday but the updates did not go that route. Will have to investigate.

What are the chances that updating Virtualmin packages on the production server could be harmful?

Fri, 01/18/2013 - 09:33
bleeker

I made the change to the production server overnight. Virtualmin is at 3.97 now and the save of the backups to S3 is working. It sends all 14 files one of them being 34GB. Thanks

Topic locked