New software repository mirror now online and old repository server going offline

6 posts / 0 new
Last post
#1 Tue, 07/25/2017 - 03:07
Joe
Joe's picture

New software repository mirror now online and old repository server going offline

Howdy all,

I've just turned on one more new mirror server for software.virtualmin.com, and disabled the old one (that had a hard disk failure a couple weeks back and still has hardware problems that make it untrustworthy). So, there are now two servers for software.virtualmin.com (one in Paris, one in San Jose, both have very fast network and SSD drives).

This should resolve the issue some folks had with yum metadata mismatches on CentOS distros (it happened because repo metadata was generated on each mirror, rather than synced between the two, and changing that required changes to the way I push packages into our repos).

It also means our CentOS 5, and some of the other old distro repositories that stuck around out of inertia, on the old mirror are no longer accessible. Because those repos don't exist on my staging server which is the "source of truth" for the repositories, they don't exist on the new mirrors which have stricter rsync commands to keep things a little cleaner and small, so I can spin mirrors up and down very quickly.

Those old distros were already unmaintained (because the distros in question have been end-of-lifed by the upstream vendor for months or years), so this doesn't actually change anything. But, if you're still using one of those EOL distros (stop doing that; upgrade ASAP), you'll need to disable the Virtualmin repository for that distro. You can keep the virtualmin-universal repository enabled, as it isn't going anywhere and is still being updated with new Virtualmin packages.

Anyway, let me know if you run into any repository related issues. I have tested across most of our distributions and versions, but it's late and I'm too tired to be extremely thorough about it. But, if you haven't seen problems over the past couple of weeks, you probably won't see problems now, as a mirror of this type has been in service for that time. And, the CentOS mirror issue should be resolved, so things should be more reliable.

Thanks for reading!

Cheers,

Joe

Tue, 07/25/2017 - 04:51
Joe
Joe's picture

BTW, if you have a system giving errors when updating or installing new software, run the following from the command line as root:

# yum clean all

And then try it again.

--

Check out the forum guidelines!

Wed, 07/26/2017 - 02:19
toreskev

Since upgrading from Debian 8 to 9 I've had one issue with the repo:

Err:12 http://software.virtualmin.com/gpl/debian virtualmin-stretch/main amd64 Packages 404 Not Found Ign:13 http://software.virtualmin.com/gpl/debian virtualmin-stretch/main Translation-en Ign:14 http://software.virtualmin.com/gpl/debian virtualmin-stretch/main Translation-en_US Reading package lists... Done W: The repository 'http://software.virtualmin.com/gpl/debian virtualmin-stretch Release' does not have a Release file. N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use. N: See apt-secure(8) manpage for repository creation and user configuration details. E: Failed to fetch http://software.virtualmin.com/gpl/debian/dists/virtualmin-stretch/main/... 404 Not Found E: Some index files failed to download. They have been ignored, or old ones used instead.

Is that somehow related to the new repo?

Wed, 07/26/2017 - 22:06 (Reply to #3)
Joe
Joe's picture

No, that's related to the Debian 9. ;-)

Debian 9 doesn't yet have repo support in the old Virtualmin repos. It does have support in the Virtualmin 6 repos, and it's pretty safe to switch to using them. I'll provide a script for automating this a day or two after VM6 goes stable, but in the short term you can do it manually:

Edit /etc/apt/sources.list

Change the repo path for Virtualmin repos to this:

deb http://software.virtualmin.com/vm/6/gpl/apt virtualmin-stretch main
deb http://software.virtualmin.com/vm/6/gpl/apt virtualmin-universal main

Note the addition of "vm/6/apt/" to the path and removal of "debian".

And, you'll need to import the new GPG key by doing:

# wget http://software.virtualmin.com/lib/RPM-GPG-KEY-virtualmin-6
# apt-key add RPM-GPG-KEY-virtualmin-6

Now things should work as expected. I'm not sure if this requires cleaning out the old repo meta data, as this isn't something I've tested at all. I'm hoping to have VM6 out the door officially really soon, and will be able to work on automating upgrading for old installations.

--

Check out the forum guidelines!

Thu, 07/27/2017 - 05:35 (Reply to #4)
toreskev

Ah, that explains it! ;-)

Thanks, I'll try that. So no spesific /debian/ on this, only virtualmin-stretch?

Crossing fingers nothing breaks :-P

Thu, 07/27/2017 - 23:52 (Reply to #5)
Joe
Joe's picture

Correct.

I have switched to a pool-based apt repository, so all of our packages across all versions and distros are in the same pool, and the repo metadata is all that lives in the OS-specific directories. This is just a side-effect of me switching to aptly to manage our Debian/Ubuntu repositories, not really a better or worse thing (but managing repos with aptly is definitely better than the old hand-rolled repo management scripts I've been using for the past decade or so).

The only thing that might break is the thing that's already broke (repository access), so no harm in something going awry. The packages in the repo that are beta have new package names, so you won't accidentally get them just by switching to the new repository. And, I'm feeling pretty confident the beta is nearly over. The only error reports I'm getting are unique and I can't reproduce them. So, we're close to calling all of it "stable".

--

Check out the forum guidelines!