How to recover broken yum and rpm in Centos 6.6 from the nss update bug

6 posts / 0 new
Last post
#1 Sun, 01/18/2015 - 03:17
fakemoth
fakemoth's picture

How to recover broken yum and rpm in Centos 6.6 from the nss update bug

Hello, first of all this thread is necessary because I can't find any answers on the Centos forums for now https://www.centos.org/forums/viewtopic.php?f=13&t=50588 and https://www.centos.org/forums/viewtopic.php?f=13&t=50644 ; maybe someone here has an ideea.

I am one of the poor souls with a server destroyed by the bug and there are only some threads by various users, but no one answers how to solve it in fact at all, or only by ambigous and useless "you shouldn't..." "you should"... Let me say it: the CentOS and RHEL teams SHOULDN'T let something like this happen, ever. I hope they realise how many machines got screwed and don't have snapshots or something like this, not with CentOS 6.x.

I didn't do any particular thing, just a yum update; there were 3 updates, 2 nss related and the wbm-server by the Webmin team. It updated just one of the nss stuff and the server went berserk. Founding merely no info about this at that time, I started to search the web about how to solve it. And I might got into a worse situation. I take pride in my backups, I have all the critical data in all kind of safe places - but I usually don't backup the whole filesystem as the backups/restores never get it quite right so it is better to go with a fresh OS; and I didn't knew that something like this might happen of course, else I would have added the /var/lib directory... For me it was about the confs, the databases, the user data and so on. So I don't have a backup of that directory.

All the posts were going with rpm -vv --rebuilddb. Which apparently did nothing but really destroying the rpm database:

[root@ns1 ~]# yum update
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Update Process
Loading mirror speeds from cached hostfile
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
 Eg. $releasever is not a valid release or hasnt been released yet/
removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/$releasever/base/mirrorlist.txt
Error: Cannot find a valid baseurl for repo: base
[root@ns1 ~]# less /var/cache/yum/x86_64/$releasever/base/mirrorlist.txt
/var/cache/yum/x86_64//base/mirrorlist.txt: No such file or directory

I than tried this:

[root@ns1 rpm]# cd /var/lib/rpm/
[root@ns1 rpm]# rm -f /var/lib/rpm/__db*
[root@ns1 rpm]# db_dump Packages-BAKUP | db_load Packages
[root@ns1 rpm]# rpm --rebuilddb
[root@ns1 rpm]# yum update
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Update Process
Loading mirror speeds from cached hostfile
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
 Eg. $releasever is not a valid release or hasnt been released yet/
removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/$releasever/base/mirrorlist.txt
Error: Cannot find a valid baseurl for repo: base
[root@ns1 rpm]# 

This is as serious as it can get, take a look:

[root@ns1 rpm]# rpm -q httpd
package httpd is not installed
[root@ns1 rpm]# yum install httpd
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Install Process
Loading mirror speeds from cached hostfile
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
 Eg. $releasever is not a valid release or hasnt been released yet/
removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/$releasever/base/mirrorlist.txt
Error: Cannot find a valid baseurl for repo: base

Just some info:

[root@ns1 rpm]# uname -a
Linux ns1.mumu.ro 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@ns1 rpm]# cat /etc/redhat-release
CentOS release 6.6 (Final) 

What else can I do? And are there any other dangers besides the rpm and yum not working? Think not but I really don't know, just asking? I am hoping to get over this week and install a CentOS 7 the next week-end...

Sun, 01/18/2015 - 12:03
andreychek

Howdy,

There's a very unfortunate package issue! There's a few folks who ran into that issue before, and most of them appear to have had success fixing it using the info in the following post -- let us know if the solution here helps in your situation:

https://www.virtualmin.com/node/35857

Mon, 01/19/2015 - 00:42
fakemoth
fakemoth's picture

Hi and thanks, but it doesn't. This is not helping, as I mentioned, the rpm database was completely ruined by [i]rpm --rebuilddb[/i] so I need to rebuild it, is it possible?

Here it is the output, just in case, of that post:

[root@ns1 ~]# yumdownloader nss-softokn-freebl
Loaded plugins: fastestmirror, refresh-packagekit
Loading mirror speeds from cached hostfile
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
 Eg. $releasever is not a valid release or hasnt been released yet/
removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/$releasever/base/mirrorlist.txt
Cannot find a valid baseurl for repo: base
[root@ns1 ~]# wget ftp://195.220.108.108/linux/centos/6.6/updates/x86_64/Packages/nss-softokn-freebl-3.14.3-19.el6_6.x86_64.rpm
--2015-01-18 16:11:10--  ftp://195.220.108.108/linux/centos/6.6/updates/x86_64/Packages/nss-softokn-freebl-3.14.3-19.el6_6.x86_64.rpm
           => “nss-softokn-freebl-3.14.3-19.el6_6.x86_64.rpm”
Connecting to 195.220.108.108:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /linux/centos/6.6/updates/x86_64/Packages ... done.
==> SIZE nss-softokn-freebl-3.14.3-19.el6_6.x86_64.rpm ... 169940
==> PASV ... done.    ==> RETR nss-softokn-freebl-3.14.3-19.el6_6.x86_64.rpm ... done.
Length: 169940 (166K) (unauthoritative)
 
100%[==============================================================================================>] 169,940      621K/s   in 0.3s    
 
2015-01-18 16:11:11 (621 KB/s) - “nss-softokn-freebl-3.14.3-19.el6_6.x86_64.rpm” saved [169940]
 
[root@ns1 ~]# rpm2cpio nss-softokn-freebl-3.14.3-19.el6_6.x86_64.rpm | cpio -idmv
./etc/prelink.conf.d
./etc/prelink.conf.d/nss-softokn-prelink.conf
./lib64/libfreebl3.so
./lib64/libfreeblpriv3.chk
./lib64/libfreeblpriv3.so
./usr/lib64/libfreebl3.so
./usr/lib64/libfreeblpriv3.chk
./usr/lib64/libfreeblpriv3.so
./usr/share/dracut/modules.d/05nss-softokn
./usr/share/dracut/modules.d/05nss-softokn/install
960 blocks
[root@ns1 ~]# cp ./lib64/libfreeblpriv3.* /lib64
[root@ns1 ~]# yum update
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Update Process
Loading mirror speeds from cached hostfile
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
 Eg. $releasever is not a valid release or hasnt been released yet/
removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/$releasever/base/mirrorlist.txt
Error: Cannot find a valid baseurl for repo: base
[root@ns1 ~]# 

Don't take the name of root in vain...

Fri, 02/13/2015 - 09:00
MatthiasMaass

Hi

I have the same problem on different servers, but I also can't fix it. Have you found a way to fix this problem?

Regards Matthias

Fri, 02/13/2015 - 15:53
cyrus

As stated by RapidX in this post (https://www.virtualmin.com/node/35857#comment-143439) do not rebbuilddb. If you do you will have to build it manually.

I took the opportunity to do a fresh install of CentOS 7 and restored my backups. Worked out for the best :)

Sun, 03/15/2015 - 05:46
fakemoth
fakemoth's picture

@MatthiasMaass Yes I did the manual one - well to get a grip of how long it takes... I am still doing it nowadays, based on CentOS announce list for upgrades lol

Basically you have to issue this command for EVERY god damn piece of software :)

yum install --setopt=tsflags=justdb named

Where "named" is the name of the package.

@cyrus Too late for a lot of admins out there... I did the nss mistake after a couple of WEEKS, after the (humongous) bug has been reported and no one at CentOS bothered to pull down the stupid update... so quite a few people go it...

Also - the dudes with weird/older RAID cards can't update easily, see this https://www.centos.org/forums/viewtopic.php?f=49&t=51302 ; the guys at Elrepo helped us to get our HP RAID arrays working with CentOS 7.

Soon I will have the exact same issue with IBM I guess... will see.

Don't take the name of root in vain...

Topic locked