webmin & SSL

Hi,

I have just done an upgrade of my CentOS and the SSL part of webmin has stopped working. SSL connections to usermin and httpd work without any problems.

Below, you can see an extract of my yum.log. When I run the update for the first time packages usermin, wbt-virtual-server-theme and virtualmin failed becuase there was not enough free RAM. At that moment, I have worked with webmin for a little moment before I rerun the upgrade of the packages that failed (everything seemed to be working at that moment). When the install of webmin-1.600-1 succeeded, I couldn't connect to it anymore. After playing with it for a while I have figured out, that when I turn off ssl, webmin works. I have tried downgrading and reinstalling webmin and doing the same with a couple of other packages but it still doesn't work. If you have any idea what got broken and how to fix it I would appreciate your input very much.

Thank you!

Greetings, tomiskou

yum.log:

Oct 25 12:46:38 Updated: 30:bind-libs-9.3.6-20.P1.el5_8.5.x86_64
Oct 25 12:46:39 Updated: 30:bind-9.3.6-20.P1.el5_8.5.x86_64
Oct 25 12:46:40 Updated: 30:caching-nameserver-9.3.6-20.P1.el5_8.5.x86_64
Oct 25 12:46:44 Updated: 30:bind-utils-9.3.6-20.P1.el5_8.5.x86_64
Oct 25 12:47:00 Updated: cyrus-sasl-lib-2.1.22-7.el5_8.1.x86_64
Oct 25 12:47:00 Updated: cyrus-sasl-lib-2.1.22-7.el5_8.1.i386
Oct 25 12:47:01 Updated: cyrus-sasl-2.1.22-7.el5_8.1.x86_64
Oct 25 12:47:02 Updated: cyrus-sasl-md5-2.1.22-7.el5_8.1.x86_64
Oct 25 12:47:03 Updated: cyrus-sasl-plain-2.1.22-7.el5_8.1.x86_64
Oct 25 12:47:03 Updated: cyrus-sasl-gssapi-2.1.22-7.el5_8.1.x86_64
Oct 25 12:47:04 Updated: cyrus-sasl-2.1.22-7.el5_8.1.i386
Oct 25 12:47:04 Updated: cyrus-sasl-gssapi-2.1.22-7.el5_8.1.i386
Oct 25 12:47:10 Updated: 12:dhclient-3.0.5-31.el5_8.1.x86_64
Oct 25 12:47:27 Updated: ghostscript-8.70-14.el5_8.1.x86_64
Oct 25 12:47:28 Updated: ghostscript-devel-8.70-14.el5_8.1.x86_64
Oct 25 12:47:29 Updated: ghostscript-8.70-14.el5_8.1.i386
Oct 25 12:48:23 Updated: glibc-common-2.5-81.el5_8.7.x86_64
Oct 25 12:48:30 Updated: glibc-2.5-81.el5_8.7.x86_64
Oct 25 12:48:31 Updated: nscd-2.5-81.el5_8.7.x86_64
Oct 25 12:48:35 Updated: glibc-headers-2.5-81.el5_8.7.x86_64
Oct 25 12:48:37 Updated: glibc-2.5-81.el5_8.7.i686
Oct 25 12:48:39 Updated: glibc-devel-2.5-81.el5_8.7.x86_64wbt-virtual-server-theme
Oct 25 12:48:48 Updated: initscripts-8.45.42-1.el5.centos.1.x86_64
Oct 25 12:49:21 Installed: kernel-2.6.18-308.16.1.el5.x86_64
Oct 25 12:49:32 Erased: kmod-dahdi-linux-fwload-vpmadt032
Oct 25 12:49:39 Updated: kernel-headers-2.6.18-308.16.1.el5.x86_64
Oct 25 12:49:44 Updated: libxml2-2.6.26-2.1.15.el5_8.5.x86_64
Oct 25 12:49:45 Updated: libxml2-2.6.26-2.1.15.el5_8.5.i386
Oct 25 12:49:49 Updated: libxml2-python-2.6.26-2.1.15.el5_8.5.x86_64
Oct 25 12:49:53 Updated: libxslt-1.1.17-4.el5_8.3.x86_64
Oct 25 12:49:54 Updated: libxslt-1.1.17-4.el5_8.3.i386
Oct 25 12:50:14 Updated: mysql-libs-5.5.28-12.el5.art.x86_64
Oct 25 12:50:18 Updated: mysql-5.5.28-12.el5.art.x86_64
Oct 25 12:50:21 Updated: mysql-server-5.5.28-12.el5.art.x86_64
Oct 25 12:50:24 Updated: mysql-libs-5.5.28-12.el5.art.i386
Oct 25 12:50:24 Updated: mysql-devel-5.5.28-12.el5.art.i386
Oct 25 12:50:25 Updated: mysql-server-5.5.28-12.el5.art.i386
Oct 25 12:50:26 Updated: mysql-devel-5.5.28-12.el5.art.x86_64
Oct 25 12:50:26 Updated: mysql-5.5.28-12.el5.art.i386
Oct 25 12:50:44 Updated: 2:nmap-6.01-2.el5.art.x86_64
Oct 25 12:50:52 Updated: perl-DBD-Pg-1.49-4.el5_8.x86_64
Oct 25 12:51:14 Installed: perl-Digest-SHA-5.71-1.el5.rf.x86_64
Oct 25 12:51:15 Updated: perl-Net-DNS-0.66-1.el5.art.x86_64
Oct 25 12:51:35 Updated: php-common-5.3.18-11.el5.art.x86_64
Oct 25 12:51:35 Updated: php-pdo-5.3.18-11.el5.art.x86_64
Oct 25 12:51:36 Updated: php-cli-5.3.18-11.el5.art.x86_64
Oct 25 12:51:37 Updated: php-5.3.18-11.el5.art.x86_64
Oct 25 12:51:37 Updated: php-xml-5.3.18-11.el5.art.i386
Oct 25 12:51:37 Updated: php-5.3.18-11.el5.art.i386
Oct 25 12:51:38 Updated: php-snmp-5.3.18-11.el5.art.x86_64
Oct 25 12:51:38 Updated: php-xmlrpc-5.3.18-11.el5.art.x86_64
Oct 25 12:51:40 Updated: php-devel-5.3.18-11.el5.art.x86_64
Oct 25 12:51:40 Updated: php-xml-5.3.18-11.el5.art.x86_64
Oct 25 12:51:40 Updated: php-mbstring-5.3.18-11.el5.art.x86_64
Oct 25 12:51:40 Updated: php-odbc-5.3.18-11.el5.art.x86_64
Oct 25 12:51:40 Updated: php-imap-5.3.18-11.el5.art.x86_64
Oct 25 12:51:41 Updated: php-gd-5.3.18-11.el5.art.x86_64
Oct 25 12:51:41 Updated: php-pgsql-5.3.18-11.el5.art.x86_64
Oct 25 12:51:41 Updated: php-mysql-5.3.18-11.el5.art.x86_64
Oct 25 12:51:51 Updated: postgresql-8.1.23-6.el5_8.x86_64
Oct 25 12:51:56 Updated: postgresql-server-8.1.23-6.el5_8.x86_64
Oct 25 12:52:04 Updated: postgresql-libs-8.1.23-6.el5_8.x86_64
Oct 25 12:52:05 Updated: postgresql-libs-8.1.23-6.el5_8.i386
Oct 25 12:52:10 Updated: sudo-1.7.2p1-14.el5_8.4.x86_64
Oct 25 12:52:19 Updated: tzdata-2012f-1.el5.x86_64
Oct 25 12:52:50 usermin-1.520-1.noarch: 100
Oct 25 12:53:20 Updated: wbm-virtual-server-3.94.gpl-2.noarch
Oct 25 12:53:25 Updated: 2:wbm-virtualmin-htpasswd-2.6-1.noarch
Oct 25 12:53:32 Updated: 2:wbm-virtualmin-registrar-2.2-1.noarch
Oct 25 12:53:38 Updated: 2:wbt-virtual-server-mobile-2.5-1.noarch
Oct 25 12:53:46 2:wbt-virtual-server-theme-8.5-2.noarch: 100
Oct 25 13:35:10 Updated: usermin-1.520-1.noarch
Oct 25 13:35:21 Updated: 2:wbt-virtual-server-theme-8.5-2.noarch
Oct 25 13:39:37 Updated: webmin-1.600-1.noarch
Status: 
Active

Comments

It may have failed to start due to lack of RAM. Try SSHing in and running /etc/webmin/start as root.

How much RAM do you have?

Hi,

Thanks for a prompt reply.

I have 512MB/1024MB (burst) which has been enough for the moment (about 2 years of running on the same machine).

webmin starts even with ssl=1 option and nmap can see it running on port 10000. When I connect with Firefox I get 'connection was interrupted'. As I said already ssl=0 works without problems as well as usermin with ssl=1.

Thanks.

tomiskou

What do you see in the Webmin logs whenever you're having that problem?

The Webmin log is in /var/webmin/miniserv.log.

Also, it sounds like you're using OpenVZ, which unfortunately often introduces resource problems -- can you paste in your /proc/user_beancounters file?

I do not see anything in webmin logs nor in any other logs.

Yep, I am using OpenVZ. The output of /proc/user_beancounters is below, but I have compared the fail counts before and after trying to connect to webmin as well as before and after restarting webmin and no new fail counts have appeared. Could you tell me which variable could be related to my problem?

I do not see why this would be a resource problem since this feature worked for me for last two years (on the same machine). If it really is a resource problem, I should be able to resolve it by downgrading to the version of webmin which worked, no? This doesn't help however and so I assume that the problem is in the other packages that I have upgraded (I just do not see which and why). I do not want to downgrade all of them since downgrading cyrus-sasl, for example cost me a lot of time to reinstall and properly reconfigure all the programs that depended on it.

I have a related question, as mentioned above I had to reinstall a couple of other packages and I screwed up my postfix. I have managed to restore its basic functionality but the user defined mail filters do not work. Would you point to the configuration file in which this could be fixed? (The filters are completely ignored and all the mail gets delivered to my mailbox instead of being delivered to appropriate subfolders defined in these filters).

Thanks.

tomiskou

/proc/user_beancounters:

Version: 2.5
       uid  resource                     held              maxheld              barrier                limit              failcnt
    19043:  kmemsize                 21720013             80268896            134217728            134217728                    0
            lockedpages                     0                 1024                 1024                 1024                  156
            privvmpages                146797               282755               262144               262144               817402
            shmpages                     5077                27743               102400               102400                    0
            dummy                           0                    0                    0                    0                    0
            numproc                        94                  359                  640                  640                    0
            physpages                   63248               171942                    0  9223372036854775807                    0
            vmguarpages                     0                    0               131072  9223372036854775807                    0
            oomguarpages                63260               171943               104857  9223372036854775807                    0
            numtcpsock                     30                  266                 1024                 1024                    0
            numflock                        7                   81                 2048                 2048                    0
            numpty                          3                    3                   64                   64                    0
            numsiginfo                      0                  220                 1024                 1024                    0
            tcpsndbuf                  651376              5429880              5368709             10737418             13539424
            tcprcvbuf                  491520              5389296              5368709             10737418                 1181
            othersockbuf               292536              4583448              5368709             10737418                    0
            dgramrcvbuf                     0               198208              1342177              2684354                    0
            numothersock                  163                 1024                 1024                 1024                  788
            dcachesize                 927732              2290221              8053063             12582912                    0
            numfile                      3213                14600                32768                32768                    0
            dummy                           0                    0                    0                    0                    0
            dummy                           0                    0                    0                    0                    0
            dummy                           0                    0                    0                    0                    0
            numiptent                      50                   50                 1536                 1536                    0

That's a good step to review the user_beancounters both before and after an issue, that does rule out that the SSL issue is due to a current restriction.

However, even though those failures may not be occurring now, it sounds like there may have been resource problems during the package updates, which could lead to corrupt packages and config files.

As you see in the user_beancounters output, there's a bunch of resource failures that your server is seeing. Even one failure in the failcnt field can mean serious problems. There's quite a few there, unfortunately.

One thing I'd suggest, is that I'd encourage you to work with your provider to increase your resources -- as so long as you have any failures, you could end up with really strange issues :-/ Show them where you're running into failcnt's, and they should be able to help with that.

We're not trying to give you a hard time, it's just an issue we've been seeing a lot of -- folks who have failcnt's not at 0 have some of the most bizarre issues! It's becoming so common that we're actually considering adding a warning into Virtualmin to notify folks when the failcnt field is non-zero.

As far as how to resolve this -- since OpenVZ limits generate hard failures, it's difficult to say what exactly may have gone awry during the package updates... but one place to start may be to go into Server Configuration -> Manage SSL Certificates for a domain that has SSL setup, and to click the "Copy to Webmin" button. That will re-setup SSL in Webmin.

As far as Postfix goes -- you'd want to verify your /etc/procmailrc file. It's possible that became corrupt. Also, in /etc/postfix/main.cf, make sure that "mailbox_command" is set to the following:

/usr/bin/procmail-wrapper -o -a $DOMAIN -d $LOGNAME

Although we can try and get you going in the right direction with this, remember that the Support area here is for Virtualmin Pro customers. For future reference, if you're using Virtualmin GPL, you'd want to use the Forums for support. We monitor those, along with lots of wonderful folks in the community... thanks!

I have solved the issue with user filters and postfix. It was actually as easy as setting "Allow mailbox users to create mail filters?" to Yes in "Email Messages -> Spam and Virus Scanning" which must have gotten reset to No when I removed postfix by mistake when downgrading cyrus-sasl.

Meanwhile I have noticed that from time to time I do get "Failed to initialize SSL connection" error message in miniserv.log. When running webmin in SSL=1 mode. It never works but the error message appears only half of the time, which is why I didn't notice it at first. "Copy to Webmin" didn't work.

I know that you are not trying yo give me hard time and I appreaciate any help. I can see how resource limits may lead to most confusing errors and that it is impossible to resolve them all. Unfortunatelly I have already had a discussion about bean_counters and resources with my provider when I had another issue, it looks like increasing some of the limits is out of question. I will just change my provider when I find some time.

Meanwhile I will keep trying to figure out what got broken during the failed upgrade.

In any case thanks for your assistance.

Meanwhile I have noticed that from time to time I do get "Failed to initialize SSL connection" error message in miniserv.log. When running webmin in SSL=1 mode. It never works but the error message appears only half of the time, which is why I didn't notice it at first. "Copy to Webmin" didn't work.

Are you or any of your users having problems? Or are you just seeing that in the logs?

It's not uncommon to see that problem in the logs -- I have that message 146 times on my own personal server over the last month.

The issue is that bots doing probing can cause that error to appear, as they don't always connect using SSL mode. That's safe to ignore though, unless you or your users are seeing problems when connecting with your browser.

Unfortunatelly, it is not something that only appears in logs. I cannot login to webmin with my browser when run in SSL=1 mode at all and so I temporarily run it in SSL=0 mode (and I have tried couple of different browsers and machines, just to be sure that it is not a client side problem).

I was digging a little bit deeper and have noticed that when run in SSL=1 mode Net::SSLeay::accept($ssl_con) in miniserv.pl either returns False or simply dies (lines after the call are ignored). I didn't yet have time to figure out how to go deeper since if I understand it correctly Net::SSLeay is a perl openssl wrapper. Otherwise I would have already traced the code. I have tried reinstalling openssl and and Perl-Net-SSLeay with yum reinstall, which didn't help.

What is funny is that usermin (apparently also driven by miniserv.pl) runs in SSL=1 mode without any problems. I have tried to use the certificates I am using in usermin also in webmin, just to be sure that the problem doesn't come from the certificates, but this doesn't help.

I will try to debug usermin and webmin side by side to see what the difference in SSL related configuration is. Perl is not in the list of my native languages however, so it might take a little while before I figure out the best of doing that.