Webmin Loses Connection, Quits Listening on 10000

13 posts / 0 new
Last post
#1 Sat, 08/24/2019 - 13:04
MitchTalmadge
MitchTalmadge's picture

Webmin Loses Connection, Quits Listening on 10000

Hi there,

Recently, without any significant changes that I'm aware of, I found that my webmin instance was not accessible. It simply said "connection refused" in chrome on the default port 10000. After checking the systemctl status, it appeared that the webmin service was still running (and in fact, all of my websites were still accessible), but using "netstat -tulpn" showed that nothing was listening on port 10000. Usermin on port 20000 was working just fine.

I ended up restarting the service, which worked, but less than 30 minutes later while I was doing normal things, I got a popup in webmin that said "lost connection." Upon refresh, the same thing was happening; nothing was listening on port 10000, usermin works fine, websites work fine, etc.

It sounds to me like miniserv is crashing, however the miniserv.error logs show absolutely nothing around the time of the crash. You might also consider memory usage, but my server has 16GB and I rarely use more than 60%. dmesg shows nothing about killing any processes.

Where should I be looking to diagnose this problem?

Thanks

Sat, 08/24/2019 - 19:59
andreychek

Howdy,

A common cause of that is resource issues... if your server is running out of RAM, it could start killing off processes such as Webmin in order to keep the server online.

What is the output of these commands:

free -m
uptime
dmesg | tail -30
Sat, 08/24/2019 - 23:24 (Reply to #2)
MitchTalmadge
MitchTalmadge's picture

Hey there, thanks for the response.

It just crashed again and it happened right after I went into the postfix config and clicked on the Relocated Mapping button. This error appeared on the page and then nothing was accessible anymore:

"Failed to query Postfix config command to get the current value of parameter relocated_maps:"

I have run the commands you mentioned, here are the outputs:

free -m

              total        used        free      shared  buff/cache   available
Mem:          16051        3224       12295         343         531       12223
Swap:             0           0           0
uptime

22:17:04 up  2:55,  2 users,  load average: 0.25, 0.34, 0.35
dmesg | tail -30

[ 9666.701185] audit: type=1400 audit(1566705780.822:320): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/dovecot.index" pid=23837 comm="imap" requested_mask="rw" denied_mask="rw" fsuid=1001 ouid=1001
[ 9666.762450] audit: type=1400 audit(1566705780.882:321): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=23837 comm="imap" requested_mask="ra" denied_mask="ra" fsuid=1001 ouid=1001
[ 9666.763760] audit: type=1400 audit(1566705780.882:322): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index" pid=23837 comm="imap" requested_mask="rw" denied_mask="rw" fsuid=1001 ouid=1001
[ 9666.765036] audit: type=1400 audit(1566705780.886:323): apparmor="ALLOWED" operation="mknod" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=23837 comm="imap" requested_mask="c" denied_mask="c" fsuid=1001 ouid=1001
[ 9666.767414] audit: type=1400 audit(1566705780.886:324): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=23837 comm="imap" requested_mask="rwc" denied_mask="rwc" fsuid=1001 ouid=1001
[ 9666.767482] audit: type=1400 audit(1566705780.886:325): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist" pid=23837 comm="imap" requested_mask="rw" denied_mask="rw" fsuid=1001 ouid=1001
[ 9666.768074] audit: type=1400 audit(1566705780.886:326): apparmor="ALLOWED" operation="file_lock" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=23837 comm="imap" requested_mask="wk" denied_mask="wk" fsuid=1001 ouid=1001
[ 9666.768179] audit: type=1400 audit(1566705780.886:327): apparmor="ALLOWED" operation="file_lock" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=23837 comm="imap" requested_mask="k" denied_mask="k" fsuid=1001 ouid=1001
[ 9666.768203] audit: type=1400 audit(1566705780.886:328): apparmor="ALLOWED" operation="unlink" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=23837 comm="imap" requested_mask="d" denied_mask="d" fsuid=1001 ouid=1001
[10085.172472] audit: type=1400 audit(1566706199.295:329): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/dovecot.index.log" pid=26086 comm="imap" requested_mask="ra" denied_mask="ra" fsuid=1001 ouid=1001
[10085.172831] audit: type=1400 audit(1566706199.295:330): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/dovecot.index" pid=26086 comm="imap" requested_mask="rw" denied_mask="rw" fsuid=1001 ouid=1001
[10085.418278] audit: type=1400 audit(1566706199.539:331): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=26086 comm="imap" requested_mask="ra" denied_mask="ra" fsuid=1001 ouid=1001
[10085.418613] audit: type=1400 audit(1566706199.539:332): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index" pid=26086 comm="imap" requested_mask="rw" denied_mask="rw" fsuid=1001 ouid=1001
[10085.419518] audit: type=1400 audit(1566706199.539:333): apparmor="ALLOWED" operation="mknod" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=26086 comm="imap" requested_mask="c" denied_mask="c" fsuid=1001 ouid=1001
[10085.421396] audit: type=1400 audit(1566706199.543:334): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=26086 comm="imap" requested_mask="rwc" denied_mask="rwc" fsuid=1001 ouid=1001
[10085.421479] audit: type=1400 audit(1566706199.543:335): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist" pid=26086 comm="imap" requested_mask="rw" denied_mask="rw" fsuid=1001 ouid=1001
[10085.422260] audit: type=1400 audit(1566706199.543:336): apparmor="ALLOWED" operation="file_lock" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=26086 comm="imap" requested_mask="wk" denied_mask="wk" fsuid=1001 ouid=1001
[10085.422322] audit: type=1400 audit(1566706199.543:337): apparmor="ALLOWED" operation="file_lock" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=26086 comm="imap" requested_mask="k" denied_mask="k" fsuid=1001 ouid=1001
[10085.422338] audit: type=1400 audit(1566706199.543:338): apparmor="ALLOWED" operation="unlink" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=26086 comm="imap" requested_mask="d" denied_mask="d" fsuid=1001 ouid=1001
[10386.739392] audit_printk_skb: 3 callbacks suppressed
[10386.739394] audit: type=1400 audit(1566706500.859:340): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/dovecot.index.log" pid=27887 comm="imap" requested_mask="ra" denied_mask="ra" fsuid=1001 ouid=1001
[10386.739801] audit: type=1400 audit(1566706500.863:341): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/dovecot.index" pid=27887 comm="imap" requested_mask="rw" denied_mask="rw" fsuid=1001 ouid=1001
[10386.802458] audit: type=1400 audit(1566706500.923:342): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=27887 comm="imap" requested_mask="ra" denied_mask="ra" fsuid=1001 ouid=1001
[10386.803610] audit: type=1400 audit(1566706500.923:343): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index" pid=27887 comm="imap" requested_mask="rw" denied_mask="rw" fsuid=1001 ouid=1001
[10386.806576] audit: type=1400 audit(1566706500.927:344): apparmor="ALLOWED" operation="mknod" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=27887 comm="imap" requested_mask="c" denied_mask="c" fsuid=1001 ouid=1001
[10386.808546] audit: type=1400 audit(1566706500.931:345): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=27887 comm="imap" requested_mask="rwc" denied_mask="rwc" fsuid=1001 ouid=1001
[10386.808619] audit: type=1400 audit(1566706500.931:346): apparmor="ALLOWED" operation="open" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist" pid=27887 comm="imap" requested_mask="rw" denied_mask="rw" fsuid=1001 ouid=1001
[10386.809092] audit: type=1400 audit(1566706500.931:347): apparmor="ALLOWED" operation="file_lock" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=27887 comm="imap" requested_mask="wk" denied_mask="wk" fsuid=1001 ouid=1001
[10386.809190] audit: type=1400 audit(1566706500.931:348): apparmor="ALLOWED" operation="file_lock" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot.index.log" pid=27887 comm="imap" requested_mask="k" denied_mask="k" fsuid=1001 ouid=1001
[10386.809218] audit: type=1400 audit(1566706500.931:349): apparmor="ALLOWED" operation="unlink" profile="/usr/lib/dovecot/imap" name="/home/aptitekk/homes/mitch.talmadge/Maildir/.Sent/dovecot-uidlist.lock" pid=27887 comm="imap" requested_mask="d" denied_mask="d" fsuid=1001 ouid=1001

Not sure what these audit things are, but there's a whole lot of them in dmesg, even as far back as -100. Thanks!

Tue, 09/03/2019 - 20:03
Falcons34

MItch, did you get it figured out?

I am having the exact same issue. It's really annoying. I don't want to keep having to restart the instance. And i really hope it's not resources, as i have double the ram as my last vps.

Tue, 09/03/2019 - 20:26
MitchTalmadge
MitchTalmadge's picture

No luck. It seems like it stays up until I start using it, then within 30 mins it will die randomly. It "feels" like a memory usage problem, but there are no traces of it being killed, I have 16GB of RAM, and this just randomly started happening without any changes to the server after it working for a year....so I have no idea

Tue, 09/03/2019 - 22:27
andreychek

@Falcons34, you may wan to try what I mentioned in Comment #1 above to see if that shows anything.

@MutchTamadge, is this a dedicated server, or a VPS? If a VPS, what kind of VPS is it? I don't see any obvious resource issues in the above, but you're right that it does sound like a resource issue of some kind. I'm curious if it could be related to a VPS with certain types of resource restrictions, such as OpenVZ.

-Eric

Wed, 09/04/2019 - 06:12
Falcons34

Hi Eric, thanks for this

Yes, I ran those and it was showing out of memory errors and killing off clam processes. So I'm assuming clam is the problem. I knew it was a memory hog but was hoping 4G of RAM would be enough. Anyhow, is there any recommendation for a program to replace clam?

Thu, 09/05/2019 - 14:00
redrum2

Maybe try boosting your php memory limit up some.

Fri, 10/11/2019 - 08:17
innovativeflare
innovativeflare's picture

I'm having the same issue after doing to latest updates.

Sat, 10/12/2019 - 16:06
innovativeflare
innovativeflare's picture

I solved my issues this: https://www.virtualmin.com/comment/818100#comment-818100

2 things I disabled afterward that really saved a lot of ram were dovecot and postfix.

Sat, 10/12/2019 - 17:36
adamjedgar

I run dovecot and postfix on a 4GB RAM system and don't have any problems.

Normally I don't use Clamav in the foreground...however,

If I start clamav

-my RAM usage increases by about 20% (normally im running at about 22%, now 42%)...stats tell me /usr/sbin/clamd --foreground=true is using 1.01GB.

  • freshclam -d is using a further 151MB

-CPU does jump for a while then drops back to normal.

AJECreative is the home of $5 webhosting, $15/month VPS servers (1cpu,1gb RAM, 25GB storage)
Centos7, Debian9, or Ubuntu18LTS
Available Control Panels = Centos-Webpanel, Cyberpanel, or Virtualmin

https://ajecreative.com.au

Sat, 10/12/2019 - 21:32
innovativeflare
innovativeflare's picture

@adamjedgar Nice site, what theme are you using?

Sun, 10/13/2019 - 10:00 (Reply to #12)
Jfro

Collin

WP 5.2.3 theme maxhost and revslider version... apache 2.4.25 ... owl carousel, tweenmax , bootstrap 3.3.6 and so more you can read out of headers and source if you are meaning these things. also some mixed content , on mail ports from "postfix" i guess The expiration date of this certificate is 2019-08-04 09:28:17 UTC. 

And Collin you are using WP 5.2.3 maybe raidiant theme parts, apache 2.4.29 Ubuntu , woocommerce 3.7.1 and so more Trigger The server does not support any cipher suites that provide forward secrecy. on thgose 2 dashboard.innovativeflare.com 54.70.248.63 scan.innovativeflare.com 34.209.57.169

on maindomain box : The server uses a 1024-bit Diffie-Hellman group., The server uses a 1024-bit RSA key. ,

and on all still also TLS 1.0