fail2ban does not start on boot anymore

18 posts / 0 new
Last post
#1 Fri, 09/15/2017 - 17:21
Nico94

fail2ban does not start on boot anymore

Hi everyone,

(Debian 9 installation)

I think I might have spotted a bug since one of the latest updates.

fail2ban seems to be properly installed and configured but does not start anymore when the server (re)boots.

Manual starting works fine.

Webmin's "Bootup and Shutdown" does not help: fail2ban is marked as "Start at boot" ... but still does'nt.

Webmin's "Fail2Ban Intrusion Detector" does not help either: changing the "Start at boot" leaves the option on "no".

All the best, Nico

Fri, 09/15/2017 - 18:17
Joe
Joe's picture

Has it ever worked? There was a period of time where installations wouldn't be able to start/restart fail2ban on some distros (inclulding Debian 9). That's been fixed in a virtualmin-config update 6.0.18.

As long as you haven't altered your fail2ban config (specifically jail.local), you can make sure you have the latest virtualmin-config package and then run:

# virtualmin config-system --include Fail2banFirewalld
# systemctl restart fail2ban

This will fix a bug in the upstream packages which have a dependency on an iptables systemd unit, which doesn't even exist on Debian/Ubuntu.

--

Check out the forum guidelines!

Sat, 09/16/2017 - 02:50
Nico94

Hello Joe, thank you for your answer.

Yes it did work, not so long ago. Here comes a little bit of context :

  • I installed a server a couple of weeks ago. I noticed the recent "ProFTP LoadModule mod_tls.c" bug but fail2ban worked ran like a charm and started without problem after reboot.

  • I installed another server yesterday. I noticed that the ProFTP bug has meanwhile been solved (cool), but the fail2ban bug poped out.

Would it be possible that the fail2ban bug might have been reintroduced with the latest update?

Last but not least :

  • I ran the 2 commands above, unfortunately it did not help.

  • The installations I am talking about are made on freshly installed Debian 9 servers.

Thanks again, Nico

Sat, 09/16/2017 - 03:00
Joe
Joe's picture

Can you run:

# fail2ban-client -x start

And, tell me what happens. If it starts without error, then there's still a systemd unit file bug. If it fails to start, it'll give us a hint about why.

--

Check out the forum guidelines!

Sat, 09/16/2017 - 03:08
Nico94

Sure.

While fail2ban is runing :

# fail2ban-client -x start
ERROR  Server already running

While fail2ban is stoped :

# fail2ban-client -x start
2017-09-16 08:04:33,661 fail2ban.server         [4102]: INFO    Starting Fail2ban v0.9.6
2017-09-16 08:04:33,661 fail2ban.server         [4102]: INFO    Starting in daemon mode

By the way this is a test server, no problem if you need an acces to it.

--n

Sat, 09/16/2017 - 03:12
Joe
Joe's picture

You may need to restart the firewalld systemd unit, before the fail2ban unit, as well...I don't remember what depends on what.

--

Check out the forum guidelines!

Sat, 09/16/2017 - 03:15
Joe
Joe's picture

So that response tells us it is systemd that is failing and not fail2ban. So, fail2ban config is good. Problem is with the systemd unit files. I know I fixed that in the most recent virtualmin-config.

What's in the file /etc/systemd/system/fail2ban.service?

--

Check out the forum guidelines!

Sat, 09/16/2017 - 03:17
Nico94
root@ns367930.ip-94-23-26.eu:~# cat /etc/systemd/system/fail2ban.service
[Unit]
Description=Fail2Ban Service
Documentation=man:fail2ban(1)
After=network.target firewalld.service
PartOf=iptables.service firewalld.service

[Service]
Type=forking
ExecStart=/usr/bin/fail2ban-client -x start
ExecStop=/usr/bin/fail2ban-client stop
ExecReload=/usr/bin/fail2ban-client reload
PIDFile=/var/run/fail2ban/fail2ban.pid
Restart=always

[Install]
WantedBy=multi-user.target

And moreover :

root@ns367930.ip-94-23-26.eu:~# systemctl status firewalld
* firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2017-09-16 07:49:40 UTC; 24min ago
     Docs: man:firewalld(1)
Main PID: 539 (firewalld)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/firewalld.service
           `-539 /usr/bin/python3 -Es /usr/sbin/firewalld --nofork --nopid

sept. 16 07:49:41 ns367930.ip-94-23-26.eu firewalld[539]: WARNING: Port 'submission/tcp' already set, ignoring.
sept. 16 07:49:41 ns367930.ip-94-23-26.eu firewalld[539]: WARNING: Port 'domain/tcp' already set, ignoring.
sept. 16 07:49:41 ns367930.ip-94-23-26.eu firewalld[539]: WARNING: Port 'ftp-data/tcp' already set, ignoring.
sept. 16 07:49:41 ns367930.ip-94-23-26.eu firewalld[539]: WARNING: Port 'domain/udp' already set, ignoring.
sept. 16 07:49:41 ns367930.ip-94-23-26.eu firewalld[539]: WARNING: INVALID_HELPER: 'nf_conntrack_ftp' not available in kernel
sept. 16 07:52:07 ns367930.ip-94-23-26.eu firewalld[539]: WARNING: '/sbin/iptables-restore -n' failed:
sept. 16 07:52:07 ns367930.ip-94-23-26.eu firewalld[539]: ERROR: COMMAND_FAILED
sept. 16 08:04:19 ns367930.ip-94-23-26.eu firewalld[539]: ERROR: NOT_ENABLED: rule '('-p', 'tcp', '-m', 'multiport', '--dports', 'smtp,465,submission,imap3,imaps,pop3,pop3s', '-m', 'set', '--ma
sept. 16 08:04:36 ns367930.ip-94-23-26.eu firewalld[539]: WARNING: '/sbin/iptables-restore -n' failed:
sept. 16 08:04:36 ns367930.ip-94-23-26.eu firewalld[539]: ERROR: COMMAND_FAILED



root@ns367930.ip-94-23-26.eu:~# systemctl restart firewalld




root@ns367930.ip-94-23-26.eu:~# systemctl status firewalld
* firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2017-09-16 08:15:01 UTC; 3s ago
     Docs: man:firewalld(1)
Main PID: 5686 (firewalld)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/firewalld.service
           `-5686 /usr/bin/python3 -Es /usr/sbin/firewalld --nofork --nopid

sept. 16 08:15:01 ns367930.ip-94-23-26.eu systemd[1]: Starting firewalld - dynamic firewall daemon...
sept. 16 08:15:01 ns367930.ip-94-23-26.eu systemd[1]: Started firewalld - dynamic firewall daemon.
sept. 16 08:15:01 ns367930.ip-94-23-26.eu firewalld[5686]: WARNING: Port 'submission/tcp' already set, ignoring.
sept. 16 08:15:01 ns367930.ip-94-23-26.eu firewalld[5686]: WARNING: Port 'domain/tcp' already set, ignoring.
sept. 16 08:15:01 ns367930.ip-94-23-26.eu firewalld[5686]: WARNING: Port 'ftp-data/tcp' already set, ignoring.
sept. 16 08:15:01 ns367930.ip-94-23-26.eu firewalld[5686]: WARNING: Port 'domain/udp' already set, ignoring.
sept. 16 08:15:01 ns367930.ip-94-23-26.eu firewalld[5686]: WARNING: INVALID_HELPER: 'nf_conntrack_ftp' not available in kernel
Sat, 09/16/2017 - 03:19
Nico94

(trying to answer here but my comment has been queued for review)

Sat, 09/16/2017 - 03:44
Nico94

(trying a shorter answer)

# cat /etc/systemd/system/fail2ban.service
[Unit]
Description=Fail2Ban Service
Documentation=man:fail2ban(1)
After=network.target firewalld.service
PartOf=iptables.service firewalld.service

[Service]
Type=forking
ExecStart=/usr/bin/fail2ban-client -x start
ExecStop=/usr/bin/fail2ban-client stop
ExecReload=/usr/bin/fail2ban-client reload
PIDFile=/var/run/fail2ban/fail2ban.pid
Restart=always

[Install]
WantedBy=multi-user.target
Sat, 09/16/2017 - 14:49
Joe
Joe's picture

That looks like it did work. Fail2ban is running! (And the errors you see in the status are merely warnings about trying to add rules that already exist, which is harmless.)

So, what's the problem we're trying to tackle now?

--

Check out the forum guidelines!

Sun, 09/17/2017 - 03:06
Nico94

Hi Joe,

Unfortunatly the problem remains and and the symptoms are the same :

  • Yes fail2ban starts, but only when started manually (command line, Webmin) but it does not start automaticaly when the server (re)boots.

  • Webmin's "Bootup and Shutdown" module does not help: fail2ban is marked as "Start at boot" ... but still does'nt start at boot.

  • Webmin's "Fail2Ban Intrusion Detector" module does not help either: changing the "Start at boot" option to "yes" has no effect (it remains on "no")

Thanks again, Nico

Mon, 09/18/2017 - 19:36
Joe
Joe's picture

What does systemctl status fail2ban say?

--

Check out the forum guidelines!

Tue, 09/19/2017 - 04:33
Nico94

Just after having reboot the server, fail2ban is not started (but can be started manually). More specificaly, this happens:

root@ns367930.ip-94-23-26.eu:~# systemctl status fail2ban
* fail2ban.service - Fail2Ban Service
   Loaded: loaded (/etc/systemd/system/fail2ban.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:fail2ban(1)


root@ns367930.ip-94-23-26.eu:~# systemctl start fail2ban



root@ns367930.ip-94-23-26.eu:~# systemctl status fail2ban
* fail2ban.service - Fail2Ban Service
   Loaded: loaded (/etc/systemd/system/fail2ban.service; disabled; vendor preset: enabled)
   Active: active (running) since Tue 2017-09-19 09:31:46 UTC; 3s ago
     Docs: man:fail2ban(1)
  Process: 2348 ExecStart=/usr/bin/fail2ban-client -x start (code=exited, status=0/SUCCESS)
Main PID: 2369 (fail2ban-server)
    Tasks: 16 (limit: 4915)
   CGroup: /system.slice/fail2ban.service
           |-2369 /usr/bin/python3 /usr/bin/fail2ban-server -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x -b
           `-2428 /usr/bin/python3 -Es /usr/bin/firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp -m multiport --dports ssh,sftp -m set --match-set fail2ban-ssh-ddos src -j RE

sept. 19 09:31:45 ns367930.ip-94-23-26.eu systemd[1]: Starting Fail2Ban Service...
sept. 19 09:31:45 ns367930.ip-94-23-26.eu fail2ban-client[2348]: 2017-09-19 09:31:45,694 fail2ban.server         [2360]: INFO    Starting Fail2ban v0.9.6
sept. 19 09:31:45 ns367930.ip-94-23-26.eu fail2ban-client[2348]: 2017-09-19 09:31:45,694 fail2ban.server         [2360]: INFO    Starting in daemon mode
sept. 19 09:31:46 ns367930.ip-94-23-26.eu systemd[1]: Started Fail2Ban Service.
Wed, 09/20/2017 - 16:06 (Reply to #15)
Joe
Joe's picture

Problem is here:

Loaded: loaded (/etc/systemd/system/fail2ban.service; disabled; vendor preset: enabled)

It's disabled. So, enable it:

# systemctl enable fail2ban

Not sure why it would have changed, as we enable it during installation, but it looks like everything is working as it should be now, so enabling it should sort out your trouble.

--

Check out the forum guidelines!

Wed, 09/20/2017 - 05:03
Nico94

(I tried to answer but my comment has been queued for review again, would it be possible to remove the restriction concerning the lenght of a post?)

For the record, I freshly re-installed the server under Debian 8 (I can go back to 9 anytime if needed) and this time fail2ban does not start at all, neither on boot, nor manually.

Wed, 09/20/2017 - 16:08 (Reply to #17)
Joe
Joe's picture

There is no length restriction. It is a spam filter called Mollom. We have very little control over the parameters of how it works. It has a lot of false positives (seemingly more lately). We'll be moving to a better forum in the future, but I just don't have the hours in the day to tackle it right now.

"For the record, I freshly re-installed the server under Debian 8 (I can go back to 9 anytime if needed) and this time fail2ban does not start at all, neither on boot, nor manually."

Why? Debian 9 is better in every way to Debian 8. It's why they made a new one.

Edit: Also, I spend way more time/effort on testing and supporting the latest version of the distros we support, because they have the longest future. My efforts spent on new distros are more valuable than effort spent on old ones. Help me maximize my time by using the latest supported version of your preferred distro.

--

Check out the forum guidelines!

Wed, 10/04/2017 - 06:04
Nico94

Hi Joe,

Sorry for the delay of this answer and thank you so much for your time.

You where right: fail2ban was disabled and enabeling it easily solved the problem.

Still, it is a little odd that fail2ban was disabled on fresh installs (I even noticed it was masked on one installation) BUT that problem has not occured anymore for about 10 days. I don't know if you have changed anything in the install script, but fail2ban has been correctly enabled on every of my latest installations. Cool, problem solved :)

Thanks again, Nico

PS: I totaly agree on the fact that the latest version of a distribution should be used for fresh installs ... but unfortunately I am not the only one to decide and I do not always have the last word :-)