VM 6 install.sh LEMP issue

18 posts / 0 new
Last post
#1 Fri, 08/11/2017 - 22:26
noisemarine

VM 6 install.sh LEMP issue

Hi Joe,

I'm moving my issue to this thread as I have confirmed it still happens with the release installer, not just the beta vm6-install.sh installer.

Recap:

On Debian 9 LEMP install (invoked with 'bash install.sh -b LEMP'), apache2 is pulled into the install preventing nginx from binding to port 80 and the install fails.

My basic install path is as follows:

My VPS host does not have a Debian 9 image, so I install a Debian 8 Minimal image. I perform some post image install steps such as creating new users, setting up passwordless ssh logins and the like. I then update all Debian 8 packages, and then change /etc/apt/sources.list to use Stretch, and dist-upgrade to Debian 9.

Prior to installing Virtualmin, my /etc/apt/sources.list contains the following:

deb http://ftp.debian.org/debian stretch main contrib non-free
deb http://ftp.debian.org/debian stretch-updates main contrib non-free
deb http://security.debian.org stretch/updates main contrib non-free

A listing of installed packages prior to running the installer can be found at https://pastebin.com/RqChqCS0 .

Here is the full virtualmin-install.log up to the point of failure. https://pastebin.com/9w7QckqA

I guess something the installer wants is pulling in apache2 as a dependency.

Let me know if I can help with any testing.

Sat, 08/12/2017 - 10:12
Joe
Joe's picture

I haven't seen if you followed up where I answered this question in the beta thread, but did you try running install.sh --uninstall first, and then re-running the install with --bundle LEMP?

I can't find anything that would pull apache2 in on a LEMP installation, and LEMP installs are working on my test systems. But, if apache2 was installed before running install.sh it will not be removed. apt-get does not handle replacing package well at all. If you get rid of apache2 first, it should Just Work.

The LEMP bundle hasn't been very well-tested yet, so it's possible that I do need to just force apache2 to be removed for the LEMP stack. I hate to add even more special cases to deal with apt-get's ridiculous inability to deal with dependencies and conflicts, but it might be necessary to make it work more reliably.

--

Check out the forum guidelines!

Sat, 08/12/2017 - 14:23 (Reply to #2)
mikrobug

^^ It could be the order in which the packages are installed with the LEMP bundle.

Installing PHP7.0 on a clean Debian 9.x systems always ends up this way:

sudo apt install php7.0 -s
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  apache2 apache2-bin apache2-data apache2-utils libapache2-mod-php7.0 libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap liblua5.2-0 php7.0-cli php7.0-common php7.0-json php7.0-opcache php7.0-readline
[...]

But installing PHP7.0-FPM first resolves this problem:

sudo apt install php7.0-fpm -s
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  php7.0-cli php7.0-common php7.0-json php7.0-opcache php7.0-readline
[...]

So I installed PHP7.0-FPM in the first place and afterwards Virtualmin with the "minimal" and "lemp" option. So far my installation of Virtualmin is working as expected. ;)

Sat, 08/12/2017 - 17:18 (Reply to #3)
Joe
Joe's picture

apt-get/dpkg is ridiculous. That doesn't even make sense.

Nonetheless, the way we're installing things should not require any hoop-jumping. I can't reproduce the behavior in my test installs...we have a metapackage that depends on a list of things, so it should satisfy dependencies automatically based on what is being installed (php-fpm is being installed as a depdency of the metapackages we install).

But, I guess I need to force it to do the right thing. I'll update the install script to force uninstall apache2. dpkg really is annoying.

Also, in any case running with the --uninstall option should clean up whatever dependency mess in dpkg that is preventing installation. So, if anyone else is running into this, just run with --uninstall and then try it again. If it doesn't complete successfully after that, please let me know.

Next version will force all the crap out of the way before install the LEMP stack.

--

Check out the forum guidelines!

Sat, 08/12/2017 - 17:51
Joe
Joe's picture

OK, I'm able to reproduce this problem. I have no idea how to fix it, though. apt-get/dpkg just seem to refuse to resolve dependencies, even though they're explicitly required in the metapackages.

I'll keep poking at it.

--

Check out the forum guidelines!

Sat, 08/12/2017 - 22:47
Joe
Joe's picture

Egads, Debian 9 is a mess. There's a cascading pile of failures now that prevents removal of apache2, because firewalld can't restart because of some issue with fail2ban. It's a bug that's been fixed on RH systems. I'm trying to figure out if there's been a release that my system just isn't getting (because at this point apt-get can't even install or remove software because of all of the errors in interconnected services). I guess there must be some way to force removal and skip postremoval scripts...

Still working on it.

--

Check out the forum guidelines!

Sat, 08/12/2017 - 23:27
Joe
Joe's picture

OK, I think it's fixed. That was a harrowing experience, and i can't believe the ridiculous hoops I had to jump through to convince it to install cleanly if apache2 has ever been installed.

The problem appears to be that apache2 has broken triggers when firewalld or fail2ban are enabled (I don't know which one). It adds iptables firewall rules (using iptables directly rather than via firewalld), which causes the systemd units for firewalld and fail2ban to both be unable to restart, so any package action that required a restart of any of those services would fail. It looks like dependency problems based on the error, but it's not really that at all. I mean, it is, but the dependencies are properly specified, it's just any package action breaks so apacche2 couldn't be completely removed and nginx couldn't be completely installed.

Edit: You'll need to grab the install script again. Some changes are in the metapackages, but some of the nonsense has to happen step-by-step and not using apt-get dependency resolution, just to make sure things get turned off and uninstalled in the right order.

--

Check out the forum guidelines!

Sun, 08/13/2017 - 00:10
Joe
Joe's picture

I spoke too soon. apache is still being brought in. Still working on it...

--

Check out the forum guidelines!

Sun, 08/13/2017 - 07:48
Joe
Joe's picture

OK, so I don't know if this is the only problem, but I did track down a bug in the Debian fail2ban package, which I've reported to the Debian folks.

That bug is here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871993

I don't know how to work around it in a reasonable way. There's not really any way to safely change the fail2ban.service file (it'll get overwritten with any fail2ban update). I could create a new package with an alternative service file for fail2ban, but that's a huge maintenance burden and I'd have to maintain it effectively forever. It would also be poor usability; adding a weirdly named new service (like fail2ban-vm.service or something).

I'm gonna wait a day or two to see if anyone from Debian replies on the issue before I seriously begin to consider that route. I may try to come up with other solutions...

This is so confusing, as this issue didn't come up in my testing (with dozens of test installs on Debian 9), but then once I ran into it, I consistently began running into it. I don't know what changed on my test system to put it into a state where it would never work anymore (without some manual fiddling), but once it got there it is seemingly a permanent problem.

Anyway, this problem causes a few issues, including making apt-get installation fail for anything that has firewalld triggers (including nginx-full, and other packages we depend on). I don't know if it's the only thing causing installation failures of the LEMP stack, but it might be...since nginx-full can't configure, it may be installing apache2 to satisfy the httpd dependency of php packages.

--

Check out the forum guidelines!

Sun, 08/13/2017 - 16:12
noisemarine

Thanks for your work on this, Joe. Don't stress too much about it. LEMP is probably a tiny fraction of Virtualmin installs. I tried doing the --uninstall after the failed install, then tried installing again and it completed successfully. I haven't had a chance to try that with a clean system yet. If nothing else, there looks to be a functioning workaround that gets people going.

Mon, 08/14/2017 - 09:46
Joe
Joe's picture

I believe this problem is now fixed in the latest 6.0.3 version of the install script (it's what you get when you download from the standard links). I was able to do a dozen or so test installs without error on both Debian 9.1 and Ubuntu 16.04 last night before rolling it out, going through all possible permutations (uninstalling it, switching from LAMP to LEMP and vice versa, both minimal and full installs, etc.)

nginx with Virtualmin is more popular than I'd assumed until recently. I still prefer Apache, just because it's familiar and despite assertions to the contrary it isn't much less performant or efficient (in terms of CPU or memory) than nginx in the majority of cases, particularly if Apache is configured with its options as stripped down as the default nginx configuration. But, the wind blows the way of nginx, and we'll go where the wind blows, since nginx is also very good software. And, now that we have installer support, I expect a lot more users will choose it. I hope folks will read up to understand that it is still less capable than Apache (partly because we just haven't had feature requests to add some of the stuff that's been available in Virtualmin in Apache for years, and our development is very much driven by user feedback).

Anyway, it boiled down to two problems, which have been mentioned above:

  • The "PHP causes installation of Apache" problem identified by mikrobug.
  • The "fail2ban breaks firewalld and apt-get/dpkg" problem caused by the default fail2ban.service unit being broken (it depends on iptables.service, which doesn't even appear to be a thing on Debian, and is apparently an incorrect use of the PartOf option.

Either could cause an install failure because either can cause package installation to fail. And, it was pretty intermittent which one would cause a problem and under what circumstances.

I worked around it by forcing installation of fail2ban before everything else, just so I can shut the damned thing down (the installation auto-starts it, and when it's running it breaks any triggers that call firewalld, which several of the packages we install do). And, then it gets turned back on during the configuration step after all the packages have been installed.

Likewise I solved the php->apache2 problem by forcing installation of php-fpm before anything else. Again, this forces apt-get to behave in a reasonable way.

The problem with the latter solution is that it breaks the install script on both Ubuntu 14.04 and on Debian 7 (and maybe Debian 8, I haven't tested yet). So, that sucks, and will be part of my day today to straighten it out. I was so enjoying Debian 9 until all these ugly little warts showed up and reminded me why I still so strongly prefer yum/RPM. (I still think Debian 9 is a great release. The problems I've had are old and kinda ingrained in apt-get/dpkg and the packaging policy of conflating package installation and configuration, and so current maintainers can't really be blamed for them.)

--

Check out the forum guidelines!

Mon, 08/14/2017 - 12:02
mikrobug

Thumbs Up!

Thank you Joe, installation is working perfectly on a clean Debian 9.1 now. :)

Wed, 08/23/2017 - 09:50
billyjbryant

microbug, did you have issues with feature selections after installing Virtualmin with LEMP stack? When I select features it only gives me Apache based features, not NGINX.

Wed, 08/23/2017 - 12:26 (Reply to #13)
Joe
Joe's picture

That means you have Apache and not nginx. So, you don't actually have an nginx install. I don't know when you did the installation, but several bugs related to nginx have been fixed in the installer and in virtualmin-config. It might be worth starting from zero by running sh install.sh --uninstall and then trying the installation again.

I am still not super confident about the LEMP install bundle. I've been knocking out bugs as quickly as I find out about them, but nginx is a whole new barrel of monkeys to sort out.

--

Check out the forum guidelines!

Wed, 08/23/2017 - 14:09
billyjbryant

I just did the installation yesterday.

Wed, 08/23/2017 - 14:38
Joe
Joe's picture

More useful data is which apache2 packages are installed. Virtualmin is clearly detecting Apache, so I'm guessing it's installed somehow (though I'm running out of ideas for avoiding its installation...dependency resolution in apt/dpkg makes no sense...some things it'll simply refuse to install automatically to resolve dependencies, other things, I can't seem to stop it from installing them).

It'd be possible to tell Virtualmin to use nginx instead of Apache, but there's a bunch of stuff not getting done right if both Apache and nginx are installed, so we really need to just stop Apache from being installed at all when the LEMP stack is selected.

I'm doing some test installs now, and will hopefully be able to sort it out.

--

Check out the forum guidelines!

Wed, 08/23/2017 - 19:15
Joe
Joe's picture

OK, update your packages (there's a new virtualmin-lemp-stack metapackage and a new virtualmin-config package). That should also install the Virtualmin nginx plugins, webmin-virtualmin-nginx and webmin-virtualmin-nginx-ssl.

If that happens as expected, then run:

# virtualmin config-system --include Nginx

Then re-check the Virtualmin configuration, and see if things start to look more like they should.

--

Check out the forum guidelines!

Wed, 08/23/2017 - 22:11
billyjbryant

ok.