Troubleshooting HAproxy with Virtualmin problems

7 posts / 0 new
Last post
#1 Sat, 06/06/2015 - 05:06
Brook

Troubleshooting HAproxy with Virtualmin problems

Can anyone please help troubleshoot my haproxy/virtualmin problems please?

I followed the instructions here: https://www.virtualmin.com/documentation/system/varnish but instead of using Varnish, I am using HAproxy to divert traffic either to my Virtualmin sites (Apache port 8080) or a docker container.

However none of my virtualmin sites work (the Docker site works fine).

I have also played with the HAproxy config, switching the default_backend from loading my apache sites, to the docker site - it works for the docker site but not apache sites so the 'default_backend' definitely works (I commented out all other lines).

Additionally, when Apache is stopped, going to a virtualmin site says that a server can't be found - so it seems that HAproxy is actually diverting traffic the correct port.

Here is my HAproxy config:

global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2

    # chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000


frontend http-in
        bind *:80
        default_backend main_apache_sites

        # Define hosts
        acl host_discourse hdr(host) -i my_discourse_site.com

        # figure out which one to use
        use_backend discourse_docker if host_discourse

backend main_apache_sites
    server server1 127.0.0.1:8080 cookie A check

backend discourse_docker
    server server2 127.0.0.1:8888 cookie A check

All I get for my virtualmin sites is a blank page with:

Index of /

[ICO] Name Last modified Size Description

One thing I did notice is that on restarting Apache I get a number of these messages:

[Sat Jun 06 10:28:17 2015] [warn] VirtualHost 2a01:4f8:150:146c::2:80 overlaps with VirtualHost 2a01:4f8:150:146c::2:80, the first has precedence, perhaps you need a NameVirtualHost directive

..which I thought was because I hadn't changed the port settings for each domain (I am trying to get one working before changing all of the other domain settings) could this be an issue? (On previous attempts I did actually change all of the domains - but that was one of the first attempts).

Any help in sorting this would be greatly appreciated.

Mon, 06/08/2015 - 10:57
andreychek

Howdy,

When accessing your sites, do you see an attempt to access them located in $HOME/logs/access_log? If not, it may be trying to access the wrong domain.

That may indicated that in Apache, the domains aren't listening for requests from the correct IP address. If HAProxy is configured to make requests to 127.0.0.1, Apache would also need to be configured to do that, meaning that the "VirtualHost" configuration would need to contain that same IP address.

If that doesn't help -- let us know what the Apache configuration for one of the domains you're having problems with looks like.

-Eric

Mon, 06/08/2015 - 12:33
Brook

Hi Eric,

I checked the logs and yes, there is an access attempt (log contains correct user agent).

Here is a sample virtualhost file:

<VirtualHost 176.59.21.99:80 [2a01:5f8:150:146c::2]:80> SuexecUserGroup "#536" "#521" ServerName mydomain.com ServerAlias www.mydomain.com DocumentRoot /home/mydomain/public_html ErrorLog /var/log/virtualmin/mydomain.com_error_log CustomLog /var/log/virtualmin/mydomain.com_access_log combined ScriptAlias /cgi-bin/ /home/mydomain/cgi-bin/ DirectoryIndex index.html index.htm index.php index.php4 index.php5 <Directory /home/mydomain/public_html> Options -Indexes +IncludesNOEXEC +SymLinksIfOwnerMatch +ExecCGI allow from all AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch AddHandler fcgid-script .php AddHandler fcgid-script .php5 FCGIWrapper /home/mydomain/fcgi-bin/php5.fcgi .php FCGIWrapper /home/mydomain/fcgi-bin/php5.fcgi .php5 </Directory> <Directory /home/mydomain/cgi-bin> allow from all AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch </Directory> RemoveHandler .php RemoveHandler .php5 php_admin_value engine Off IPCCommTimeout 31 FcgidMaxRequestLen 1073741824 </VirtualHost>

Are you saying the first line should be changed to:

VirtualHost 127.0.0.1:8080 [2a01:5f8:150:146c::2]:80

(Not sure about the second bit)

Tue, 06/09/2015 - 07:16
Brook

Hello again Eric, adding the above manually works, but when you then change the port via Virtualmin > Server Configuration > Change IP it starts adding the IP and IPv6 addresses. Is there anyway to set Virtualmin up to reflect the correct settings?

These are my current settings:

Wembin > Servers > Apache > Global > Networking and Addresses:

LISTEN ON ADDRESSES AND PORTS: Address (blank - so listens for all IPs) Port: 8080 Address (blank - so listens for all IPs) Port: 443 (Note: I have removed the IPv6 address for now - is it needed?) ADDRESSES FOR NAME VIRTUAL SERVERS: *:8080 *:443

And then in the httpd.conf:

The first line of each VirtualHost is set to:

<VirtualHost *:8080>

Listen is set to:

Listen *:8080

I have also changed NameVirtualHost's to:

NameVirtualHost *:8080 NameVirtualHost *:443 (Note: I have removed the IPv6 address here too)

All that seems fine to me, but then when you change IP address for a domain via Virtualmin it starts adding the ip.add.ress:8080 and IPv6 lines again in Webmin > Servers > Apache > Global > Networking and Addresses and httpd.conf.

Do I need to change Virtualmin > System Settings > Virtualmin Configuration > Networking Settings to reflect the new settings too? If so what do you recommend for the following:

  • Network interface for virtual addresses
  • Default virtual server IP address
  • Default IP address for DNS records
  • Default virtual server IPv6 address
  • Network interface for IPv6 addresses
  • Netmask size for IPv6 addresses
  • Use default IPv6 address for new virtual servers?

Thanks Eric.

Sat, 06/13/2015 - 06:57
Brook

Hi Eric, I know you're busy but did you have any thoughts on this after?

Edit: No worries Eric, I found it!

Virtualmin > Virtualmin config > defaults for new domains > Address format for Apache virtual hosts

:-)

Edit2: Just in case anyone else needs it, you also need to forward IP addresses from the proxy, and so for apache you need to yum install mod_extract_forwarded see this: http://albertech.net/2014/03/preserve-remote-ip-address-with-haproxy-cen...

[quote] If you are using Apache, here are the instructions to preserve the remote IP address:

Install Apache Mod_Extract_Forwarded package: (you need to have the EPEL repository)

yum install mod_extract_forwarded Configure: /etc/httpd/conf.d/mod_extract_forwarded

MEFaccept 127.0.0.1 Replace the 127.0.0.1 with your proxy IP address (e.g. 192.168.50.50)

Restart Apache [/quote]

Sun, 06/14/2015 - 13:15
andreychek

My apologies for the delay -- I had marked your notification so that I'd remember to respond, but it ended up getting buried under a bunch of other notifications :-)

What did you end up setting "Address format for Apache virtual hosts" to?

Thanks for all the info you've shared about how you got it working!

-Eric

Sun, 06/14/2015 - 19:08
Brook

Hi Eric, no problem.

The first one: 'Always use *'

:-)