proxy: error FCGI: attempt to connect to 127.0.0.1 failed

26 posts / 0 new
Last post
#1 Sun, 07/28/2019 - 03:17
Mad_Turnip

proxy: error FCGI: attempt to connect to 127.0.0.1 failed

Hello,

Since yesterday I encounter a rather strange issue. I have 2 VPS on OVH. Yesterday after the virtualmin/webmin update and a restart all the websites using PHP were throwing this kind of error:

[Sun Jul 28 11:00:19.583179 2019] [proxy:error] [pid 11605:tid 140145000912640] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:8021 (*) failed
[Sun Jul 28 11:00:19.583228 2019] [proxy_fcgi:error] [pid 11605:tid 140145000912640] [client 80.110.94.10:38808] AH01079: failed to make connection to backend: localhost

HTML websites were working, Apache was working ok, PHP7.0-FPM was running.

This config was running ok like for 2 years ago, no issues.

www.conf:

[www]
user = www-data
group = www-data

listen = 9003

listen.owner = www-data
listen.group = www-data
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

Virtualhost config:

SetHandler proxy:fcgi://localhost:8021
  • I've let only relevant lines

All the virtualhosts are using another port for proxy:fcgi

The only solution I found to work is to go for every website to Server configuration - Website options and to change from FPM, save, then change back to FPM and everything is working again.

I did this yesterday for all the websites but after midnight (probably some cron, log rotate run) all the websites were again down with:

Service Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

I did the trick to change the execution mode and they are running again. I tried to run the log rotate again and the websites were still running.

After the trick with changing the execution mode, no config files are modified, apparently. The www.conf, the pool conf are unchanged, the Virtualhost conf, port etc are unchanged.

I'm runnin Ubuntu 16.04LTS, Linux 4.4.0-157-generic on x86_64, Apache version 2.4.18, PHP7.0-FPM

Please advise.

Thank you.

Sun, 07/28/2019 - 03:55
Mad_Turnip
php7.0-fpm.service: Unit entered failed state.
php7.0-fpm.service: Failed with result 'exit-code'.
polkitd(authority=local)[1504]: Unregistered Authentication Agent for unix-process:5033:17069 (system bus name
Sun, 07/28/2019 - 04:24
Mad_Turnip

i managed to start php7.0-fpm by resetting another website as the error was that address in use by another pool .... but the main issue is still active

Sun, 07/28/2019 - 05:14
anahata
anahata's picture

This happened to me (Debian 9) and I had to manually edit every file in /etc/php/7.0/fpm/pool.d to make the listening port match the "SetHandler proxy:fcgi://localhost:xxxx" line in the corresponding file in /etc/apache2/sites-available.

I'm told if you let Virtualmin do a 'check config' they all get screwed up again. I hope this is being treated as a bug and will get fixed.

Anahata www.treewind.co.uk West Yorkshire, UK

Sun, 07/28/2019 - 05:25
Mad_Turnip

LE:...

Sun, 07/28/2019 - 05:29
Mad_Turnip

indeed, I just run a check config and

Fixing port clash for PHP-FPM version 7.0.4

and now all are dead again ...

Mon, 07/29/2019 - 03:40
ayalon

I have the same problem! All VirtualHost went down. The only way to fix it was to change to FCGI and back PHP-FPM. I'm currently in holidays and the issue most probably was introduced by an update. I'm very scared, that it will break again. @Jamie: Could you have a look into this issue? I guess a lot of people with automatic update are affected.

I tried to switch to FCGI but this is not possible. For some reasons, I still tried to connect by using PHP-FPM:

[Mon Jul 29 10:15:53.018661 2019] [proxy:error] [pid 8453:tid 140485996254976] (2)No such file or directory: AH02454: FCGI: attempt to connect to Unix domain socket /run/php/php7.2-fpm.sock () failed [Mon Jul 29 10:15:53.018692 2019] [proxy_fcgi:error] [pid 8453:tid 140485996254976] [client 84.226.100.92:53003] AH01079: failed to make connection to backend: httpd-UDS [Mon Jul 29 10:23:26.110972 2019] [proxy:error] [pid 8453:tid 140485226133248] (2)No such file or directory: AH02454: FCGI: attempt to connect to Unix domain socket /run/php/php7.2-fpm.sock () failed [Mon Jul 29 10:23:26.111004 2019] [proxy_fcgi:error] [pid 8453:tid 140485226133248] [client 193.246.50.124:52463] AH01079: failed to make connection to backend: httpd-UDS

I'm still trying to get FCGI to work again as a fallback. Maybe the issue is /etc/apache2/conf-available/php7.2-fpm.conf.

Mon, 07/29/2019 - 03:55
ayalon

I think, this is a huge bug introduced by this "Fixing port clash for PHP-FPM version" task. This breaks all virtualhosts again. I only have PHP 7.2 on Ubuntu 18.04 and this broke all my virtualhosts.

Unfortunatly I'm not able to switch back to PHP FCGI. Eventhough everything seems to be correct setup, it still trys to connect via proxy to php-fpm.

Mon, 07/29/2019 - 03:57
Mad_Turnip

Hi,

@ayalon - I'm back to CGI wrapper, even if I have a performance impact, better slower then offline ...

I'll wait for an update from Virtualmin

Mon, 07/29/2019 - 04:00
ayalon

I had to rename

/etc/apache2/conf-available/php7.2-cgi.conf.conf to /etc/apache2/conf-available/php7.2-cgi.conf_disabled

/etc/apache2/conf-available/php7.2-fpm.conf to /etc/apache2/conf-available/php7.2-fpm.conf_disabled

Then sudo service apache2 restart.

This allowed me to switch to FCGI instead of PHP-FPM as a workaround.

Mon, 07/29/2019 - 04:01
ayalon

@Mad_Turnip: Will do the same and switch everything to FCGId. It's better that beeing offline.

Sun, 08/04/2019 - 09:36
Wira.Soenaryo

I register to comment..

I do have same issue.. using Ubuntu 18.04... have to go to website option, change to fcgi then back to fpm to make the webaite back online. The same with what others do.

Hope will fixed soon..

Sun, 08/18/2019 - 02:40
cahams

I have the same issue on centos 7.6, terrible this.

I fix ot switching from php 7.2 to php 5.6 and works inmediately, then I can change again to php 7.2

Sun, 08/18/2019 - 02:44
Mad_Turnip

probably until the next check config script.

Sun, 08/18/2019 - 03:40 (Reply to #14)
cahams

Yes, or the next reboot.

Someone know how to fix this? I have the correct ports between virtual servers in apache and his php-fpm pool but I can't make it working if I don't change mode or php version.

Thank you

Sun, 08/18/2019 - 03:23
adamjedgar

I think i too am having a similar issue. twice now after running virtualmin updates in the last month, php stops functioning after restart. The fix has been to go to just one of my VirtualServers>website options> change from FPM to FCGI and back again.

As soon as i do this, php starts working again.

I have made a post about mine in the Issue Tracker https://www.virtualmin.com/node/66880

AJECreative is the home of $5 webhosting, $15/month VPS servers (1cpu,1gb RAM, 25GB storage)
Centos7, Debian9, or Ubuntu18LTS
Available Control Panels = Centos-Webpanel, Cyberpanel, or Virtualmin

https://ajecreative.com.au

Sun, 08/18/2019 - 11:45 (Reply to #16)
eddieb

i cannot view this issue, adam

Sun, 08/18/2019 - 15:08 (Reply to #17)
adamjedgar

Sorry I inadvertently marked it private. You can read it now.

AJECreative is the home of $5 webhosting, $15/month VPS servers (1cpu,1gb RAM, 25GB storage)
Centos7, Debian9, or Ubuntu18LTS
Available Control Panels = Centos-Webpanel, Cyberpanel, or Virtualmin

https://ajecreative.com.au

Sun, 08/18/2019 - 11:43
eddieb

same issue here. everytime I update virtualmin i have this problem.

Sun, 08/18/2019 - 13:23
ayalon

Maybe you should read the whole issue and apply the workaround? Switching back and forward does not help. Use FCGI. PHP-FPM is broken since almost a month now.

Sun, 08/18/2019 - 14:31 (Reply to #20)
eddieb

I switched to fCGI. I don't want to mess with workarounds.

Sun, 08/18/2019 - 13:46
cahams

Hi again,

I think that I've located the problem, I'm going to try to explain it, sorry for my english.

I have Centos 7.x so the paths and versions conrrespond to this in my server.

I have two php versions availables, php 5.4 and php 7.2, the files of the pools for virtual servers are in:

5.4 path -> /etc/php-fpm.d/

7.2 path -> /etc/opt/rh/rh-php72/php-fpm.d/

When you use php 5.4 for a virtual server the file idvirtualserver.conf (123121232143xxx.conf) is in 5.4 path when you change to 7.2 that file is moved to 7.2 path. At this moment all works fine but there must be a cron or task that is changing internally this version 7.2 to version 5.4 and like the pool file isn't just in 5.4 path the server crash, if you copy the file idvirtualserver.conf from 7.2 path to 5.4 path works again but with this version instead of selected.

Anyone knows what script runs when we are changing the version of php in a virtual server? and the script that is changing this when reboot or is running every x minutes?

Thanks

Sat, 08/31/2019 - 11:06
aaronroydhouse

I think there is some problem with the FPM version detection and the 'port clash for PHP-FPM' test that causes any update or 'Re-check Configuration' to change the FPM port number of each website. I have only one FPM version installed (7.2.3), but Virtualmin detects it as two: The following PHP-FPM versions are available on this system : 7.2 7.2.3 The it changes the perfectly good and correct port numbers, breaking all the website. My example is below.

I have a fresh vanilla install of Virtualmin 6.07 on Ubuntu 18.04 with just one version of PHP (7.2.19) with PHP-FPM (7.2.3). It works fine until I run 'Re-check Configuration' and after that all PHP-FPM sites are broken.

Initially all is fine, the website config says:

<FilesMatch \.php$>
SetHandler proxy:fcgi://localhost:8000
</FilesMatch>

and /etc/php/7.2/fpm/pool.d/12345678901234.conf says:

listen = 8000

The website works great. Life is good. But every we time I run 'Re-Check Configuration' it says the following and all the PHP sites stop working:

The following PHP versions are available : 7.2.19 (/usr/bin/php-cgi7.2)

The following PHP-FPM versions are available on this system : 7.2 7.2.3

Fixing port clash for PHP-FPM version 7.2.3

Fixing port clash for PHP-FPM version 7.2.3

Restarting PHP-FPM server ..
.. done

Now the website config still says:

<FilesMatch \.php$>
SetHandler proxy:fcgi://localhost:8000
</FilesMatch>

But virtualmin has changed /etc/php/7.2/fpm/pool.d/12345678901234.conf to a different port!

listen = 8001

If I run 'Re-Check Configuration' it changes it to port 8002, and so on. Every time it changes the PHP-FPM port but does not update the website config.

If I manually change 800X back to 8000 and restart php-fpm that the website works again.

Seems like the FPM version check and 'port clash for PHP-FPM' clash detection is buggy?

Sat, 08/31/2019 - 11:20
aaronroydhouse

This looks like the bug report, apparently fix will be to stop changing the port numbers: https://github.com/virtualmin/virtualmin-gpl/issues/96

Sun, 11/10/2019 - 20:29
csscommunications

I was able to go into the apache configuration for the website and remove this line SetHandler proxy:fcgi://localhost:8001 at that point I could then go into website options and change it to fpm or fcgid

Thu, 11/14/2019 - 05:31
adamjedgar

has this been fixed at all yet? I just had to perform a system reboot (after a kernel update to debian 9), and i put this reboot off for a while...after rebooting, the newest of my client wordpress websites went down with the following error in virtualmin>Logs and reports>apache error log

[proxy:error] [pid 1342:tid 140691015382784] (111)Connection refused: AH00957: FCGI: attempt to connect to 127.0.0.1:8011 (*) failed

not sure if the following has any relevance but i saw it on another forum post about this kind of error

netstat -tulpn | grep 8011
tcp6       0      0 :::8011                 :::*                    LISTEN      3388/php-fpm: maste


this time, it appears that only one of the virtual servers php websites on my system went offline...the newest one (pain in the butt really because thats hardly a good look when a new client comes on board).

Also, all i had to do to get website backup again was goto Virtualmin>Server Configuration>Website Options and then simply click save (which restarted php-fpm for said website and it immediately came back online)

Perhaps there is some caching on other sites and not on new website...and i simply didnt wait long enough after reboot before checking wordpress websites to see if they were working or not?

AJECreative is the home of $5 webhosting, $15/month VPS servers (1cpu,1gb RAM, 25GB storage)
Centos7, Debian9, or Ubuntu18LTS
Available Control Panels = Centos-Webpanel, Cyberpanel, or Virtualmin

https://ajecreative.com.au