Php-cgi ends using all avail memory and I got to reset apache each 24 hours :(

13 posts / 0 new
Last post
#1 Sat, 03/04/2017 - 10:24
sigma

Php-cgi ends using all avail memory and I got to reset apache each 24 hours :(

Howdy all,

This problem started some months ago. I did read almost every post about it and similar situations in here and other forums. I cannot seem to fix my problem with any solution that applied to other cases.

I am using FCGid, so each php executes as each virtual server owner. The number of running php-cgi keeps getting bigger, until all ram is eaten and other processes start shutting down (usually the first one is clamd)

My server got 9 Gb of assigned RAM. Webmin manages about 88 websites.

After much tinkering, my current httpd.conf is:

Timeout 60
KeepAlive On
MaxKeepAliveRequests 75
KeepAliveTimeout 3

<IfModule prefork.c>
StartServers       32
MinSpareServers    3
MaxSpareServers   32
ServerLimit     256
MaxClients      128
MaxRequestsPerChild  4096
</IfModule>

<IfModule worker.c>
StartServers         32
MaxClients        128
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  4096
</IfModule>

And (I did reset apache some 10 minutes ago) my server's memory usage is:

Private  +   Shared  =  RAM used Program

  4.0 KiB +   9.5 KiB =  13.5 KiB acpid
  4.0 KiB +  16.0 KiB =  20.0 KiB mysqld_safe
  4.0 KiB +  16.5 KiB =  20.5 KiB udevd
24.0 KiB +  51.0 KiB =  75.0 KiB mingetty (6)
132.0 KiB +  25.0 KiB = 157.0 KiB crond
140.0 KiB +  35.5 KiB = 175.5 KiB dovecot
152.0 KiB +  34.0 KiB = 186.0 KiB log
212.0 KiB +  26.0 KiB = 238.0 KiB auditd
160.0 KiB +  78.0 KiB = 238.0 KiB master
104.0 KiB + 149.5 KiB = 253.5 KiB mailmanctl
176.0 KiB +  79.0 KiB = 255.0 KiB qmgr
260.0 KiB +  23.0 KiB = 283.0 KiB init
488.0 KiB +  19.0 KiB = 507.0 KiB bash
532.0 KiB +  43.0 KiB = 575.0 KiB ssl-params
520.0 KiB + 114.5 KiB = 634.5 KiB miniserv.pl
932.0 KiB +  80.0 KiB =   1.0 MiB pickup
  1.0 MiB + 122.0 KiB =   1.1 MiB proxymap
  1.1 MiB + 120.0 KiB =   1.3 MiB anvil (2)
960.0 KiB + 480.5 KiB =   1.4 MiB saslauthd (5)
  1.7 MiB + 535.0 KiB =   2.3 MiB auth (2)
  2.3 MiB +  93.0 KiB =   2.4 MiB config
  2.4 MiB +  34.0 KiB =   2.4 MiB rsyslogd
  4.5 MiB +   1.2 MiB =   5.7 MiB smtpd (3)
  6.6 MiB +  71.5 KiB =   6.7 MiB named
  8.8 MiB +  10.4 MiB =  19.2 MiB sshd (30)
27.1 MiB + 994.0 KiB =  28.1 MiB python (8)
29.7 MiB +   2.6 MiB =  32.3 MiB perl (3)
38.2 MiB + 150.5 KiB =  38.3 MiB mysqld
40.4 MiB +   1.2 MiB =  41.6 MiB spamd (3)
65.4 MiB +  29.9 MiB =  95.3 MiB httpd (64)
371.6 MiB + 321.5 KiB = 371.9 MiB fail2ban-server
475.6 MiB +  61.0 KiB = 475.6 MiB clamd
  5.7 GiB +  24.5 MiB =   5.7 GiB php-cgi (103)
---------------------------------
                          6.8 GiB
=================================

In a couple of hours, php-cgi will eat up to 8gb of ram or so, and then "bad things will start to happen" (other processes start shutting down).

I am out of ideas. My knowledge of Linux and server intricacies is sadly enough not as good as it should. I am hoping one of the experts over here can suggest something in order to point me into the right direction.

Thanks,

Enrique.

Sat, 03/04/2017 - 11:06
Diabolico
Diabolico's picture

Check limits in php.ini and what settings you have with fcgid.

- I often come to the conclusion that my brain has too many tabs open. -
Failing at desktop publishing & graphic design since 1994.

Sat, 03/04/2017 - 13:33
sigma

Hi Diabolico,

Each virtual server has its own php.ini which I limited to:

max_execution_time = 30 
max_input_time = 60
memory_limit = 72M

Note: Memory limit was at 128M ... I downsized it into 64M but then several wordpress websites started showwing 500 errors, so I found the 72M was a low working value. Not that it did have any effect on my problem, anyways.

The rest of the parameters are as "they were" (I assume, default values for virtualmin / webmin).

About fcgid ... maybe I don't know where to check that !

Inside httpd.conf there is no mention of mod_fcgid.c ... Also, inside php.ini there is no mention of fcgid either.

Inside /etc/httpd/conf.d/ I found a fcgid.conf which is quite small:

LoadModule fcgid_module modules/mod_fcgid.so
AddHandler fcgid-script fcg fcgi fpl
FcgidIPCDir /var/run/mod_fcgid
FcgidProcessTableFile /var/run/mod_fcgid/fcgid_shm

As a side note, right now, 3 hours after my first post, memory and count usage on php-cgi has escalated into: (private - Shared - Total RAM used - Program)   7.6 GiB +  29.0 MiB =   7.6 GiB php-cgi (130)

Any help os greatly appreciated!

Thanks,

Enrique

Sat, 03/04/2017 - 20:54
Diabolico
Diabolico's picture

On Centos 7 fcgid settings can be found in /etc/httpd/conf.d/httpd.conf or /etc/httpd/conf.d/fcgid.conf. If you are using different distro then probably there will be some differences so try to find appropriate match for your OS. I remember on Centos fcgid was left to the default settings which are bad, with 1-2 settings inside httpd.conf (not even sure about this) and same for fcgid.conf.

Aside of this, you should check Apache access/error log files and (if missing) install network-tools so you can see how many active connections you have. It could even be someone is trying to mess around with your server with DOS, slow get/post requests or in some other way. Either way is worth to check your current active connections and see what is going on.

- I often come to the conclusion that my brain has too many tabs open. -
Failing at desktop publishing & graphic design since 1994.

Sat, 03/04/2017 - 23:33
sigma

Hi Again!

Thanks again, Diabolico, for your comments. I am using Centos 6.8.

My Apache access/error log files Seems pretty much on the clean side. I can the last apache reset as the last lines logged, so nothing to note.

By issuing a server-status on Apache, I can see about 122 server threads. like 1/2 of them are currently "legal" GET requests.

The rest are all the same:

virtualmin01.myserverdomain.com OPTIONS * HTTP/1.0

... being "virtualmin01.myserverdomain.com" my server.

Then, when the mod_fcgid status is listed, I can see a total of 143 FastCGI processes, distributed among the different virtual servers ...

I can see that some virtual servers got only one php5.fcgi assigned, while others got 2 or even 3 php5.fcgi active on them.

I would think that I need to lower that 143 FastCGI active processes into at most 128 instances or less.

But maybe I am wrong.

Anyway, I don't even know how to cap the number of FastCGI instances that my server may start.

At this time, my server declares the following memory usage on php-cgi:

  7.9 GiB +  29.1 MiB =   7.9 GiB php-cgi (143)

Regards,

Enrique

Sun, 03/05/2017 - 00:02
Diabolico
Diabolico's picture

Could you post here content of your fcgid.conf file and parts of httpd.conf containing fcgid lines (if any). It would help to know what kind of server is that (VPS or dedicated), how many cores and total ram.

- I often come to the conclusion that my brain has too many tabs open. -
Failing at desktop publishing & graphic design since 1994.

Sun, 03/05/2017 - 00:42
sigma

Diabolico,

It is a dedicated server with 16 gb RAM, an Intel SSD (3700 series) and an INTEL Xeon E3-1240 processor running the latest proxmox virtualization environment.

Inside proxmox I have a qemu based virtual machine which has the linux + virtualmin official ISO installed. Everything is updated to the latest official packages.

I already placed the entire fcgid.conf file content (minus comments) on my first answer to you. It is just 4 lines of settings:

LoadModule fcgid_module modules/mod_fcgid.so
AddHandler fcgid-script fcg fcgi fpl
FcgidIPCDir /var/run/mod_fcgid
FcgidProcessTableFile /var/run/mod_fcgid/fcgid_shm

There are no fcgid lines inside httpd.conf file.

I am suspecting that the relevant configuration lines are missing (if so, such is the way it came inside the official virtualmin iso - centos download I installed originally).

I would think that there are some commands that may cap the amount of php5.fcgi that each virtual server can span. If so, then I would -hopefully- be able to control the amount of ram being used by them.

Regards,

Enrique

Sun, 03/05/2017 - 03:02
Diabolico
Diabolico's picture

First of all make a off-server backup of any file you intend to modify. Saying this add next to your fcgid.conf file:

FcgidIdleTimeout 3
FcgidIdleScanInterval 1
FcgidBusyTimeout 300
FcgidBusyScanInterval 1
FcgidZombieScanInterval 1
FcgidErrorScanInterval 12
FcgidProcessLifeTime 300

FcgidSpawnScoreUpLimit 10

FcgidSpawnScore 1

FcgidTerminationScore 2

FcgidMaxRequestLen 1073741824

FcgidMaxRequestsPerProcess 100000
FcgidMinProcessesPerClass 0
FcgidMaxProcessesPerClass 20
FcgidMaxProcesses 40
FcgidIOTimeout 41

FcgidOutputBufferSize 1073741824

Just so you know, this isnt "one-win-button" for all and i'm sure you will need to modify this settings, still right now your primary problem is Apache taking all your memory so best would be to go step by step.

Once done be sure to re-check your httpd.conf file for any Fcgid* lines and remove them (backup old file!).

- I often come to the conclusion that my brain has too many tabs open. -
Failing at desktop publishing & graphic design since 1994.

Sun, 03/05/2017 - 08:43
sigma

Hi Diabolico! Thanks for the configuration for fcgid.conf . I did paste it and now reset Apache. I am eagerly awaiting for it to work a couple of hours in order to see if it improves. I am also searching for description on each configuration line you gave me, so I can better understand and eventually tweak each one.

Inside httpd.conf I finally found some Fcgid related config lines.

Each declared website includes the following lines:

AddHandler fcgid-script .php
AddHandler fcgid-script .php5
FCGIWrapper /home/(virtual server user)/fcgi-bin/php5.fcgi .php
FCGIWrapper /home/(virtual server user)/fcgi-bin/php5.fcgi .php5
AddType application/x-httpd-php .php
FcgidMaxRequestLen 1073741824

I did not remove any of those lines yet. I suspect that I may be spawning two fcgi-bin threads when only one is necessary, but then again it probably is my ignorance. As I type this, I am already searching for configuration examples and explanations on the web about how Fcgid needs to be configured / specified inside httpd.conf but of course, any idea, suggestion or comment is very much appreciated.

Regards and thank you again,

Enrique

Sun, 03/05/2017 - 09:51
Diabolico
Diabolico's picture

Ok, so:
1. FcgidMaxRequestLen you already have in fcgid.conf file so you can delete this line.
2. AddHandler, AddType and FCGIWrapper* leave it.

Settings i gave you (fcgid.conf) need to be modified depending on your server spec and traffic on your website(s), so this numbers are not set in stone. After you see the situation in few hours you can start the adjustments.

- I often come to the conclusion that my brain has too many tabs open. -
Failing at desktop publishing & graphic design since 1994.

Sun, 03/05/2017 - 13:18
sigma

DIabolico,

With your settings RAM usage went down a lot, that is the server never gets more than 3gb of ram in use between all its running services.

On the other hand, now the CPU usage has increased from 5-10% up to 30%-40% every time I check it.

I started reading about each parameter available on fcgid.conf.

Now I started tweaking / tuning while understanding each thing I touch:

Some of the parameters you gave me are the actual default values, so I erased those from the config file.

Right now I GOT:

FcgidMaxProcessesPerClass 20
FcgidMaxProcesses 90

FcgidIdleTimeout 3
FcgidMinProcessesPerClass 0
FcgidProcessLifeTime 30
FcgidIdleScanInterval 3

FcgidBusyScanInterval 3
FcgidBusyTimeout 300

FcgidZombieScanInterval 3
FcgidErrorScanInterval 6

FcgidMaxRequestsPerProcess 100000
FcgidOutputBufferSize 1073741824

-Happily enough, I suppose- now memory is on the low side (I am at about 30% of real memory usage). On the CPU side, It is oscillating between 14% and 30% usage.

I will closely follow server behavior today and tomorrow (tomorrow it will receive heavier traffic).

Thanks for your help, and of course if you notice something odd on the parameters I touched, please tell me so I can be aware of it!

Regards,

Enrique

Sun, 03/05/2017 - 22:24
Diabolico
Diabolico's picture

There is no harm if you leave settings with default values as this could change in some future software update, but its up to you.

For the CPU and the rest its just a game of balancing everything. For example, with 16GB memory you could easily dedicate 8-10GB (or more) for Apache and MySQL caching. This alone should decrease the overall pressure on your server but it will use much more memory. Still to have 10+GB of unused memory isnt doing any good and you should find the edge to accommodate everything. The problem is without live testing and not knowing your server details (like my.cnf file and many others) its really hard to make any smart suggestion, actually it could do more harm than good.

At least now you dont need to worry your Apache will crash if you do not restart every X hours, and that is good start.

- I often come to the conclusion that my brain has too many tabs open. -
Failing at desktop publishing & graphic design since 1994.

Mon, 03/06/2017 - 06:21
sigma

Exactly! Thanks !

Topic locked