Virtualmin php5-cgi memory issue

13 posts / 0 new
Last post
#1 Sat, 02/08/2014 - 03:13
nisse

Virtualmin php5-cgi memory issue

So, our server is maxing out memory pretty fast. And since it is on a shared host (VPS) i think services are terminated when they reach the hard memory limit.

Anyways, looking at the proccesses it looks like this from a snapshot: Should they really take 70m ? Can i reduce this somehow?

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5268 mysql 15 0 343m 79m 8272 S 0 7.8 15:29.21 mysqld
12115 page1 15 0 205m 70m 7520 S 0 6.9 0:04.12 php5-cgi
12113 page1 16 0 191m 56m 7024 S 0 5.5 0:03.14 php5-cgi
3094 page2 15 0 182m 48m 7132 S 0 4.7 0:40.71 php5-cgi
3706 page4 15 0 180m 45m 7776 S 0 4.5 3:12.19 php5-cgi
17860 page5 15 0 180m 45m 7080 S 0 4.5 1:52.96 php5-cgi
3454 page5 15 0 180m 45m 7012 S 0 4.4 0:25.02 php5-cgi
28364 page6 15 0 177m 41m 7112 S 0 4.1 0:31.82 php5-cgi
28356 page6 15 0 175m 40m 7128 S 0 4.0 0:32.77 php5-cgi
29979 page2 15 0 174m 40m 8192 S 0 3.9 5:46.05 php5-cgi
20299 page6 15 0 173m 40m 6768 S 0 3.9 0:00.64 php5-cgi
3184 page2 15 0 173m 39m 6960 S 0 3.9 0:03.89 php5-cgi
3185 page2 15 0 172m 39m 6784 S 0 3.9 0:01.78 php5-cgi
7574 page2 15 0 174m 39m 7208 S 0 3.9 0:13.02 php5-cgi
3160 page2 16 0 172m 39m 6920 S 0 3.9 0:03.40 php5-cgi
13749 page2 15 0 170m 35m 7200 S 0 3.5 0:07.47 php5-cgi
26101 page2 15 0 166m 31m 7600 S 0 3.1 3:12.07 php5-cgi
1433 root 18 0 260m 27m 19m S 0 2.7 3:32.30 apache2
30566 page 15 0 155m 19m 6880 S 0 2.0 0:00.70 php5-cgi
30574 page5 15 0 152m 19m 6808 S 0 1.9 0:00.50 php5-cgi
24040 root 18 0 80356 17m 1872 S 0 1.7 0:09.90 perl
14227 page3 15 0 148m 12m 6860 S 0 1.2 0:02.16 php5-cgi
18398 page3 15 0 148m 12m 6776 S 0 1.2 0:00.13 php5-cgi
18232 page3 15 0 148m 12m 6804 S 0 1.2 0:00.36 php5-cgi
28286 list 18 0 55784 12m 3204 S 0 1.2 0:07.41 python
19679 www-data 18 0 261m 12m 2732 S 0 1.2 0:00.08 apache2
28289 list 18 0 55456 12m 3248 S 0 1.2 0:06.99 python
28283 list 18 0 55224 11m 3160 S 0 1.1 0:07.16 python
21668 www-data 15 0 261m 11m 2160 S 0 1.1 0:00.00 apache2
28280 list 18 0 55156 11m 3172 S 0 1.1 0:06.92 python
28281 list 18 0 54956 11m 3168 S 0 1.1 0:07.30 python
21666 www-data 15 0 261m 11m 2140 S 0 1.1 0:00.00 apache2
21665 www-data 18 0 261m 11m 2108 S 0 1.1 0:00.00 apache2
21650 www-data 15 0 261m 11m 2104 S 0 1.1 0:00.01 apache2
21670 www-data 18 0 261m 11m 2172 S 0 1.1 0:00.00 apache2
21671 www-data 15 0 261m 11m 2124 S 0 1.1 0:00.00 apache2
21667 www-data 15 0 261m 11m 2156 S 0 1.1 0:00.00 apache2

Sat, 02/08/2014 - 03:45
Hal9000

I had various problems with fcgid over the years related to memory and process management... I eventually gave up and now run everything in cgi mode. It's slightly slower but whatever, still fast enough.

Sat, 02/08/2014 - 05:29
Locutus

@nisse: Do you actually have memory issues? Have processes been killed due to out-of-memory (you'll find that in the kernel log if it happens)?

What's the output of free -m (put it in [code][/code] tags please)?

Having a few php-cgi processes with around 70 MB of RAM is rather normal, situation is the same on my server. Not sure how you can reduce its memory usage, except for turning off PHP caches if you are using one.

Sat, 02/08/2014 - 08:29 (Reply to #3)
nisse

Well services randomly stop working... i suspected it was due to memory issues. Where exactly should i look for that in kernel log? And which log file is that? On ubuntu btw.

Now i just restarted apache2 so the memory from free -m might is not what i usually is at.

             total       used       free     shared    buffers     cached
Mem:          1024        431        592          0          0          0
-/+ buffers/cache:        431        592
Swap:            0          0          0

Usually there is like 100 free and rest used...

Sat, 02/08/2014 - 07:31
rapidwebs

the daemon pool is just that: it will stay resident in memory for on demand usage which scales inherently over time;

after taking the advice posted above, take this in to account: depending on the manner in which you read the process output, you may be noticing one of a few different things: for example, and first off, you may be reading the output wrong.

it may be describing the entire pool, or each of its threads- not processes. or maybe even any memory it may have cached from the file system for later use. cached memory is usually temporal and not dedicated to the application, and may be reused on the fly-

also, considering it is a dynamic worker pool, it will take some configuration to get it working properly with your setup. unfortunately the default config. out of the box is designed for a somewhat powerful server, and dedicated towards one or few high traffic sites;

what does your /etc/apache2/mod-available/fcgid.conf look like? here is an example, which should scale well on a Shared Server (using mpm_event in my case..):

<IfModule mod_fcgid.c>
  AddHandler fcgid-script .fcgi
  FcgidMaxRequestsPerProcess 512
  FcgidMaxProcesses 10
  FcgidProcessLifeTime 150
  FcgidIOTimeout 45
  FcgidIdleTimeout 45
  FcgidIdleScanInterval 45
  FcgidBusyTimeout 60
  FcgidBusyScanInterval 30
  FcgidErrorScanInterval 30
  FcgidZombieScanInterval 25
  FcgidMinProcessesPerClass 0
  FcgidMaxProcessesPerClass 3
  FcgidMaxRequestLen  5242880
  FcgidConnectTimeout 10
</IfModule>

p.s. Fail2Ban "says" its using over 1GB of memory on my box, however, i usually experience OOM conditions from things rather like ClamScan, etc.

Good Luck!

Sat, 02/08/2014 - 08:26
nisse

The fcgid.conf looks like this:

<IfModule mod_fcgid.c>
  AddHandler    fcgid-script .fcgi
  FcgidConnectTimeout 20
  MaxRequestsPerProcess 512
  MaxProcessCount 20
 
  # Number of seconds of idle time before a process is terminated
  IPCCommTimeout        240
  IdleTimeout           240
</IfModule>
Sat, 02/08/2014 - 19:55
rapidwebs

i would have to look at the man page for mod_fcgid, but i am fairly certain that the default values for some things are set fairly high. furthermore, these values (for example):

FcgidMinProcessesPerClass 0
  FcgidMaxProcessesPerClass 5

They specify the maximum number of processes spawned for each user, or "site" in your case. This, combined with the proper configuration of timeouts, and idle scans, should help terminate and rid the system of un-needed fcgi related processes and threads.

Each user can have either a Pool, or single processes, launched to handle each new request by apache2. worker pools are just a process which loads several threads, to process new requests without having to spin up a new process. each one will report the same amount of memory, but im also fairly certainly that they "share" this,

however, on ubuntu, you can do a few things to restrict the amount of memory that each process uses.

first off, php.ini specifies the maximum amount of memory each process can use. usually set to atleast 64MB; remember, that each thread does not use 64MB, each process will.

second, i am fairly certain you can restrict the amount of virtual, paged, and real memory allowed for use by each process, or thread, by using "ulimit" in /etc/default/apache2, or by adding the www-data, and site users to /etc/security/limits.conf

moreover, i think you could add "ulimit" to your init.d script for apache2 to restrict memory consumption as well.

note: if i got anything wrong here, someone can feel free to right my mistakes and interject at any time

^_^

Good luck!

Sat, 02/08/2014 - 20:12
rapidwebs

the idea i presume, for a shared server, would be to limit each User, or "Site", to at most just a few Processes, which terminate rather quickly. each process can use several Threads to increase scalability in high traffic scenarios.

If each user is entitled to 2 consecutive worker pools, which also have 3 threads each:

max processes = Users * 2 (Process Count)

max classes = max processes * 3 (Threads per Process)

note: i see you have no swap file. if you are running a container on openvz or proxmox, you should install and configure a monitor server like "Monit". sometimes, especially on a shared server, fcgid time out settings Need to be low, because if offending applications are using too much burst memory, applications will be killed seemingly at random.

monit, can detect if a PID dies, and launch the application if it ever goes down. it can also restart apache2 (for example) if it starts to use too much memory, or if it begins hogging CPU. this is a MUST on a virtualized container to ensure your websites stay online

Mon, 02/10/2014 - 02:48 (Reply to #8)
nisse

This sounds like what we are running into.

Any good URL to some guide on how to install this "monit" ?

Thanks!

Mon, 02/10/2014 - 02:35
lp86

Is this by any chance an OpenVZ VPS?

Do you get anything from this command? cat /proc/user_beancounters

also

uname -a

Apache memory usage in OpenVZ is crazy, there is a method to get it down quite a bit, but only if you are using OpenVZ.

Mon, 02/10/2014 - 02:44 (Reply to #10)
nisse

This is what we get:

cat /proc/user_beancounters
Version: 2.5
       uid  resource                     held              maxheld              barrier                limit              failcnt
     7304:  kmemsize                 30249144             30617957  9223372036854775807  9223372036854775807                    0
            lockedpages                     0                    0  9223372036854775807  9223372036854775807                    0
            privvmpages                326600               329256  9223372036854775807  9223372036854775807                    0
            shmpages                    11115                11131  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numproc                       102                  105  9223372036854775807  9223372036854775807                    0
            physpages                  210109               212479  9223372036854775807  9223372036854775807                    0
            vmguarpages                     0                    0  9223372036854775807  9223372036854775807                    0
            oomguarpages               210895               213265  9223372036854775807  9223372036854775807                    0
            numtcpsock                     23                   23  9223372036854775807  9223372036854775807                    0
            numflock                      206                  207  9223372036854775807  9223372036854775807                    0
            numpty                          1                    1  9223372036854775807  9223372036854775807                    0
            numsiginfo                      0                    1  9223372036854775807  9223372036854775807                    0
            tcpsndbuf                  919168               919168  9223372036854775807  9223372036854775807                    0
            tcprcvbuf                  376832               376832  9223372036854775807  9223372036854775807                    0
            othersockbuf               269904               320224  9223372036854775807  9223372036854775807                    0
            dgramrcvbuf                     0                 8472  9223372036854775807  9223372036854775807                    0
            numothersock                  191                  197  9223372036854775807  9223372036854775807                    0
            dcachesize                1836684              1858281  9223372036854775807  9223372036854775807                    0
            numfile                      7385                 7461  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0                    0                    0                    0
            dummy                           0                    0                    0                    0                    0
            dummy                           0                    0                    0                    0                    0
            numiptent                      14                   14  9223372036854775807  9223372036854775807                    0
uname -a
Linux OURDOMAIN 2.6.18-028stab092.1 #1 SMP Wed Jul 20 19:47:12 MSD 2011 x86_64 GNU/Linux
Mon, 02/10/2014 - 13:11 (Reply to #11)
lp86

Here is the information, I found this at IntoVPS a few years ago, it has helped tremendously.

There are cases when you may notice that the memory usage is too high inside your VPS without a specific reason.
This may happen because of how OpenVZ calculates the memory used by your applications, most visible with Java applications or other servers using threads (Apache with mpm_worker or mpm_event).
You can read more here: http://wiki.openvz.org/Stack_size
A solution as explained in the article above is to decrease the stack size since the default value is too high in almost all cases.
For example, we installed Apache with mpm_worker in a VPS with MaxClients set to 25 and the threads per child set to 25 also. While 25 concurrent requests is not much the memory usage for that Apache spiked to 300MB. That's a little too high.
We then edited /etc/init.d/httpd (on CentOS, for other distributions it may be called apache2) and added: ulimit -s 256  on a separate line in the file, at the beginning like this:

# Source function library.
. /etc/rc.d/init.d/functions
ulimit -s 128

if [ -f /etc/sysconfig/httpd ]; then
. /etc/sysconfig/httpd
fi
After restarting Apache, the total memory used went down to around 30MB, which is acceptable.
This also works for other applications like MySQL for example.
Mon, 02/10/2014 - 05:04
rapidwebs

monit is a really nice alternative to some other bloated monitor software available. i've heard a few minor complaints, but in all honesty, it has One file to configure, and once running, i have never experienced any issues. used in conjunction with the webmin service monitor for email and SMS: this makes a great pair!

on ubuntu, you can install monit using:

sudo apt-get install monit

the config is located under /etc/monit/monitrc, and everything you need is already located in this file (some lines are commented)

personally, i ignore any "alert", or email functionality (substituting webmin with its SMS functionality). and the only issue i had was with its "internal server". anyways, here is some working config:

## Monit has an embedded web server which can be used to view status of
## services monitored and manage services from a web interface. See the
## Monit Wiki if you want to enable SSL for the web server.
#
set httpd port 2812 and
    use address localhost  # only accept connection from localhost
    allow localhost        # allow localhost to connect to the server and
    allow admin:monit      # require user 'admin' with password 'monit'
    allow @monit           # allow users of group 'monit' to connect (rw)

ignore racking you brain over this block if you can: its primarily used by the service it self, and does not consume very much memory or constitute any additional overhead that i can tell..

(however, you may want to change "@monit" to your users primary group)

Finally, you will want to configure each Mission Critical application on your server. you do this by simply pointing Monit to the applications PID file under /var/run. if this file is deleted or returns 0, the application is considered "down", and the appropriate action is taken

check process apache with pidfile /var/run/apache2.pid
    start program = "/etc/init.d/apache2 start" with timeout 45 seconds
    stop program  = "/etc/init.d/apache2 stop"
    if cpu > 60% for 5 cycles then alert
    if cpu > 80% for 10 cycles then restart
    if totalmem > 512.0 MB for 15 cycles then restart
    if children > 99 then restart
    if loadavg(5min) greater than 25 for 15 cycles then restart

This is an example on how to monitor apache2. this will also, in turn, monitor any php-cgi processes, due to being launched by Apache2. atleast this is my understanding on how the fastcgid server is handled (and that it is killed when apache2 shuts down)

edit: ill have to look into whether or not the fastcgid server maintains its own PID, and if "totalmem" evaluates to "apache2 + all children processes". ether way, this is a good place to start, and should be done for most applications which can consume large amounts of memory and cause DOS or OOM conditions inside a virtualized container,

anyways, openVZ can be perfectly acceptable to use in a production environment, and i haven't had any problems: as long as you take the correct precautions!

Regards, - Steve

Topic locked