Apache appeared to hit OOM due to ungraceful restart

Hi,

We have observed a strange behaviour on a customer's virtual server. In the Apache error logs, after every daily cronjob, there was this:

[Sat Jan 31 06:25:08 2015] [notice] Graceful restart requested, doing restart
[Sat Jan 31 06:25:08 2015] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)

It seems, from a bit of digging, that this could be related to htcacheclean and/or apachectl not properly closing off FCGId PHP processes, causing more to build up day after day.

As an interim measure, we have put in a full daily Apache restart at 7am. Ideally, of course, it'd be good to see if this issue can be resolved and remove the non-standard cron.

Could you suggest any pointers for investigating/fixing this?

Many thanks, Steve

Status: 
Active

Comments

Howdy -- hmm, I'm not quite sure what might cause that, though I might start by looking at two things. What is the output of these two commands:

dpkg -l 'apache*'
ls /etc/apache2/mods-enabled

Also, you do always have the option of switching to CGI rather than FCGID, if the performance difference isn't that great in your case. Many websites don't experience a noticeable performance change when switching from one to another.

I'll switch to CGId and see how we go on that.

Here's the output:

# dpkg -l 'apache*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                            Version              Architecture         Description
+++-===============================-====================-====================-===================================================================
un  apache                          <none>                                    (no description available)
un  apache-common                   <none>                                    (no description available)
un  apache-utils                    <none>                                    (no description available)
ii  apache2                         2.2.22-13+deb7u4     amd64                Apache HTTP Server metapackage
un  apache2-common                  <none>                                    (no description available)
ii  apache2-doc                     2.2.22-13+deb7u4     all                  Apache HTTP Server documentation
un  apache2-mpm                     <none>                                    (no description available)
un  apache2-mpm-event               <none>                                    (no description available)
un  apache2-mpm-itk                 <none>                                    (no description available)
ii  apache2-mpm-prefork             2.2.22-13+deb7u4     amd64                Apache HTTP Server - traditional non-threaded model
un  apache2-mpm-worker              <none>                                    (no description available)
un  apache2-suexec                  <none>                                    (no description available)
ii  apache2-suexec-custom           2.2.22-13+deb7u4     amd64                Configurable suexec program for Apache 2 mod_suexec
ii  apache2-utils                   2.2.22-13+deb7u4     amd64                utility programs for webservers
ii  apache2.2-bin                   2.2.22-13+deb7u4     amd64                Apache HTTP Server common binary files
ii  apache2.2-common                2.2.22-13+deb7u4     amd64                Apache HTTP Server common files


# ls /etc/apache2/mods-enabled
actions.conf      authz_default.load    cgi.load      deflate.load    mem_cache.load    proxy_balancer.conf  reqtimeout.load  status.conf
actions.load      authz_groupfile.load  dav_fs.conf   dir.conf        mime.conf         proxy_balancer.load  rewrite.load     status.load
alias.conf        authz_host.load       dav_fs.load   dir.load        mime.load         proxy.conf           ruby.load        suexec.load
alias.load        authz_user.load       dav.load      env.load        negotiation.conf  proxy_connect.load   setenvif.conf
auth_basic.load   autoindex.conf        dav_svn.conf  fcgid.conf      negotiation.load  proxy_http.load      setenvif.load
auth_digest.load  autoindex.load        dav_svn.load  fcgid.load      php5.conf         proxy.load           ssl.conf
authn_file.load   cache.load            deflate.conf  mem_cache.conf  php5.load         reqtimeout.conf      ssl.load

Many thanks!

Steve

I don't see anything unusual about your Apache version or the modules that are running. Has that issue been an on-going problem, or did it just occur once?

Hi,

Sorry for the mega-late reply. At the time, this issue hadn't persisted for a long time - it was just discovered over the course of a day or two and this "fix" put in place.

Since I set up this restart cron job, it's been running fine. I think it's probably worth leaving this as-is for now, rather than mess about with it. Clearly, it's been 5-odd months since the problem was discovered/fixed and no issues have occurred since.

Cheers, Steve

I'm glad it's been working for you! I'll go ahead and mark this as fixed, but feel free to let us know if you have any additional questions.