mod_fcgid and error 500

17 posts / 0 new
Last post
#1 Tue, 11/01/2011 - 19:20
MDS85

mod_fcgid and error 500

My website's traffic is starting to pick up, and more and more in my nightly logwatch reports am I seeing multiple Internal Server errors (500). It's due to mod_fcgid.

I've done my research, and luckily have found a (possible) solution on Apache's website: http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html

If you scroll down a little bit on that page, you'll see a section that perfectly matches my problem:
By default, PHP FastCGI processes exit after handling 500 requests, and they may exit after this module has already connected to the application and sent the next request. When that occurs, an error will be logged and 500 Internal Server Error will be returned to the client. This PHP behavior can be disabled by setting PHP_FCGI_MAX_REQUESTS to 0, but that can be a problem if the PHP application leaks resources. Alternatively, PHP_FCGI_MAX_REQUESTS can be set to a much higher value than the default to reduce the frequency of this problem. FcgidMaxRequestsPerProcess can be set to a value less than or equal to PHP_FCGI_MAX_REQUESTS to resolve the problem.
So I want to take their advice and set PHP_FCGI_MAX_REQUESTS to a much higher value than the default so that I may reduce the frequency of this problem.

The example they give is this:

#!/bin/sh
# Set desired PHP_FCGI_* environment variables.
# Example:
# PHP FastCGI processes exit after 500 requests by default.
PHP_FCGI_MAX_REQUESTS=10000
export PHP_FCGI_MAX_REQUESTS

Where should I add/modify the current wrapper in Virtualmin to apply this? I know that it matters which OS, so I am currently using Ubuntu 10.04 LTS which uses the same directory structure as Debian.

Thank you (and I hope this helps other people with semi-frequent 500 errors).

Tue, 11/01/2011 - 22:06
andreychek

Howdy,

You could set FcgidMaxRequestsPerProcess in the Apache config, in the .conf file for your domain (/etc/apache2/sites-enabled/domain.tld.conf).

And, PHP_FCGI_MAX_REQUESTS could be set in the php5.cgi (or php5.fcgi) script) in either $HOME/fcgi-bin or $HOME/cgi-bin.

That file is set to immutable for security reasons, so you'd first need to run "chattr -i FILENAME" in order to remove the immutable attribute.

-Eric

Sat, 11/05/2011 - 00:02
MDS85

Good stuff, thank you again Eric.

Sat, 11/05/2011 - 08:45
helpmin

You could also consider to switch apache prefork (process based) to apache worker (thread based). This is so supposed to be better performing (but can only be used with fcgid and not mod_php)

Sat, 11/05/2011 - 22:13
MDS85

Hi helpmin,

Funny story: I had actually used worker.c for the first half of the year, but when the new Apache security update was released the upgrade reverted my changes. I was going to restore worker.c, however it turns out that prefork.c is a lot faster on my system. Figure that. :-P

I haven't used mod_php in quite some time. Despite it's minor ticks, mod_fcgid is top notch. Even though my server is private (it's my hobby, not business) I just like the convenience it brings.

Cheers.

Tue, 11/08/2011 - 22:26
MDS85

I actually took helpmin's advice and switched my MPM back to worker.c; I figured I would give whoever uses Ubuntu or Debian some easy instructions:

You're going to need to use either CGI or FCGI, you can't use mod_php with this method.

  • * Go to Webmin > Servers > Apache Webserver > Configure Apache Modules
  • * Disable mod_php5
  • * Stop the Apache Server
  • * Open the File Manager and navigate to /usr/sbin/ and click on apache2, then click the "Info" button.
  • * Edit ../lib/apache2/mpm-prefork/apache2 to ../lib/apache2/mpm-worker/apache2 and save.
  • * Start Apache again
  • You'll notice that /usr/sbin/apache2 is nothing more than a symlink; Apache on Ubuntu comes with Prefork, Worker, Event and ITK MPMs already installed, to switch between them, all you need to do is change where the apache2 symlink points to.

    I hope this helps someone.

    Thu, 11/10/2011 - 21:26
    MDS85

    I just want to report that unfortunately this doesn't fix anything.. I'm still seeing the 500 errors. Oddly enough, even though the 500 gets recorded in the access log, there is nothing referring to it in the error log..

    Thu, 11/10/2011 - 22:22
    helpmin

    You are not very specific about the problem (for example exact error message) or your system specs or traffic (e.g. hits, visitors) etc

    It is possible that your system can not handle peak traffic, but hard to say without all the details. If you have a VPS there could be also IO wait issues.

    Fri, 11/11/2011 - 00:44
    MDS85

    That is the problem helpmin, there is no error message. The access log marks a connection as 500 instead or 200, but there is no relevant error log entry to go along with it.

    Examining the vague access log, the errors are all within 2 seconds of each other and are infrequent.

    IO is surely not the issue. It's a dedicated server with four RE3 disks in RAID10.

    Fri, 11/11/2011 - 03:04
    helpmin

    It would be still helpful to see the complete (relevant) log entries.

    Sat, 11/12/2011 - 21:58
    MDS85

    Sure thing... This is an example of what throws the 500.. however it can just be about any php script, like I said, it's random.

    /var/log/virtualmin/(redacted)_access_log:(redacted) - - [10/Nov/2011:03:37:58 -0600] "GET /showthread.php?t=24420&page=2&p=140635 HTTP/1.1" 500 39096 "-" "JS-Kit URL Resolver, http://js-kit.com/"
    /var/log/virtualmin/(redacted)_access_log:(redacted) - - [10/Nov/2011:03:37:59 -0600] "GET /showthread.php?t=24420&page=2&p=140635 HTTP/1.1" 200 11043 "-" "Twitterbot/1.0"
    /var/log/virtualmin/(redacted)_access_log:(redacted) - - [10/Nov/2011:03:37:59 -0600] "GET /showthread.php?t=24420&page=2&p=140635 HTTP/1.1" 500 33304 "-" "Mozilla/5.0 (compatible; TweetmemeBot/2.11; +http://tweetmeme.com/)"
    /var/log/virtualmin/(redacted)_access_log:(redacted) - - [10/Nov/2011:03:37:59 -0600] "GET /showthread.php?t=24420&page=2&p=140635 HTTP/1.1" 200 11046 "-" "Mozilla/5.0 (compatible"
    /var/log/virtualmin/(redacted)_access_log:(redacted) - - [10/Nov/2011:03:37:59 -0600] "GET /showthread.php?t=24420&page=2&p=140635 HTTP/1.1" 500 33304 "-" "Mozilla/5.0 (compatible; TweetmemeBot/2.11; +http://tweetmeme.com/)"
    /var/log/virtualmin/(redacted)_access_log:(redacted) - - [10/Nov/2011:03:38:02 -0600] "GET /showthread.php?t=24420&page=2&p=140635 HTTP/1.0" 200 68955 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"

    That's the access log. As you can see, one second it's 200 and the next it's 500. There is nothing in the error log, it's clean.

    I researched on the internet, and one common thing they recommended to do was to add PHP_Fix_Pathinfo_Enable 1 to the fcgid.conf file and then cgi.fix_pathinfo=1 to the php.ini file. I did this yesterday, and while everything worked still, however it produced more errors:

    Attempts to use known hacks by 31 hosts were logged 304 time(s) from:
       (redacted): 194 Time(s)
          ^null$ 194 Time(s)

    That's was my IP, so the "hack tool" warning is false, however once I removed cgi.fix_pathinfo=1, that false warning went away.

    This is all of the relevant fcgid stuff in my virtualhost:

    IdleTimeout 10800
    ProcessLifeTime 10800
    BusyTimeout 10800
    IPCCommTimeout 10800
    MaxRequestsPerProcess 10000

    And this is the contents of my fcgid.conf file:

    <IfModule mod_fcgid.c>
      AddHandler fcgid-script .fcgi
      FcgidConnectTimeout 20
      IdleScanInterval 300
      BusyScanInterval 300
      ZombieScanInterval 60
      PHP_Fix_Pathinfo_Enable 1
    </IfModule>

    I'm at wits end at this point.

    Sun, 11/13/2011 - 06:53
    helpmin

    No idea either. There should be something in the error log (without it hard to tell what is wrong). Maybe a problem with your Apache log settings?

    Mon, 11/14/2011 - 08:59
    MDS85

    Not that I'm aware of. I can force a regular 500, and it'll appear. I access a forbidden area and a 403 will appear, etc.. It's just the FCGID-based 500's that aren't appearing. It's really odd.

    I do know, that Ubuntu 10.04 (and by relation, Virtualmin) uses the old version of mod_fcgid. It's not the same version that the Apache Foundation released. So I'm hoping with the release of 12.04 and the newer mod_fcgid, this will be fixed. A quick Google search shows I'm not alone with this, so at least I know I'm not going crazy. :-D

    It really isn't degrading the quality of my site, it's just a random annoyance. I guess I can live with it until I understand the problem.

    Thanks Helpmin.. :-)

    Wed, 12/14/2011 - 16:37
    superdav42

    I was having this same problem with my sites using virtualmin with fcgid. It ocured with worker and prefork mpms. I was able to fix it by following what the docs said: FcgidMaxRequestsPerProcess should be <= PHP_FCGI_MAX_REQUESTS

    by default PHP_FCGI_MAX_REQUESTS = 500 So I set FcgidMaxRequestsPerProcess = 495 and no longer do I get random 500 errors.

    This is easy to troubleshoot using ab, I ran: ab -n 100000 -c 100 http://www.mysite.com/ on the server with generated lots of requests. Looking at the access_log relieved a handful of them generated 500 errors. After changed FcgidMaxRequestsPerProcess and ran ab there was no 500 errors in the access_log.

    I would think that FcgidMaxRequestsPerProcess should be set to less than PHP_FCGI_MAX_REQUESTS by default instead of unlimited. It seems like a very bad idea to send 500 errors as part of your server's regular operations.

    Fri, 03/15/2013 - 10:30
    eldk

    Hello,

    I have "non real" error 500 in access_log files.

    I was looking at what was occuring, when find those things about "non relevant 500 errors" when using mod_fcgid (only a 500 error in apache access log, but no error in error_log).

    I look at the php5.fcgi file, and found PHP_FCGI_MAX_REQUESTS = 99999.

    http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html#fcgidmaxrequestsper...

    I set PHP_FCGI_MAX_REQUESTS = 500 and modify apache domain.conf file with a value of FcgidMaxRequestsPerProcess = 495 (nothing set by default configuration ?).

    wait to look if those "irrelevant" error 500 messages still occure.

    Greatings,

    Eric

    Ubuntu 12.04 server

    Fri, 03/15/2013 - 12:15 (Reply to #15)
    eldk

    Hello,

    when changing the values as said in up post, server load seem to be much higher (x2 to x3), and the 500 error message still occure, but more often.

    With default config (PHP_FCGI_MAX_REQUESTS = 99999, no FcgidMaxRequestsPerProcess set in apachedomain.conf), I have a 500 error message every 1 hour. With changed values, It's one message every ten minutes.

    Look forward ...

    Greatings,

    Eric

    PS : no error in suexec, files permissions are good, everything seem to be ok, except those "false error message"

    PS2 : those messages are handled by search engine robot, some visitors with no luck, it's something to go deeper in ...

    PS3 : http://www.virtualmin.com/node/19921

    Fri, 02/14/2014 - 21:19 (Reply to #16)
    eldk

    Hello,

    I try to test my website with ab :

    "500 errors" may occur when the client break the connection, so I think there is nothing to do in server side.

    I have change the values as said here :

    "By default, PHP FastCGI processes exit after handling 500 requests, and they may exit after this module has already connected to the application and sent the next request. When that occurs, an error will be logged and 500 Internal Server Error will be returned to the client. This PHP behavior can be disabled by setting PHP_FCGI_MAX_REQUESTS to 0, but that can be a problem if the PHP application leaks resources. Alternatively, PHP_FCGI_MAX_REQUESTS can be set to a much higher value than the default to reduce the frequency of this problem. FcgidMaxRequestsPerProcess can be set to a value less than or equal to PHP_FCGI_MAX_REQUESTS to resolve the problem."

    There are two things : - one is the FastCGI behavior (we should give more request to process before exit) - the second is when the client break the connection or the request before all the page is load, a "500 error" is written in apache access log

    You should try with ab and ctrl+c : a 500 error is written all the time.

    So for the second, it should be a timeout for a crawler (googlebot, especially adsense bot ...), the visitor or something that break the connection.

    The problem should be elsewhere : a too long script, a page with big data ..

    Continue ...

    Greatings,

    Eric