Service Temporary Unavailable

16 posts / 0 new
Last post
#1 Sat, 09/22/2007 - 11:24
HamidHashemi

Service Temporary Unavailable

Hi,

After switching to fcgi I got a very serious problem which is going to be more and more after reach up to 500 Virtual Servers. Many PHP customers who are using fcgi php have this problem that after a while the php pages won't serve anymore and the error page "Service Temporary Unavailable" shown. meanwhile other virtual hosts php pages serving well with fcgi. I discovered that this problem only sovled when I restart httpd. Also while this error happened in the error_log I got this :

[warn] mod_fcgid: can't apply process slot for /home/domain/public_html/modules.php

How can I solve the problem ? Any suggestion ?

_Hamid

Sun, 09/23/2007 - 01:36
ah...lifes...good

I wonder if this is due to resource constraint - insufficient RAM?

Or something set as a default in the php.ini

Anyway, this sounds serious enough to warrant a submission of bug report at "Bugs and Issues" link below.

Sun, 09/23/2007 - 12:43 (Reply to #2)
HamidHashemi

There should be no resouce issue. Cause as I told before while a virtual host php pages can't serve with the mentioned error page other virtual hosts on the server working without any problem. Also running top saying that there is no server resource problem.
Also there is nothing in php.ini to make the apache fcgi won't work I think. It seems that there is a limitation in fcgi itself which cause this problem.
Joe can you approve that this can be a bug for submission ?

_Hamid

Mon, 09/24/2007 - 12:22 (Reply to #3)
Joe
Joe's picture

This may be an issue with the mod_fcgid settings. It's possible to set the number of child processes that mod_fcgid will spawn. I believe this is now configurable in the current version of Virtualmin.

Try browsing to Server Configuration:Website Options in the virtual server in question. Find the option labeled "PHP FCGId sub-processes", and set it to 8 or possibly higher. See if that resolves the issue. If not, it might be a bug in mod_fcgid.

What OS/version are you running? It may be able to roll up a new mod_fcgid package and see if that makes the problem go away.

--

Check out the forum guidelines!

Mon, 09/24/2007 - 15:06 (Reply to #4)
HamidHashemi

Joe,

I already tried the same but directly from fcgi-bin/php5.fcgi. I did edit the virtual host php5.fcgi file and changed the PHP_FCGI_CHILDREN to higher and higher numbers, but this just make more delay for the virtual host to have the issue again. This won't solve the problem.
I am using CentOS 4.5 with last yum update packages. I think this a the second big bug in mod_fcgid ( remember the previous one ? The fcgi-bin directory removal problem which makes the apache to won't came up anymore ). I think that we need to find a solution for mod_fcgid problem urgently as it is going to be a very big problem for big scale servers which handling more than 500 virtual hosts. I will be appreciated if you just put it as very high priority task in your tasks for solving.

_Hamid

Mon, 09/24/2007 - 15:27 (Reply to #5)
Joe
Joe's picture

Hey Hamid,

I'll try rolling out a new version of mod_fcgid for CentOS 4 and we'll see what happens. I'll ping this thread when it's available.

--

Check out the forum guidelines!

Tue, 09/25/2007 - 01:57 (Reply to #6)
webinger

Do you have to much fcgi processes?
I Had the same problem. I Commented PHP_FCGI_CHILDREN=4 out and never had any memory problems or sth like that again. What is your Apache KeepAliveTimeout? Try lower values like:

KeepAliveTimeout 15
MaxKeepAliveRequests 200

regards Martin

Tue, 09/25/2007 - 09:49 (Reply to #7)
Joe
Joe's picture

Thanks for chiming in, Martin. Those are also good ideas--it approaches the problem from higher up, in Apache, where we know things are pretty reliable. ;-)

I've also rolled out version 2.2 of mod_fcgid to CentOS 4 and 5 last night. Give it a try.

--

Check out the forum guidelines!

Tue, 09/25/2007 - 13:06 (Reply to #8)
HamidHashemi

Martin,
Thanks for your suggestions. Yes ! I have too many fcgi processes which related to different virtual servers. I checked the httpd.conf and it seems that default Virtualmin httpd configuration is to make KeepAlive Off by default and that's why my KeepAlive is set to off on my server. Do you suggest to change to On or what ?
Anyway the MaxKeepAliveRequests is 100 and KeepAliveTimeout is 15.

Joe,
Thanks for the new mod_fcgid package. I did update my mod_fcgid package and I am going to see the feedbacks from the customers. also I noticed that there is a new version of fcgid.conf available in the new package which is saved as fcgid.conf.rpmnew after the upgrade. I checked the diffrence with the old one and I discovered that the only change is a new line :
SharememPath run/fcgid_shm
I did change the old one to fcgid.conf.rpmsave and change the new one as fcgid.conf and restarted the httpd. Is that ok ?
I will give you update when I got the feedbacks.
Thanks for following up the issue.

_Hamid

Sun, 09/30/2007 - 12:53 (Reply to #9)
HamidHashemi

Dear Joe,

I experience the Service Temporary Unavailable problem again with the new package. I tried all of martin suggestion and it happened again also. Is there any way that we trace the problem and see which script or program have problem which cause the fcgi won't work for the virtual server anymore ?

_Hamid

Fri, 10/05/2007 - 10:36 (Reply to #10)
HamidHashemi

I checked /var/log/httpd/error_log while I had problem with one of my virtual host and found this error appears while the virtual host have problem :

[Fri Oct 05 22:59:27 2007] [notice] mod_fcgid: /home/domain/public_html/ebtekar/phpBB3/viewtopic.php total process count 62 >= 50, skip the spawn request

As you can see it says that the total process count is 62 which is more that 50 and it can't spawn any new request. While the site is standard phpBB forum which doesn't have so much load ! What can I do ? Please help . This is going to be a reason to loose the customers really.

_Hamid

Fri, 10/05/2007 - 11:01 (Reply to #11)
HamidHashemi

Joe,

I noticed that there is bug report in php which indicates the same problem :

http://bugs.php.net/bug.php?id=40286

can you confirm that this as same as we have here ? Can you please upgrade the php5 to php-5.2.4 to check if we still have the same problem or not ? It seems that there is some php security improvement in the new php release also.

_Hamid

Fri, 10/12/2007 - 10:36 (Reply to #12)
Joe
Joe's picture

Hey Hamid,

What OS/version are you on? I'll try to get an update into the repo in the next 24 hours or so.

--

Check out the forum guidelines!

Tue, 10/16/2007 - 12:36 (Reply to #13)
HamidHashemi

Joe,
I am still waiting for the PHP package in repository of CentOS 4.

_Hamid

Fri, 10/12/2007 - 14:41 (Reply to #14)
HamidHashemi

Hi Joe,

My OS is CentOS 4.5. I will be appreciate if you do so.

_Hamid

Tue, 09/25/2007 - 22:41
webinger

Hm no if you have turned KeepAlive off you should not change to on as i don't know your whole configuration and the number of simultaneous http connections your server's got to handle.
But i can tell you that raising the PHP_FCGI_CHILDREN value needs more memory. If you comment it out you have 1 fcgi process for one connection. If you set it to 4 you have 1 main process and 4 childs working for one connection.

A lot of FCGI users say that it is better to NOT set PHP_FCGI_CHILDREN then setting to a wrong value ;). Then only apache 'controls' PHP_FCGI processes. You could also set MaxRequestsPerChild Not to infinite. That prevents memory leaks on some systems.
And of course use your Max Client/Server Limit and your Start and Spare servers wisely and only load those Apache Modules you really need!

I also played with that configuration thingy for some time and got the same error when setting it high. I also had memory problems (2gig ram AMD64 Etch) and Service Temporary Unavailable. But the main problem on my system was the keep alive timeout, since i have a lot of users that stay for a long time. Now i have about 600MB RAM in use. I am running postix and some virtual servers which produce an average of about 30 simultaneous connections. Nothing big until now.

Maybe your configration is good and you are 'only' running dirty php code on your server. ^^ Check it ;)

regards Martin