Lighttpd

php version which we used for testing is 5.2.5 (important because quite a bit has changed in 5.2.x)

it also has had this patch applied to it:

http://cvs.php.net/viewvc.cgi/php-src/main/SAPI.c?r1=1.202.2.7.2.15&r2=1.202.2.7.2.16&pathrev=PHP_5_2&diff_format=u

to eliminate a bug in 5.2.5.

refer here for detailed setup:

http://bugs.php.net/bug.php?id=43610

how to interpret a PROC_STATE_DIED_WAIT_FOR_PID the same as PROC_STATE_OVERLOADED.

I think it's a better idea to close the connection to the FCGI backend when it times out and not go into PROC_STATE_DIED_WAIT_FOR_PID.
If it really died, a connect() should refuse the connection and then you can go into PROC_STATE_DIED_WAIT_FOR_PID.

I think Lighttpd should be limited to open just PHP_FCGI_CHILDREN as PHP is not able to handle more connections.

@Olaf van der Spek

I think your suggestion is a good alternative to the patch I suggested. ie working around php's misbehaviour in an essentially similar way.

However I am still not convinced that php can't return FCGI_OVERLOADED when its request queue (over and above the number of already busy workers) grows above a configurable limit, see my reply to you here:

http://bugs.php.net/bug.php?id=43610

php cannot return FCGI_OVERLOADED because it's prefork, and staticly not dynamic forked, it won't grow. there's no "request queue" coded in php but os tcp listen backlog which php set to 128

timeout/busy issue should be fixed.
we should analyze the most if not all the cases that might happend even though we might make a simple solution at last, "Instead of", find a way that seems to fix this issue but might cause other new problem or leave other old problem not discovered/fixed

i haven't look deep into this problem yet, but imho, the solution should be defined as "convert timeout to busy" but "queue full server busy", and/or "too old (timeout) requests that queued drop, and response server busy"

1. queue?
   a. problem: when your client request rate <= server capability it's ok but once it burst some clients will sure get server busy error
   b. solution: queue delays clients' requests to after burst
2. timeout?
   a. problem: user and client won't wait for minutes for a request to begin download, they'll stop the connection.
   b. solution: drop requests that are still in queue, and return server busy to client.
3. drop requests when timeout or when queue full? should we drop new incoming requests so it won't be queued? or should we drop old requests that'are still in queue but not being handled by fcgi? should we timedout requests those still being handled?

but os tcp listen backlog which php set to 128

If persistent FCGI connections are used, that's not a good idea.

but imho, the solution should be defined as "convert timeout to busy"

What about:
I think Lighttpd should be limited to open just PHP_FCGI_CHILDREN as PHP is not able to handle more connections.

Replying to moo:

php cannot return FCGI_OVERLOADED because it's prefork, and staticly not dynamic forked, it
won't grow.

I am aware of this, but that is not the issue here. A static number of processes is fine, because at a certain number of requests (depending on your php app) the server just becomes CPU bound anyway.

there's no "request queue" coded in php but os tcp listen backlog which php set to 128

I was not aware of that and this does not quite make sense to me. I thought the FASTCGI spec allowed for many requests over the same socket, and I thought that the php parent dispatches these to its child worker procs. If this is not the case, then that would be a better design.

Replying to Olaf van der Spek:

but os tcp listen backlog which php set to 128

If persistent FCGI connections are used, that's not a good idea.

I agree

I was not aware of that and this does not quite make sense to me. I

thought the FASTCGI spec allowed for many requests over the same socket,
and I thought that the php parent dispatches these to its child worker
procs. If this is not the case, then that would be a better design.

Eh, no, that's not how it works. Each PHP child handles requests over exactly one connection. So there's almost no communication between the PHP parent and it's children.

oops, i was not focused when i wrote the first part of my reply. for those who are curious why php fcgi can't return FCGI_OVERLOADED: php parent bind() a port, fork child workers, and the child accept the connection from the bind()'ed socket (port) directly, not accept/handle requests from parent.

What about:
I think Lighttpd should be limited to open just PHP_FCGI_CHILDREN as PHP is not able to handle more connections.

yeah, this is a good idea to eliminate/skip/avoid using php tcp-backlog "queue", but queue have to be enabled in either php or lighttpd and make sure it works as expected.

but queue have to be enabled in either php or lighttpd and make sure it works as expected.

Why?
Currently Lighttpd doesn't use a queue either and when it hits the PHP queue, you're already running into trouble.

Replying to moo:

for those who are curious why php fcgi can't return FCGI_OVERLOADED: php parent bind() a port, fork child workers, and the child accept the connection from the bind()'ed socket (port) directly, not accept/handle requests from parent.

This seems like a very poor implementation to me. If php is serious about supporting a multi worker parent/child fcgi server SAPI, the surely it needs to implement some queue management and implement the FASTCGI spec properly. Never mind what it implements now (which seems like a hacky variant of CGI SAPI). Running apache/mod_php just does not scale (unless you run a separate apache for php requests only) so some scalable SAPI from the php crowd would be useful and the best candidate for that is fcgi. Has anyone come across these people:

http://php-fpm.anight.org/current_php_fastcgi_problems.html

or in (sort of) english ;-)

http://babelfish.altavista.com/babelfish/trurl_pagecontent?lp=ru_en&url=http%3A%2F%2Fphp-fpm.anight.org%2Fcurrent_php_fastcgi_problems.html

In the mean time lighty can just use some workaround strategy for detecting timeouts. Or just run php via spawn-fcgi as I suggested in the original post, which makes lighty/php behave nicely together under high load bursts. That is a very simple solution and moves you to the config you need anyway for multiple physical fcgi servers.

Replying to Olaf van der Spek:

Why?
Currently Lighttpd doesn't use a queue either and when it hits the PHP queue, you're already running into trouble.

there is php listen backlog, as i said already. and yes, i have some problem with current implemention. and reasons like this are why ppl say "it's safer to load server <=80% instead of >=90%~100%"

Replying to anonymous:

and reasons like this are why ppl say "it's safer to load server <=80% instead of >=90%~100%"

Our average load is more like 20% but there can always be unpredictable peaks.

there is php listen backlog, as i said already. and yes, i have some problem with current implemention. and reasons like this are why ppl say "it's safer to load server <=80% instead of >=90%~100%"

But you want to avoid hitting the listen backlog, because when you do, you run into this bug and the requests in the listen backlog aren't handled (for minutes).