Lighttpd

We can´t tell, we would need a coredump or a trace of the events.

thank you for your answer

honestly, im affraid to do that, as it runs on live environment - does it have any impact on performance ?

if the impact is not major, can you please advice how to do that ?

To give you an idea about the load of the server, here are the server-status info

Uptime 1 days 1 hours 50 min 42 s
Started at 2009-09-14 07:32:24
absolute (since start)
Requests 2 Mreq
Traffic 38.11 Gbyte
average (since start)
Requests 23 req/s
Traffic 429.53 kbyte/s
average (5s sliding average)
Requests 57 req/s
Traffic 793.55 kbyte/s

the traffic is mostly php pages together with images and stylesheets from the design of the page, no fancy video or image gallery hosting service. Powered by mySQL database, two servers, dualcore for web, quadcore for db, both 4Gb rams, SAS hardware raid disks.

also, small question i noticed : when using multiple fast-cgi engines, the load is totally different for each of them

fastcgi.active-requests: 1
fastcgi.backend.localhost.0.connected: 90409
fastcgi.backend.localhost.0.died: 0
fastcgi.backend.localhost.0.disabled: 0
fastcgi.backend.localhost.0.load: 0
fastcgi.backend.localhost.0.overloaded: 0
fastcgi.backend.localhost.1.connected: 195773
fastcgi.backend.localhost.1.died: 0
fastcgi.backend.localhost.1.disabled: 0
fastcgi.backend.localhost.1.load: 0
fastcgi.backend.localhost.1.overloaded: 0
fastcgi.backend.localhost.2.connected: 391253
fastcgi.backend.localhost.2.died: 0
fastcgi.backend.localhost.2.disabled: 0
fastcgi.backend.localhost.2.load: 1
fastcgi.backend.localhost.2.overloaded: 0
fastcgi.backend.localhost.load: 1
fastcgi.requests: 677435

id expect the number to be +- equal for each. Why is the "last" engine used at most, while the "first" one is much less used ?

when working with the engines, i tried all possible combinations, having one and many child, having two, three with less child ... set lifetime of childs for 499 requests (to avoid that "500 served pages counter bug", set it to 10000 ... many combinations i came across while looking for solution to that Error 500 problem, but so far, i had no luck :( Upgrade seems to be my last chance, but it means to modify a live server, and for this project, I have no test environment due to money (its public non-profit service) - so im bit scared to do it.

Else LIGHTTPD is a great software for us for many years now, so i hope this can be fixed, im not planning to move to other webserver at all.

In 1.4.x lighty tends to stick to one backend, which is not really a bad thing, as this backend probably has a better cache than the other backends. It will move ofc to another one if one backend isn´t avail or loaded.

If you use an opcode-cacher like xcache, you should tend to fewer max-procs (1 or 2) and more children (like 8 or 16). Try to measure the time a normal request takes * requests per second = num of backends needed. E.g. 0.1s per request * 60 r/s -> 6 php children. This is ofc a naive calculation, so try 2 max-rpocs and 8 children and in combination with an opcode-cacher you should be on the safe side for requests of 0.1s avg. If its a known app, it might also come with some kind of caching strategy for db queries, maybe even for xcache´s var-cache (basically same as memcached without functionality for distributed caching over many servers).

I´d also choose a sane setting for max-requests, 499/500 in your case would mean a backend is killed and respawned about every 10s! So a setting of 5k or 10k probably won´t hurt.

Thats all i can suggest from the information avail.

thank you

i'm using APC cache, as the app was designed to use it. Using one proc with children or multiple procs with less children doesnt make any difference (im aware that having multiple procs means multiple APC memory allocation), and I agree that having max-requests set to 499/500 means kill and respawn very often for our load.

When doing the calculation, i get to approx 10 requests per second, but this isnt equally split during the day, there are peak hours when the number of requests rises to 60-80/sec, and there some hours where number of requests is hardly 5 per second. For this, i keep this setting

fastcgi.server = ( ".php" =>
                            ( "localhost" =>
                                    (
                                            "bin-path"  =>              "/usr/bin/php-cgi",
                                            "socket" => "/tmp/php.socket" + var.PID,
                                            "max-procs" => 3,
                                              "bin-environment" => (
                                                "PHP_FCGI_CHILDREN" => "15",
                                                "PHP_FCGI_MAX_REQUESTS" => "99999"),
                                                "bin-copy-environment" => (
                                                "PATH", "SHELL", "USER" ),
                                              "broken-scriptfilename" => "enable" 
                                    )
                            ))

but sometimes i end up with the 500 error.

I also found some users having problems when using APC cache, but some other stated they've run into this error even if they have XCache or some other cache, so im not sure APC is the cause of the problems.

this is the actual crash :

fastcgi.active-requests: 410
fastcgi.backend.localhost.0.connected: 578407
fastcgi.backend.localhost.0.died: 0
fastcgi.backend.localhost.0.disabled: 0
fastcgi.backend.localhost.0.load: 138
fastcgi.backend.localhost.0.overloaded: 34
fastcgi.backend.localhost.1.connected: 905653
fastcgi.backend.localhost.1.died: 0
fastcgi.backend.localhost.1.disabled: 0
fastcgi.backend.localhost.1.load: 134
fastcgi.backend.localhost.1.overloaded: 35
fastcgi.backend.localhost.2.connected: 2127438
fastcgi.backend.localhost.2.died: 0
fastcgi.backend.localhost.2.disabled: 0
fastcgi.backend.localhost.2.load: 138
fastcgi.backend.localhost.2.overloaded: 34
fastcgi.backend.localhost.load: 410
fastcgi.requests: 3611498

Server-Status
Hostname x ()
Uptime 5 days 22 hours 42 min 5 s
Started at 2009-09-15 12:14:46
absolute (since start)
Requests 11 Mreq
Traffic 202.87 Gbyte
average (since start)
Requests 21 req/s
Traffic 414.08 kbyte/s
average (5s sliding average)
Requests 3 req/s
Traffic 10.76 kbyte/s

legend
. = connect, C = close, E = hard error
r = read, R = read-POST, W = write, h = handle-request
q = request-start, Q = request-end
s = response-start, S = response-end
415 connections
hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhrhhhhhhhhhhhhhhhhhh
hhhhhhhhrhhhhrr

server goes balistic, and need lighttp stop and start, and also to kill all php-cgi processes that stay running after lighttp is stoped :(

Your php crashed. It might have only little to do with lighty, as its still serving static content.

You could try the following code-change and compile lighty from source:
http://paste.nerdbase.de/159 <- Please report back if this helped, we have no comments on this yet.

will try and report back ...

just to the note "your php crashed" - there is no notice about any kind of crash in weblog, where errors from PHP are sent - i see some minor error from some pages (usually caused by bad URL syntax - missing parameters etc), but no major breakdown. Then messages about backend being overloaded start to come up and it never recovers. No idea whats happening, im pretty sure we are not using any "special" PHP abilities, just pure mysql-select-display-update-delete mechanics ...

well, i've upgraded to spawn-fcgi and lighty 1.4.23, without the patch, but im not very happy with the results. As you see, before, i used to have this settings

"max-procs" => 3,
"bin-environment" => (
"PHP_FCGI_CHILDREN" => "15",
"PHP_FCGI_MAX_REQUESTS" => "99999"),

which was using up to 45 php-cgi to do the job. Now, i set up spawn-fcgi via supervise to look like this

exec 2>&1
exec /usr/local/bin/spawn-fcgi -n -s /var/run/fcgi.sock -ulighttpd -Ulighttpd -C 12  -- /usr/bin/php-cgi

as taken from the spawn wiki examples.

I tried to set -C parameter to range of 2 - 45. Setting it to 2 means the queue is growing wild almost immediatly. The pages are slow, and the queue grows ...

fastcgi.active-requests: 98
fastcgi.backend.localhost.0.connected: 470923
fastcgi.backend.localhost.0.died: 9
fastcgi.backend.localhost.0.disabled: 0
fastcgi.backend.localhost.0.load: 98
fastcgi.backend.localhost.0.overloaded: 30
fastcgi.backend.localhost.load: 98
fastcgi.requests: 470923

setting it to higher number than 12 means the server starts to show heavy load in TOP - on previous version of lightty, i hardly got load higher than 1.00, now within few minutes, i see load 2.8, but, whats worse, the queue grows too and the pages are slow again.

My current fast-cgi.conf

fastcgi.server = ( ".php" =>
                            ( "localhost" =>
                                    (
                                            "socket" => "/var/run/fcgi.sock",
                                            "max-procs" => 3,
                                              "bin-environment" => (
                                                "PHP_FCGI_CHILDREN" => "15",
                                                "PHP_FCGI_MAX_REQUESTS" => "99999"),
                                                "bin-copy-environment" => (
                                                "PATH", "SHELL", "USER" ),
                                              "broken-scriptfilename" => "enable" 
                                    )
                            ),

TOP shows PHP proccesses using 376Mb RAM each ?

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
22021 lighttpd  16   0  376m  35m  29m S    4  0.9   0:16.82 php-cgi
22160 lighttpd  17   0  375m  30m  25m S    4  0.8   0:14.16 php-cgi
22101 lighttpd  16   0  376m  36m  31m S    3  0.9   0:15.08 php-cgi
 2025 lighttpd  15   0 41096  17m  804 S    3  0.4  23:40.20 lighttpd

PS. the current lighty documentation is bit misleading, as it still speaks about spawning php-cgi internally (and putting the settings into lighty fast-cgi.conf file), but 1.4.23 version is not working that way anymore ... no clear example how to set that up in doc :(

fastcgi.active-requests: 98 <-- that seems very high, tho i have no idea about your setup, users and alike, but i "feel" its your problem. php is too slow, maybe the backend can´t cope with the load, maybe simple caching might help, maybe opcode caching like xcache. all i can say is its php crashing, the "h" state is lighty´s desperate effort to hand some requests over to php until it gives up and backlogg is full.

concentrate on php, your backend, caching and opcode-caching, 3x15 sounds reasonable, but its also really alot! for 60 r/s 2x8 or 2x16 should be reasonable.

what confuses me is, that if i use 3x15 (the "old server", lighty 1.4.20), the system works great, the pages are fast, it simply works ... if i use new lighty 1.4.23 ("new server") and go spawn-fcgi -C 45 , it slows down and doesnt work. I dont understand the relationship between the new spawn-fcgi settings and the settings in lighty

"PHP_FCGI_CHILDREN" => "15"

because, i dont see this particular settings working. How is it suppose to work ? if i start spawn-fcgi with -C 10 (10 php-cgi processes start, i see in process list), and have this php_fcgi_children set up to 15, does it mean there are 150 threads ready to handle php requests ?

because, in old server, when having settings 3x15, i see 45 php-cgi processes in proces list, 3 are launched by Lighttpd, and 15 are launched by each of those 3. In new server, i see -C processes launched by spawn-fcgi, but nothing more. Still, it lacks the performance.

So I guess i must have done some mistake in configuration of spawn-fcgi and ligty fast-cgi.conf, as seen above, but im not sure where.

(PS. Im using opcode caching - APC, caching doesnt help here (cant cache the result of php pages as they are changing frequently and static files are not affected by this 500 error), and to change php code - there is no reason for it, if it runs ok on the current server - there must be something wrong with the new server setting)

really no help here ? at least the correct fast-cgi spawn config for lighttpd and spawn-fcgi combination would be helpfull, explanation of some of the variables mentioned above and how they affect performance ? please ?

1.4.23 still supports internal spawning, so I don't see the reason to switch to spawn-fcgi.

there are several possibilities - internal spawning, spawn-fcgi, php own spawning ... i just havent found any solid comparation of these methods, what is better and why, so I stick with the lighttpd wiki how to set up php with lighttpd. But, wiki is somehow incosinstent about new/standalone spawn-fcgi method, I dont understand how to use some of the settings (PHP_FCGI_CHILDREN for example) in this environment and how they affect performance, thats why I ask for some example how to do make these two things (lighty and spawn) work together.

there is one interesting note to this issue - the server dies when the total amount of requests reaches 3 million ... as above, the server crashed when handling requests around this number

3.611.498

today, it crashed around

3.200.000

the current fast_cgi config (running the original 1.4.20 version with internal spawn)

server.modules += ("mod_fastcgi")
fastcgi.debug = 0
fastcgi.server = ( ".php" =>
                            ( "localhost" =>
                                    (
                                            "bin-path"  =>              "/usr/bin/php-cgi",
                                            "socket" => "/tmp/php.socket" + var.PID,
                                            "max-procs" => 3,
                                              "bin-environment" => (
                                                "PHP_FCGI_CHILDREN" => "15",
                                                "PHP_FCGI_MAX_REQUESTS" => "99999"),
                                                "bin-copy-environment" => (
                                                "PATH", "SHELL", "USER" ),
                                              "broken-scriptfilename" => "enable" 
                                    )
                            ),
                    ".phtml" =>
                            ( "localhost" =>
                                    (
                                            "bin-path"  =>              "/usr/bin/php-cgi",
                                            "socket" => "/tmp/php2.socket" + var.PID,
                                            "max-procs" => 1,
                                              "bin-environment" => (
                                                "PHP_FCGI_CHILDREN" => "10",
                                                "PHP_FCGI_MAX_REQUESTS" => "100"),
                                              "bin-copy-environment" => (
                                                "PATH", "SHELL", "USER" ),
                                              "broken-scriptfilename" => "enable" 
                                    )
                            )
                    )

another point is, that server-counters is not showing the stats correctly. As above, you can see i have 4 PHP procs running, but the server-counters shows only the three used in the first case.

fastcgi.active-requests: 0
fastcgi.backend.localhost.0.connected: 53
fastcgi.backend.localhost.0.died: 0
fastcgi.backend.localhost.0.disabled: 0
fastcgi.backend.localhost.0.load: 0
fastcgi.backend.localhost.0.overloaded: 0
fastcgi.backend.localhost.1.connected: 120
fastcgi.backend.localhost.1.died: 0
fastcgi.backend.localhost.1.disabled: 0
fastcgi.backend.localhost.1.load: 0
fastcgi.backend.localhost.1.overloaded: 0
fastcgi.backend.localhost.2.connected: 335
fastcgi.backend.localhost.2.died: 0
fastcgi.backend.localhost.2.disabled: 0
fastcgi.backend.localhost.2.load: 0
fastcgi.backend.localhost.2.overloaded: 0
fastcgi.backend.localhost.load: 2
fastcgi.requests: 508

See #1825 - I guess that bug is connected some way to this. If you want you could test current 1.4.x from svn (DevelSubversion), i hope i fixed it.