Project

General

Profile

Problems when reaching more than 80req/s

Added by dererkazo about 15 years ago

Hello there.
I would like to exchange some knowledge/problems about tweaking and fine tuning lighttpd to handle massively high requests serving static content (images/js/css).

I've handled happily perfectly traffic through my lighttps since a year or so, with huge amounts of traffic (40mbits/s w/2 dualcore servers), and different caching techniques from CDN to Proxy caches (at the same time). The problem showed up since two weeks ago, when the traffic reached from 10% to 20% more than the expected.

Incredibly fast, my setups failed at that load, massively affecting at response's time from my lighttpds. Since then I've been working hardly to get the most I can from them, but seems that I've reached a top response rate at no more than 80requests/s until it dies with, strangely, way different rates compared with inet benchmarks.

I'm running 1.4.22 on a rhel4, as minimalist as I can, just with zlib+bzip2+crypt+PCRE features activated, using sendfile network handler, with RO&&noatime filesystem attrs, with optimized tcp timeouts, etc.
Configured with --prefix=/usr/local/lighttpd --bindir=/usr/local/bin --sbindir=/usr/local/sbin --sysconfdir=/etc/lighttpd --datadir=/home/httpd/static --localstatedir=/var --mandir=/usr/local/man --infodir=/usr/share/info --with-pcre --with-zlib --with-bzip2 --with-rewrite --with-redirect --with-ssi --disable-ipv6 --with-linux-aio

This is the current config:
  1. Connections
    server.max-connections = 8192
    server.max-fds = 16384
    server.max-worker = 16
  2. Requests
    server.max-keep-alive-requests = 0
    server.max-keep-alive-idle = 0
    server.max-read-idle = 60
    server.max-write-idle = 360
    server.event-handler = "linux-sysepoll"

I've tried almost all possibilities lowing and raising fds and workers values, with no apparent difference.
The only thing I could do to make lighttpd to answer again is to restart it waiting a few seconds in the middle. I think this might be because of the queued requests waiting for syn-ack's... but I've sincerely don't understand why it dies at approx that number of requests or peaks.

Something to remark is that the 2 servers have one dualcore processor, and I don't really understand if this might be a problem in the way lighttpd workers behave.
Maybe that's why when I strace lighttpd's threads, most of them are idle all time, even at the time running threads stops responding to new requests.

I'm currently testing the FAM approach, but even though I've yet not certain numbers, seems not to help in the matter.
Any help would be very well received!

Thanks in advance.


Replies (2)

RE: Problems when reaching more than 80req/s - Added by icy about 15 years ago

In most cases, setting max-keep-alive-requests and max-keep-alive-idle to 0 isn't a good idea.
Suppose you have a site with 2 css files and 10 pictures. Then disabling keep alive will make browsers reconnect for every request.
I suggest setting max-keep-alive-requests to maybe 20 (high numbers here actually don't affect performance in a bad way) and max-keep-alive-idle to maybe 3.
Also max-worker is a dangerous option. Better not use that.
I know of at least one instance where a 1.4.22 lighty serves around 1000 req/s of static files (images/js/css) just like you do so it definately is capable of that (and much more :)).

The fact that you have to wait a few minutes before you can start lighty up successfully is interesting and could indicate that your kernel is running out of sockets.
Enabling keep-alive could help in this case as well as tuning the kernel options.

Check netstat -atpn | wc -l if this could be the case.

Hope this helps.

RE: Problems when reaching more than 80req/s - Added by MasterRoot about 15 years ago

Provides fine tuning for request handling.

max-connections == max-fds/2 (maybe /3)

As in, if you want to support 2048 concurrent connections, you may also want to set server.max-fds = 4096 (or 6144).
Why more file handles than connections? The other file handles are used for fastcgi and/or files.

    (1-2/2)