Lighttpd

Tested on 1.5.0-r1992 and 1.5.0-r2169 (current svn as of 13 May 2008). Hardware is an dual-core Athlon64 5000 with 3GB of RAM running Gentoo (2.6.23). I'm http reverse proxying (static) to a set of load-balanced 1.4.19 servers (behind LVS/keepalived) serving large static content (up to 200-250MB, flv video). Proxying a large file through the 1.5.0 machine results in excessive memory usage, far beyond what I would expect from a full memory cache. For example, after proxying one 195MB file, lighttpd is using 1995m of virtual memory, and 683m of resident memory (this from a startup memory usage of about 40m virt/1.5m res).

As I understand it, lighttpd is supposed to re-use this memory once allocated, so having that much allocated memory should be ok. However, the re-use appears to have a problem as well. Additional file requests (other files, or, after another file, the first file again) result in rapid additional memory allocation. Eventually, the server memory usage builds up to the point where the OS cleanly kills the lighttpd process due to lack of available memory. With large files (50-200MB) on a busy site this happens within a few minutes of server startup. Cutting off a connection in the middle on the client side, without finishing the download, seems to aggravate the problem; if a file finishes downloading, it appears that some memory is freed, but not all, and the total allocated still keeps going up until the process is killed.

So I fired up lighttpd under valgrind ... and the problem goes away. Memory allocated is similar to file size and the memory is re-used efficiently - allocated memory does not grow excessively and the server proxies a number of files (20ish) both partial and full without growing significantly in memory usage. I'm attaching the valgrind log, though I couldn't really find anything useful in it. Example config is also attached.