[Solved] stream-response-body>0 results in lighttpd heap usage size of response body (=> upgrade to newer lighttpd version)
Added by dave4444 over 2 years ago
Hi,
Running lighttpd on debian arm (1.4.53-4+deb10u1)
I've got a memory constrained system that I also want to avoid wearing out the flash on and a CGI program that generates large response data (>100MB).
After reading the helpful Resource Tuning and stream-response-body docs, I've settled on using 'server.stream-response-body = 2' so that the CGI program will receive backpressue and no caching/buffering should occur in the system using up flash or RAM.
However, when I have stream-response-body > 0, the heap of lighttpd itself grows in the order of the response size while streaming the response and does not go back down.
ex:
<< start fresh >>
$ sudo systemctl restart lighttpd
<< initial ram footprint >>
$ ps u --user www-data
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
www-data 28850 0.7 1.0 9844 5144 ? Ss 17:16 0:00 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
<< download data from CGI program (~130MB) >>
<< ram usage after download >>
$ ps u --user www-data
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
www-data 28850 12.2 31.4 161984 157252 ? Ss 17:16 0:08 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
After generating and downloading the data in the CGI program, the RAM for lighttpd has grown by ~150MB and doesn't go back down.
Note that generating and downloading the data again does NOT increase RAM usage further, but it still remains well above the time-zero amount.
It appears that while setting stream-response-body > 0 has eliminated disk usage and buffering, the response data has ended up in the heap of lighttpd AND is now likely cached or heap fragmented.
Thoughts?
Replies (3)
RE: stream-response-body>0 results in lighttpd heap usage size of response body - Added by gstrauss over 2 years ago
Thoughts?
lighttpd 1.4.53 was released over 2 1/2 years ago, and there have been 7 full releases of lighttpd since. The latest lighttpd is lighttpd 1.4.60. Your distro is WAY, WAY behind, which is not unusual. That said, before spending huge amounts of your own time troubleshooting, and before you post to Forums, it would have been a really good idea to check if the issue was already identified and fixed. You might imagine how much those of us answering forums enjoy repeating ourselves or revisiting issues that were addressed a long, long time ago.
Hint: lighttpd 1.4.59 is available in buster-backports.
FYI: server.stream-response-body = 2
does not eliminate use of temporary files, but does apply backpressure.
In addition to that, you should consider using an in-memory filesystem for temporary files, e.g. server.upload-dirs = ("/dev/shm")
RE: stream-response-body>0 results in lighttpd heap usage size of response body - Added by dave4444 over 2 years ago
Yes, I know, developers always want their users to use the latest version. But in the real world users aren't the developers or package maintainers and rely on distributions and upstream providers.
In any event, you were of course right. Retesting with 1.4.59-1~bpo10+1 and it does not exhibit this problem.
<< on startup >>
$ ps u --user www-data
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
www-data 12186 1.0 0.5 4184 2628 ? Ss 19:28 0:00 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
<< after large streamed response >>
$ ps u --user www-data
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
www-data 12186 1.5 0.6 4656 3108 ? Ss 19:28 0:01 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
RE: stream-response-body>0 results in lighttpd heap usage size of response body - Added by gstrauss over 2 years ago
Yes, I know, developers always want their users to use the latest version. But in the real world users aren't the developers or package maintainers and rely on distributions and upstream providers.
Hah. Yes. For those more hesitant users, I suggest waiting a month after a new release, just in case there are any regressions. After that, there is a pretty good chance that if an issue exists that will affect many users, it will likely have been found and documented in the lighttpd issue tracker.
lighttpd 1.4.60, which was recently released, further reduces lighttpd memory footprint in some common scenarios.
Also, another option: if your CGI program has generated the response in a temporary file on disk, or in an in-memory filesystem, then you can configure lighttpd mod_cgi to accept X-Sendfile response header from the CGI. This can further reduce memory use, and lighttpd can send the response to the client by reading the file, instead of having the CGI program read the file and write to a pipe, and then for lighttpd to read from the pipe.