Project

General

Profile

Actions

Bug #2102

closed

Memory leak when using CGI

Added by mlcreech over 14 years ago. Updated over 14 years ago.

Status:
Duplicate
Priority:
Normal
Category:
mod_cgi
Target version:
-
ASK QUESTIONS IN Forums:

Description

Lighttpd appears to leak memory when executing a certain type of CGI program. I had this problem on an embedded PPC device with a fairly complex script, but boiled it down to a simple test case on the desktop.

Just enable mod_cgi, then create a script like this:

#!/bin/sh
echo "Content-type: text/plain" 
echo "" 
cat file.txt
cat file.txt

Ensure that file.txt is fairly large - a few hundred kB should do the trick. Requesting this page/script in a loop causes Lighttpd's RSS to jump upward very quickly, until eventually it's unable to allocate any more memory. For some reason having 2 "cat"s also seems to make a difference.

This problem happens on both 1.4.23 and 1.4.24 (haven't tried previous versions).


Files

memory-usage.txt (2.54 MB) memory-usage.txt Massif trace mlcreech, 2009-11-19 16:41
blah.cgi (74 Bytes) blah.cgi CGI script mlcreech, 2009-11-20 22:48
blah.csv (663 KB) blah.csv Sample text file mlcreech, 2009-11-20 22:48
lighttpd.conf (2.86 KB) lighttpd.conf Lighttpd config mlcreech, 2009-11-20 22:48
Actions #1

Updated by stbuehler over 14 years ago

  • Status changed from New to Duplicate

memory usage != memory leak.
Known problem - don't push big files from *cgi/proxy.

Actions #2

Updated by mlcreech over 14 years ago

How is it not a leak if pushing 500kB of output over and over causes Lighttpd to creep up to over 100MB of memory usage? Perhaps that memory is still accessible, so it's not technically "leaked", but the effect is the same...

Actions #3

Updated by darix over 14 years ago

you want a fastcgi application and x-lighttpd-send-file.

Actions #4

Updated by icy over 14 years ago

If pushing a 500kb file over cgi multiple times causes memory usage to climb further and further, then I think it's a leak or problem or whatever.

Actions #5

Updated by mlcreech over 14 years ago

Using a 650kB file.txt, I had "wget" requesting this CGI script in a loop ~3000 times, over the course of about 5 minutes. Lighttpd's actual memory usage (gleaned from VmRSS in /proc/`pidof lighttpd`/status) climbed steadily from less than 2MB to around 750MB, at which point I killed the script. It never seems to fall back to a reasonable level after that.

Actions #6

Updated by mlcreech over 14 years ago

darix: this example is obviously trivialized. :) In the real script, there's additional output computed in the script before and after the cat operations, and there are multiple files in different locations being pulled cat-ed. I don't think x-lighttpd-send-file would even work in such a situation.

Actions #7

Updated by darix over 14 years ago

mlcreech wrote:

darix: this example is obviously trivialized. :) In the real script, there's additional output computed in the script before and after the cat operations, and there are multiple files in different locations being pulled cat-ed. I don't think x-lighttpd-send-file would even work in such a situation.

well you can try to argue now. as already said. this is not a memory leak. lighttpd just buffers the response from your *CGI application in memory. if you request it in parallel it needs to buffer it multiple times. that memory is reused later and properly freed when the server is stopped.

as already mentioned you utilize x-lighttpd-send-file. in 1.5 we also got x-lighttpd-send-temp-file. where it would delete the file after sending.

last but not least you could utilize mod_magnet.

see http://redmine.lighttpd.net/wiki/1/AbsoLUAtion#Code-Snippets (e.g. the FLV streaming part)

Actions #8

Updated by mlcreech over 14 years ago

I did a run under "massif", in case that helps.

Interestingly, even though I used a larger file (2MB) to try and speed things up, when running under valgrind Lighttpd grows slower than it does when running natively. That is, slower per-request - obviously it's slower overall due to emulation. It might just be an artifact of valgrind instrumenting memory allocations (maybe it packs the requested memory, whereas the real-thing has much more internal fragmentation because performance is a priority). So even with the same # of requests (~3000), lighttpd only grew to a little over 250MB in this case.

Anyway, the results are attached in case that sheds some light on this.

Actions #9

Updated by icy over 14 years ago

Is your script doing parallel fetches (wget in background) or in order?
Maybe attach that too

Actions #10

Updated by mlcreech over 14 years ago

darix wrote:

well you can try to argue now. as already said. this is not a memory leak. lighttpd just buffers the response from your *CGI application in memory. if you request it in parallel it needs to buffer it multiple times. that memory is reused later and properly freed when the server is stopped.

as already mentioned you utilize x-lighttpd-send-file. in 1.5 we also got x-lighttpd-send-temp-file. where it would delete the file after sending.

I'm not requesting it in parallel, though. In fact, in the real-world case in which we encountered this, it was a CRON job being run once per hour on a client that eventually exhausted the server machine's memory. Growing continually without bound in response to repeated, serial requests for the exact same data is a "memory leak" in all but the most pedantic sense. If there's some kind of caveat in which Lighttpd grows indefinitely whenever you output more than N kB of data, we probably need to quantify exactly what N is. Otherwise I have no way of ensuring that my server doesn't eventually run out of memory because I happened to return just slightly too much data from a CGI script, or whatever.

And as I mentioned previously, I don't think X-lighttpd-send-file will work when inter-mingling multiple files and non-file output (though correct me if I'm wrong).

Actions #11

Updated by mlcreech over 14 years ago

icy wrote:

Is your script doing parallel fetches (wget in background) or in order?
Maybe attach that too

No, in fact all I'm doing is:

while [ 1 ]; do
   wget http://localhost/test.cgi
done
Actions #12

Updated by stbuehler over 14 years ago

I just tried it and couldn't even reproduce it - didn't even reach 4mb RES. (running two loops brought lighty to 6.5mb RES).

Actions #13

Updated by mlcreech over 14 years ago

Interesting... I'll attach my exact CGI script and file, though I'm not sure whether that'll make a difference.

This is a x86_64 machine, I'm running coreutils 7.5 and bash 4.0. Though I see the same problem on a ppc machine with BusyBox/ash, so I doubt those are related.

One thing shared between test platforms is that they're pretty up to date - kernel 2.6.31 or above, glibc 2.10.1, all built with GCC 4.4.

Let me know if there's any other information I can provide or tests you want me to run, it happens 100% of the time here. Thanks!

Actions #15

Updated by mlcreech over 14 years ago

Also, I found that this problem does not show up when using HTTP/1.1. So if you use curl (which defaults to HTTP/1.1) to fetch the CGI output repeatedly, the memory usage will be fine. But if you pass it the --http1.0 flag, RSS starts growing. Presumably it has something to do with the lack of chunking in HTTP 1.0.

Actions

Also available in: Atom