Project

General

Profile

Actions

Bug #1642

closed

Lighttpd 1.4.19 Memory Leak

Added by Anonymous almost 16 years ago. Updated almost 16 years ago.

Status:
Fixed
Priority:
High
Category:
core
Target version:
-
ASK QUESTIONS IN Forums:

Description

I was attempting to upgrade lighttpd from 1.4.18 to 1.4.19 (in order to pull in the security fixes) for our site, but it never made it through testing. Specifically, when I use apachebench to throw a lot of traffic at it, lighttpd 1.4.19 grows in an unbounded way. It appears that there is a memory leak in the server. After approximately one million hits, the lighttpd process had grown to be more than 600MB in size.

I have created two identical server images -- one with lighttpd 1.4.19 and one with lighttpd 1.4.18 plus the DOS patch lighttpd-1.4.x_high_load_dos.patch. When I send identical traffic to each of them, I see the following results.


lighttpd 1.4.18+dos_patch memory usage (DRS/RSS) before traffic: 4276 / 2140
lighttpd 1.4.18+dos_patch memory usage (DRS/RSS) after traffic: 
lighttpd 1.4.19 memory usage (DRS/RSS) before traffic: 4267 / 2132
lighttpd 1.4.19 memory usage (DRS/RSS) after traffic: 227863 / 226004

The URLs requested were all very small files (most smaller than 1,024 bytes; all smaller than 4,096 bytes) that lighttpd is supposed to send using x-sendfile. Basically the request comes in and is sent through to an FCGI backend which responds with an x-sendfile location that lighttpd then sends.

I will attempt to respond to this ticket with more valgrind information as I garner it.

-- remi

Actions #1

Updated by Anonymous almost 16 years ago

Oops. Typo.


lighttpd 1.4.18+dos_patch memory usage (DRS/RSS) after traffic: 

Should be:


lighttpd 1.4.18+dos_patch memory usage (DRS/RSS) after traffic: 2668 / 2860
Actions #2

Updated by Anonymous almost 16 years ago

Alright, I ran lighttpd within valgrind (needed to disable "server.max-fds" in order to make it work), and got the attached file from sending 13,000 requests at the server.

-- remi

Actions #3

Updated by Anonymous almost 16 years ago

Annoyingly, Trac refuses the attachment as spam. I will e-mail it to .

Actions #4

Updated by Anonymous almost 16 years ago

Hullo

I am hit by this as well. Right now I have a script that shoots lighty when it reaches the 50MB mark.

Can I help in any way?

-- oscar

Actions #5

Updated by Anonymous almost 16 years ago

We are also experiencing this. I just upgraded our server to 1.4.19 using the FreeBSD ports system a week ago. The main lighttpd process appears to add 1M of resident use every few minutes.

The server uses php 5.2.6 fastcgi to serve a lot of X-sendfile files and dynamically generated pages. Static images and cs/jss are a small part of the load. 30-50 reqs/second baseline load.

FreeBSD 6-2-RELEASE-p11.

Has there been any motion on this?

Actions #6

Updated by Anonymous almost 16 years ago

Here's the outline of a test scenario I just created that seems to point to sendfile handling as the problem:

stop lighty
start lighty
lighty RES size ~3M
100k reqs for php script that generates 60k text output
RES size 4.8M
another 100k reqs
RES size 4.852k (unchanged)

stop lighty
start lighty
RES size ~ 3M
100k reqs for static jpg
RES size ~ 3.5M
another 100k reqs
RES size 3.508M (unchanged)

stop lighty
start lighty
RES size ~ 3M
100 k reqs for same static jpg sent using x-sendfile from php
RES size 35M
another 100k reqs
RES size 67M

I can upload the actual scripts but you see the problem: getting a jpg 100k times is fine, getting that SAME jpg 100k times with sendfile results in massive growth of the resident size of the process.

The FreeBSD port has a patch to fix a lighty sendfile bug when a connection dies. I compiled a version without this patch, and the memory leak behavior is still there.

That leads me to believe it's not port specific, but it still might be FreeBSD specific since sendfile is handled from within network_freebsd_sendfile.c.

I hope this jogs some developer's mind or sheds a little light... something about memory cleanup after sendfile changed with the version change from 1.4.18 to 1.4.19.

Actions #7

Updated by Anonymous almost 16 years ago

I realized my earlier post might have been slightly incorrect, since even the static output is done with sendfile, the issue is really the handler for the X-LIGHTTPD-sendfile header set from within php.

With that in mind, there is an extremely suspicious diff between the 1.4.18 and 1.4.19 versions of mod_fastcgi.c

On line 2532 of the 1.4.18 version, it checks for the header, does 3 lines of work and the test block ends.

In the 1.4.19 version, the test block starts on line 2541. It inits a variable with this line within the test block scope:
data_string *dcls = data_string_init();
There are some new buffer copies (to set the Content-length header it looks like)

Most suspiciously, the dcls variable is used as an lval once again and then freed:

dcls = (data_string*) array_replace(con->response.headers, (data_unset )dcls);
if (dcls) dcls->free((data_unset
)dcls);

(line 2555)

I am not a lighty developer but is there any chance this block has the leak in it?

Actions #8

Updated by stbuehler almost 16 years ago

  • Status changed from New to Fixed
  • Resolution set to fixed

Fixed in r2143

Actions

Also available in: Atom