Lighttpd

1. what most people considered a memory leak, was not a memory leak.

2. some numbers for how much memory lighty was using would be interesting. without some details there is only useless guessing.

I have never seen an memory issue.

Lighttpd doesn't claim to be a low memory using webserver though.

The amount of memory needed, in your case, is going depend on how many concurrent php accesses there are, times, how large those replies are.

If you have 100 users browsing the website, and they all download a 1Meg file, that is going use 100Megs, plus the 10megs or so lighttpd uses, giving lighttpd a total of 110megs of memory.

Now once lighttpd hits a max user load peek, it's memory usage will not go up, but it's using isn't suppost to stay at the starting point (10MB in your case).

(sorry about long delay, I apparently didn't set up email notifications for this thread)

Thanks. I didn't mean to suggest there had been a memory leak. It's just one of the things that (unfortunately) constantly comes up when you search the web for 'lighttpd'. I'm going to try not restarting the server and seeing whether there's a 'ceiling'.

As for the load, this server is pretty unique. It hosts a single 5 kilobyte flash file that is used in a widget I created. It's being used by a lot of websites, worldwide. This results in a very 'even' load. There's sine-like wave pattern with a wavelength of one day (peak during the day in the US, lowest at night), but it has a small amplitude. I don't have mod_accesslog running, so I don't know any exacte numbers, but it amounts to about 2.5 hits every second, and 30GB of traffic monthly.

I'm still pretty new to Linux system administration, so if you need additional numbers, please tell me how to gather them (but please keep in mind that I'd like to install as little additional software as possible on this machine).

BTW, 12MB is of course absolutely fine. It's a 128MB VPS, so there's room to spare. I just need to eventually be able to not check how it's doing every day.

hi, can you attach some more details if memory usage is high and the load not:

- libc version: dpkg -l libc6
- memory usage: ps u -C lighttpd
- memory map: pmap -x $(ps h -o pid -C lighttpd)

I've been playing with something similar. I set up a little cron job that adds the output from ps (custom format including RSS and VIRT memory usage) to a text file every 15 minutes. I'll attach the file. The first line is right before issueing a restart, from there on in it seems that memory usage is evening out (see graph.png).

I'll also attach the bandwidth graphs from the hosts control panel. Since every request is pretty much identical (the 5KB file), it should closely match the server's load over the course of a day. The pattern does however not show up in the ps text file.

@ralf: libc version: 2.11.3-4, output from pmap attached.

hey slight,

thanks for the informations.

we talk about ~1m increase in a time (how long?)?

the "anon" parts in your pmap.txt should be the memory dynamically allocated.

i think lets take a look to it some more time, your graph seams to be nice for it.

I'll leave this running and do a graph again in a week or so. This one spans roughly two days (there's a 'time running' column in the txt file). Will post results here.

I agree that a 1 MB increase is tiny, but I'm curious about the pattern. Memory usage seems to almost never decrease, and it can't keep going up.

yes lighty will not release buffer memory that was allocated until shutdown. then it is released properly. It keeps reusing the buffer memory over and over. That's why it is important to not do stupid things like streaming iso images/videos from a php script instead of using x-lighttpd-sendfile.

@darix: I've been watching my server since your last reply, and the behavior I'm seeing seems to confirm what you're saying. Memory consumption still increases occasionally, but less frequent. I assume the total memory is adjusted upwards every time there's a little peak in requests.

If I understand you correctly, this is "by design", which is reassuring. Thanks.

well, lighttpd does not keep all buffers allocated, but some of them (and also keeps other structs around); the main problem is usually "memory fragmentation" - even if lighty frees some memory, it can't be returned to the system.

its possible to give some memory back to the system using mallopt() or malloc_trim(0).
however, this is not portable.
i did some tests: http://ralf.stormbind.net/junk/test_malloc.c
the test avoid usage of printf() and so on because they may allocate buffers internally.
running that test on OSX shows me that the memory allocator works/looks different from the linux one.

usage no trim: ./test_malloc
usage trim: ./test_malloc --trim

the test do some allocation (around 160m):

A = malloc(something)
B = malloc(many small buffers)
C = malloc(something)
    free(all B's)
    malloc_trim(0) depending on command line arg "--trim" 

if i run this without malloc_trim() the memory usage is still high after "B" is free'd.
using malloc_trim(0) the RES memory usage is small but VIRT is still high.
my understanding of VIRT is that this is memory which can be swapped out and/or used by other applications.

Fragmentation can not be fixed by calling a trim function.

I promised to keep my stats running for a week and report back, so here's an updated graph, showing memory consumption over almost nine days (8 days, 22 hours, sampled every 15 minutes).

Blast from the past. Just happened across this looking for something else.

Even with the addition of HTTP/2, lighttpd 1.4.60+ uses less memory than lighttpd did 10 years ago, and also uses code patterns to reduce memory fragmentation.