Project

General

Profile

Slight memory consumption increase over time

Added by roytanck about 10 years ago

Recently, I set up a VPS to serve one single (static) file. It runs a dressed-down version of Debian (6.0, kernel 2.6.32-238.19.1.el5.028stab092.2). I installed Lighttpd, with modules 'expire' and 'simple_vhost', nothing else.

Services running are init, syslog-ng, cron, xinetd, lighttpd, sh, stats.sh, top, sshd. Over the last week, I noticed that the total amount of RAM used by the VPS gradually increased from ~10MB to ~12MB. Every time I checked, the number was slightly higher. Considering how there's almost nothing running, I assume this is caused by Lighty.

Restarting lighttpd returns the memory usage to 10MB, but it then starts to slowly climb again. The number of requests varies over the course of a day, but the pattern is identical for each day.

I'm new to lighttpd, and I of course read about memory leak issues in the past. Is it normal for lighttpd's memory footprint to increase over time? Should I be worried?


Replies (15)

RE: Slight memory consumption increase over time - Added by darix about 10 years ago

1. what most people considered a memory leak, was not a memory leak.

2. some numbers for how much memory lighty was using would be interesting. without some details there is only useless guessing.

RE: Slight memory consumption increase over time - Added by patrickdk about 10 years ago

I have never seen an memory issue.

Lighttpd doesn't claim to be a low memory using webserver though.

The amount of memory needed, in your case, is going depend on how many concurrent php accesses there are, times, how large those replies are.

If you have 100 users browsing the website, and they all download a 1Meg file, that is going use 100Megs, plus the 10megs or so lighttpd uses, giving lighttpd a total of 110megs of memory.

Now once lighttpd hits a max user load peek, it's memory usage will not go up, but it's using isn't suppost to stay at the starting point (10MB in your case).

RE: Slight memory consumption increase over time - Added by roytanck about 10 years ago

(sorry about long delay, I apparently didn't set up email notifications for this thread)

Thanks. I didn't mean to suggest there had been a memory leak. It's just one of the things that (unfortunately) constantly comes up when you search the web for 'lighttpd'. I'm going to try not restarting the server and seeing whether there's a 'ceiling'.

As for the load, this server is pretty unique. It hosts a single 5 kilobyte flash file that is used in a widget I created. It's being used by a lot of websites, worldwide. This results in a very 'even' load. There's sine-like wave pattern with a wavelength of one day (peak during the day in the US, lowest at night), but it has a small amplitude. I don't have mod_accesslog running, so I don't know any exacte numbers, but it amounts to about 2.5 hits every second, and 30GB of traffic monthly.

I'm still pretty new to Linux system administration, so if you need additional numbers, please tell me how to gather them (but please keep in mind that I'd like to install as little additional software as possible on this machine).

BTW, 12MB is of course absolutely fine. It's a 128MB VPS, so there's room to spare. I just need to eventually be able to not check how it's doing every day.

RE: Slight memory consumption increase over time - Added by ralf about 10 years ago

hi, can you attach some more details if memory usage is high and the load not:

- libc version: dpkg -l libc6
- memory usage: ps u -C lighttpd
- memory map: pmap -x $(ps h -o pid -C lighttpd)

RE: Slight memory consumption increase over time - Added by roytanck about 10 years ago

I've been playing with something similar. I set up a little cron job that adds the output from ps (custom format including RSS and VIRT memory usage) to a text file every 15 minutes. I'll attach the file. The first line is right before issueing a restart, from there on in it seems that memory usage is evening out (see graph.png).

I'll also attach the bandwidth graphs from the hosts control panel. Since every request is pretty much identical (the 5KB file), it should closely match the server's load over the course of a day. The pattern does however not show up in the ps text file.

RE: Slight memory consumption increase over time - Added by roytanck about 10 years ago

@ralf: libc version: 2.11.3-4, output from pmap attached.

pmap.txt (4.24 KB) pmap.txt

RE: Slight memory consumption increase over time - Added by ralf about 10 years ago

hey slight,

thanks for the informations.

we talk about ~1m increase in a time (how long?)?

the "anon" parts in your pmap.txt should be the memory dynamically allocated.

i think lets take a look to it some more time, your graph seams to be nice for it.

RE: Slight memory consumption increase over time - Added by roytanck about 10 years ago

I'll leave this running and do a graph again in a week or so. This one spans roughly two days (there's a 'time running' column in the txt file). Will post results here.

I agree that a 1 MB increase is tiny, but I'm curious about the pattern. Memory usage seems to almost never decrease, and it can't keep going up.

RE: Slight memory consumption increase over time - Added by darix about 10 years ago

yes lighty will not release buffer memory that was allocated until shutdown. then it is released properly. It keeps reusing the buffer memory over and over. That's why it is important to not do stupid things like streaming iso images/videos from a php script instead of using x-lighttpd-sendfile.

RE: Slight memory consumption increase over time - Added by roytanck about 10 years ago

@darix: I've been watching my server since your last reply, and the behavior I'm seeing seems to confirm what you're saying. Memory consumption still increases occasionally, but less frequent. I assume the total memory is adjusted upwards every time there's a little peak in requests.

If I understand you correctly, this is "by design", which is reassuring. Thanks.

RE: Slight memory consumption increase over time - Added by stbuehler about 10 years ago

well, lighttpd does not keep all buffers allocated, but some of them (and also keeps other structs around); the main problem is usually "memory fragmentation" - even if lighty frees some memory, it can't be returned to the system.

RE: Slight memory consumption increase over time - Added by ralf about 10 years ago

its possible to give some memory back to the system using mallopt() or malloc_trim(0).
however, this is not portable.
i did some tests: http://ralf.stormbind.net/junk/test_malloc.c
the test avoid usage of printf() and so on because they may allocate buffers internally.
running that test on OSX shows me that the memory allocator works/looks different from the linux one.

usage no trim: ./test_malloc
usage trim: ./test_malloc --trim

the test do some allocation (around 160m):

A = malloc(something)
B = malloc(many small buffers)
C = malloc(something)
    free(all B's)
    malloc_trim(0) depending on command line arg "--trim" 

if i run this without malloc_trim() the memory usage is still high after "B" is free'd.
using malloc_trim(0) the RES memory usage is small but VIRT is still high.
my understanding of VIRT is that this is memory which can be swapped out and/or used by other applications.

RE: Slight memory consumption increase over time - Added by stbuehler about 10 years ago

Fragmentation can not be fixed by calling a trim function.

RE: Slight memory consumption increase over time - Added by roytanck about 10 years ago

I promised to keep my stats running for a week and report back, so here's an updated graph, showing memory consumption over almost nine days (8 days, 22 hours, sampled every 15 minutes).

RE: Slight memory consumption increase over time - Added by gstrauss about 1 month ago

Blast from the past. Just happened across this looking for something else.

Even with the addition of HTTP/2, lighttpd 1.4.60+ uses less memory than lighttpd did 10 years ago, and also uses code patterns to reduce memory fragmentation.

    (1-15/15)