Bug #2948


[regression][Bisected] lighttpd uses way more memory with POST since 1.4.52

Added by rgenoud almost 2 years ago. Updated almost 2 years ago.

Target version:


Since commit 88ee73d0a216 ("[multiple] perf: simplify chunkqueue_get_memory()")
uploading a big file via POST uses a lot more memory than in v1.4.51

It's actually the next part of #2922.
At the time, I didn't pay attention to the memory used by the lighttpd process.

I did my tests onto a1077d18cb36 ("[tests] more test config cleanup")
and then track down the memory problem on commit 88ee73d0a216, even with patches:
fe3dc1796894 ("[mod_fastcgi] fix NULL ptr deref from bugfix #2922 (fixes #2923)")
a1b527e47374 ("[multiple] reduce initial buffer sz if large POST (fixes #2922)")

I've used the same config as in #2922 with: = 2 = 2
server.chunkqueue-chunk-sz = 1024

To highlight the high memory usage, here's simple index.php:
$errors= array();
$file_name = $_FILES['image']['name'];
$file_size =$_FILES['image']['size'];
$file_tmp =$_FILES['image']['tmp_name'];
echo "Success";
<form action="" method="POST" enctype="multipart/form-data">
<input type="file" name="image" />
<input type="submit"/>

and, on the server side :
top -b -d 1 | grep lighttpd

Upload a 50MB file, and watch the memory used by lighttpd grow.

at version 1.4.51, the memory used by lighttpd was between 4012K and 4528K during the upload.

since commit fccc7fc607ff ("[core] perf: chunk.c chunk pool") it's between 4012K and 7856K

and since commit 88ee73d0a216 ("[multiple] perf: simplify chunkqueue_get_memory()"), with its 2 fixes, it's between 4012K and 51156K (it's actually the whole file size)

It seems that the whole file is mapped in lighttpd memory.

Actions #1

Updated by gstrauss almost 2 years ago

Would you please test with commit c83fff1d (on lighttpd git master branch) which is already slated to be part of lighttpd 1.4.54?

(As an aside, please do not set the target version when filing a bug report. You do not have enough information to accurately set that)

Actions #2

Updated by rgenoud almost 2 years ago

Tested with commit c83fff1d, it's the same, lighttpd uses 50MB in memory

(sorry for setting the target version, I don't know why I clicked on that! :) )

Actions #3

Updated by gstrauss almost 2 years ago

  • Target version changed from 1.4.54 to 1.4.x

Sorry rgenoud, this is not easy to reproduce on a typical system. Please summarize your setup and try to provide more information (e.g. an strace) and the specs and speed of the embedded (or tiny) system on which your are running. 32-bit or 64-bit? architecture? openssl? CGI or FastCGI?

I was unable to reproduce this with and without mod_openssl, with CGI or with FastCGI on a 3 year old x86_64 laptop (64-bit lighttpd)

I created a 50 MB file for upload, a simple target PHP for the POST request, and made requests with
curl -k -d /dev/shm/50M-file https://localhost:8080/index.php
. = 2 = 2
server.chunkqueue-chunk-sz = 1024

Please verify that you have not set = "enable". The default is disabled, and should be disabled on slow systems.

Actions #4

Updated by gstrauss almost 2 years ago

Able to reproduce. Now working on a fix.

Actions #5

Updated by gstrauss almost 2 years ago

  • Status changed from New to Patch Pending
  • Target version changed from 1.4.x to 1.4.54

In reusing buffers, the code looked at the first entry on an (unsorted) linked list and if the chunk at the head of the linked list of reusable chunks was not large enough, a new chunk would be allocated. With small response headers and small chunks being sent to backend combined with large request body which filled kernel socket buffers could lead to new chunks being allocated and added to the reusable linked list, but ending up behind a small chunk at the head of the linked list.

I have commited a fix on my development branch.

Actions #6

Updated by rgenoud almost 2 years ago

Tested with commit commit:0e62fb893 , the memory usage is back to normal.

Thanks a lot !

Actions #7

Updated by gstrauss almost 2 years ago

  • Status changed from Patch Pending to Fixed
  • % Done changed from 0 to 100

Also available in: Atom