Project

General

Profile

Bug #881

memory usage when ssl.engine used and large data uploaded through CGI

Added by Anonymous about 11 years ago. Updated over 1 year ago.

Status:
Fixed
Priority:
Normal
Assignee:
-
Category:
core
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Missing in 1.5.x:

Description

system specs: Red Hat Enterprise Linux 4, 2.6.9-34.0.2.ELsmp kernel, lighttpd-1.4.12 (built with ssl support), 4Gigs memory, openssl-0.9.7a-43.8

This problem does not happen when not using the ssl.engine.
Below is my lighttpd.conf file, a CGI perl script, and a client perl script to hit the CGI perl script. It simply uploads a file and write is to /tmp.

lighttpd is configured such that the files in the document root are treated as cgi (for example, creating a soap service to be executed through cgi). The certificate being used by lighttpd is a self signed certificate (which should not matter).

The contents of the test file I use (testfile.dat) that gets uploaded should not matter (I've tried different files and got the same behavior). To recreate this, for example, simply use a file around 130 megs.

When the ssl.engine engine is enabled, uploading the 130Meg test file causes lighttpd to eat over 130Megs of memory (over 3% according to top)- and it keeps it and never releases it. When another file is uploaded - larger than the first, more memory is used - and the memory not released. However, if a smaller file is then uploaded the memory usage will drop back down to the 130Megs. It remains there even when lighttpd is idle.

When the ssl.engine is NOT enabled this is not the case. It uses a little memory while the file is uploaded, but then goes back to 0% usage and the memory footprint of lighttpd returns to being normal (about what it is at startup).

I'm sure the memory usage while the file is being uploaded with ssl is due to how ssl works. But, I didn't expect lighttpd to continue to keep the memory after the upload was complete.

lighttpd.conf (678 Bytes) lighttpd.conf Anonymous, 2006-10-03 22:23
upload_file.pl (321 Bytes) upload_file.pl Anonymous, 2006-10-03 22:24
testapp (808 Bytes) testapp Anonymous, 2006-10-03 22:32

Related issues

Related to Bug #758: memory fragmentation leads to high memory usage after peaksFixed

Related to Bug #1265: SSL + file upload = lots of memoryFixed

Related to Bug #949: fastcgi, cgi, flush, php5 problem.Fixed

Associated revisions

Revision 5ab7944d (diff)
Added by gstrauss over 1 year ago

[TLS] release openssl buffers as used (fixes #1265, fixes #1283, #881)

use SSL_MODE_RELEASE_BUFFERS (OpenSSL >= 1.0.0) to free buffers
as they are used, to potentially reduce memory footprint of
idle SSL connections

x-ref:
"memory usage when ssl.engine used and large data uploaded through CGI"
https://redmine.lighttpd.net/issues/881
"SSL + file upload = lots of memory"
https://redmine.lighttpd.net/issues/1265
"Memory usage increases when proxy+ssl+large file"
https://redmine.lighttpd.net/issues/1283

History

#1 Updated by Anonymous about 11 years ago

I had to base64 encode the testapp perl script because Trac kept flagging it as spam. So, you'll need to decode it before using

#2 Updated by stbuehler about 9 years ago

  • Status changed from New to Fixed
  • Resolution set to wontfix

The memory-not-freed is a known issue. and a wontfix for 1.4.x

#3 Updated by stbuehler about 9 years ago

  • Status changed from Fixed to Wontfix

#4 Updated by gstrauss over 1 year ago

  • Related to Bug #1265: SSL + file upload = lots of memory added

#5 Updated by gstrauss over 1 year ago

  • Description updated (diff)
  • Status changed from Wontfix to Fixed
  • Target version set to 1.4.x

Large request body input is saved in temporary files instead of completely buffered into memory. This was fixed a long time ago.

#6 Updated by gstrauss over 1 year ago

Will additionally soon commit a change to use SSL_MODE_MODE_RELEASE_BUFFERS in OpenSSL >= 1.0.0 to potentially reduce memory footprint of idle SSL connections.

If memory usage with SSL is still an issue for some embedded systems, please reopen this ticket or create a new ticket. #1265 suggests disabling SSL read ahead to trade off reduced memory usage for additional CPU usage. Would be a one line patch to change network.c call to SSL_CTX_set_default_read_ahead(). If there was a demonstrable need where this is the solution, we might make enabling/disabling SSL read ahead into a configuration option. For now, lighttpd continues to enable SSL read ahead.

#7 Updated by gstrauss over 1 year ago

  • Related to Bug #949: fastcgi, cgi, flush, php5 problem. added

#8 Updated by gstrauss over 1 year ago

New: asynchronous, bidirectional streaming support for request and response
Submitted pull request: https://github.com/lighttpd/lighttpd1.4/pull/66

included in the pull request are flags to openssl (SSL_MODE_RELEASE_BUFFERS) to release memory buffers when finished with them, instead of holding onto the buffers.

#9 Updated by stbuehler over 1 year ago

  • Target version changed from 1.4.x to 1.4.40

Also available in: Atom