memory usage when ssl.engine used and large data uploaded through CGI
system specs: Red Hat Enterprise Linux 4, 2.6.9-34.0.2.ELsmp kernel, lighttpd-1.4.12 (built with ssl support), 4Gigs memory, openssl-0.9.7a-43.8
This problem does not happen when not using the ssl.engine.
Below is my lighttpd.conf file, a CGI perl script, and a client perl script to hit the CGI perl script. It simply uploads a file and write is to /tmp.
lighttpd is configured such that the files in the document root are treated as cgi (for example, creating a soap service to be executed through cgi). The certificate being used by lighttpd is a self signed certificate (which should not matter).
The contents of the test file I use (testfile.dat) that gets uploaded should not matter (I've tried different files and got the same behavior). To recreate this, for example, simply use a file around 130 megs.
When the ssl.engine engine is enabled, uploading the 130Meg test file causes lighttpd to eat over 130Megs of memory (over 3% according to top)- and it keeps it and never releases it. When another file is uploaded - larger than the first, more memory is used - and the memory not released. However, if a smaller file is then uploaded the memory usage will drop back down to the 130Megs. It remains there even when lighttpd is idle.
When the ssl.engine is NOT enabled this is not the case. It uses a little memory while the file is uploaded, but then goes back to 0% usage and the memory footprint of lighttpd returns to being normal (about what it is at startup).
I'm sure the memory usage while the file is being uploaded with ssl is due to how ssl works. But, I didn't expect lighttpd to continue to keep the memory after the upload was complete.
use SSL_MODE_RELEASE_BUFFERS (OpenSSL >= 1.0.0) to free buffers
as they are used, to potentially reduce memory footprint of
idle SSL connections
"memory usage when ssl.engine used and large data uploaded through CGI"
"SSL + file upload = lots of memory"
"Memory usage increases when proxy+ssl+large file"
#6 Updated by gstrauss about 1 year ago
Will additionally soon commit a change to use SSL_MODE_MODE_RELEASE_BUFFERS in OpenSSL >= 1.0.0 to potentially reduce memory footprint of idle SSL connections.
If memory usage with SSL is still an issue for some embedded systems, please reopen this ticket or create a new ticket. #1265 suggests disabling SSL read ahead to trade off reduced memory usage for additional CPU usage. Would be a one line patch to change network.c call to SSL_CTX_set_default_read_ahead(). If there was a demonstrable need where this is the solution, we might make enabling/disabling SSL read ahead into a configuration option. For now, lighttpd continues to enable SSL read ahead.
#8 Updated by gstrauss about 1 year ago
New: asynchronous, bidirectional streaming support for request and response
Submitted pull request: https://github.com/lighttpd/lighttpd1.4/pull/66
included in the pull request are flags to openssl (SSL_MODE_RELEASE_BUFFERS) to release memory buffers when finished with them, instead of holding onto the buffers.
Also available in: Atom