[Solved] https POST requests buffered in RAM since v1.4.41?
so I'm using lighttpd on a small embedded platform, at91sam9g20 SoC, 32 MB of main memory, board support package based on ptxdist 2017.07.0 built with OSELAS.Toolchain-2014.12.3 for arm-v5te-linux-gnueabi, lighttpd built from source through autotools without additional patches and the following options:
./configure \ --prefix=/usr \ --sysconfdir=/etc \ --localstatedir=/var \ --libdir=/usr/lib \ --build=x86_64-host-linux-gnu \ --host=arm-v5te-linux-gnueabi \ --libdir=/usr/lib/lighttpd \ --enable-lfs \ --disable-ipv6 \ --disable-mmap \ --enable-extra-warnings \ --without-libev \ --without-mysql \ --without-ldap \ --without-attr \ --without-valgrind \ --without-libunwind \ --with-openssl \ --without-kerberos5 \ --with-pcre \ --without-zlib \ --without-bzip2 \ --without-fam \ --without-webdav-props \ --without-libxml \ --without-sqlite \ --without-webdav-locks \ --without-uuid \ --without-gdbm \ --without-memcached \ --without-lua
Compile time features from
$ lighttpd -V lighttpd/1.4.41 (ssl) - a light and fast webserver Build-Date: Aug 17 2017 16:17:43 Event Handlers: + select (generic) + poll (Unix) - rt-signals (Linux 2.4+) + epoll (Linux 2.6) - /dev/poll (Solaris) - eventports (Solaris) - kqueue (FreeBSD) - libev (generic) Network handler: - linux-sendfile - freebsd-sendfile - darwin-sendfile - solaris-sendfilev + writev + write - mmap support Features: - IPv6 support - zlib support - bzip2 support + crypt support + SSL Support + PCRE support - mySQL support - LDAP support - memcached support - FAM support - LUA support - xml support - SQLite support - GDBM support
I do file uploads via POST requests with mod_cgi or mod_fcgi and those files are up to 15 MB of size. Free memory is often lower than this upload file size. The upload file dir is some folder in the ubifs file system. The cgi is written in C.
After migrating to v1.4.45 I noticed oom killer killing lighttpd and/or the cgi process, in htop you see memory usage of lighttpd rising rapidly when uploading a file until memory is full and oom starts its work.
The old behavior with v1.4.39 was the memory consumption of lighttpd staying more or less constant, while some tmp files where written to the upload dir, the webserver passed all the data to the cgi, and in the cgi parsed the POST request, extracted the uploaded file and wrote it to disk in a single file.
I did some bisecting. With v1.4.40 the old behaviour still works, although I had one case where it took unusually long until the cgi had written the uploaded file to flash. With v1.4.41, v1.4.42 and v1.4.45 the memory filling up happens. It does not make any difference how I set the option 'server.stream-request-body', which was introduced with v1.4.40, right?
I'll attach both the lighttpd.conf and my fcgi.conf (other configs included are mime.conf and ssl.conf, which probably don't matter).
So what I need is more or less the old behavior with quite constant memory use, there's no room for additional memory use, not even or especially not for buffering big POST requests in memory. Is this still possible with a recent version and how? Or is this a bug introduced in v1.4.40 or v1.4.41?
I can import patches on top of a release and test with those.
So I did some more bisecting on the git sources this morning. The failing behavior (memory fills up) was introduced with a95aaa9de984dc004dd1d4302147e8a0c23efb10 between 1.4.40 and 1.4.41 and was an openssl related change. (Did I mention I use https for this?)
Reverting just this changeset in 1.4.41 restores the old behavior. The changeset can be cleanly reverted in 1.4.42, 1.4.43 and 1.4.44 (but not 1.4.45), but then the browser tells me the ssl connection failed.
So because it's not related to server.stream-request-body config option, I would say it's a bug. Should I open a ticket for that?
I got the essential hint in IRC now to have a look at https://redmine.lighttpd.net/projects/1/wiki/docs_ssl especially ssl.read-ahead
With lighttpd 1.4.45 and
ssl.read-ahead = "disable" the problem does not occur anymore. :-)