Lighttpd

Hei hei,

so I'm using lighttpd on a small embedded platform, at91sam9g20 SoC, 32 MB of main memory, board support package based on ptxdist 2017.07.0 built with OSELAS.Toolchain-2014.12.3 for arm-v5te-linux-gnueabi, lighttpd built from source through autotools without additional patches and the following options:

./configure \
--prefix=/usr \
--sysconfdir=/etc \
--localstatedir=/var \
--libdir=/usr/lib \
--build=x86_64-host-linux-gnu \
--host=arm-v5te-linux-gnueabi \
--libdir=/usr/lib/lighttpd \
--enable-lfs \
--disable-ipv6 \
--disable-mmap \
--enable-extra-warnings \
--without-libev \
--without-mysql \
--without-ldap \
--without-attr \
--without-valgrind \
--without-libunwind \
--with-openssl \
--without-kerberos5 \
--with-pcre \
--without-zlib \
--without-bzip2 \
--without-fam \
--without-webdav-props \
--without-libxml \
--without-sqlite \
--without-webdav-locks \
--without-uuid \
--without-gdbm \
--without-memcached \
--without-lua

Compile time features from lighttpd -V:

$ lighttpd -V
lighttpd/1.4.41 (ssl) - a light and fast webserver
Build-Date: Aug 17 2017 16:17:43

Event Handlers:

        + select (generic)
        + poll (Unix)
        - rt-signals (Linux 2.4+)
        + epoll (Linux 2.6)
        - /dev/poll (Solaris)
        - eventports (Solaris)
        - kqueue (FreeBSD)
        - libev (generic)

Network handler:

        - linux-sendfile
        - freebsd-sendfile
        - darwin-sendfile
        - solaris-sendfilev
        + writev
        + write
        - mmap support

Features:

        - IPv6 support
        - zlib support
        - bzip2 support
        + crypt support
        + SSL Support
        + PCRE support
        - mySQL support
        - LDAP support
        - memcached support
        - FAM support
        - LUA support
        - xml support
        - SQLite support
        - GDBM support

I do file uploads via POST requests with mod_cgi or mod_fcgi and those files are up to 15 MB of size. Free memory is often lower than this upload file size. The upload file dir is some folder in the ubifs file system. The cgi is written in C.

After migrating to v1.4.45 I noticed oom killer killing lighttpd and/or the cgi process, in htop you see memory usage of lighttpd rising rapidly when uploading a file until memory is full and oom starts its work.

The old behavior with v1.4.39 was the memory consumption of lighttpd staying more or less constant, while some tmp files where written to the upload dir, the webserver passed all the data to the cgi, and in the cgi parsed the POST request, extracted the uploaded file and wrote it to disk in a single file.

I did some bisecting. With v1.4.40 the old behaviour still works, although I had one case where it took unusually long until the cgi had written the uploaded file to flash. With v1.4.41, v1.4.42 and v1.4.45 the memory filling up happens. It does not make any difference how I set the option 'server.stream-request-body', which was introduced with v1.4.40, right?

I'll attach both the lighttpd.conf and my fcgi.conf (other configs included are mime.conf and ssl.conf, which probably don't matter).

So what I need is more or less the old behavior with quite constant memory use, there's no room for additional memory use, not even or especially not for buffering big POST requests in memory. Is this still possible with a recent version and how? Or is this a bug introduced in v1.4.40 or v1.4.41?

I can import patches on top of a release and test with those.