Bug #3161


gzip dynamic compression broken with source size bigger than 128k

Added by flynn 9 months ago. Updated 9 months ago.

Target version:


Dynamic files compressed with gzip bigger than 128kB are broken, consist only of binary zeroes.

Brotli compressed files of the same source file are ok. Small files are also without errors.

To exclude local build problems or the current debian package I built version 1.4.65 from source and used the following minimal configuration:

server.document-root = "/var/www/" 

server.port = 30000

mimetype.assign = ( ".js" => "text/javascript" )

server.modules += ( "mod_deflate" )

deflate.cache-dir = "/usr/local/lighttpd/cache/" 
deflate.mimetypes = ( "text/javascript" )
deflate.allowed-encodings = ( "brotli", "gzip", "deflate" ) # "bzip2" also supported

A request with wget -S --compression=gzip to file bigger than 128k results in read errors:

  HTTP/1.1 200 OK
  Content-Type: text/javascript
  ETag: "532812655-gzip" 
  Last-Modified: Wed, 14 Oct 2020 19:29:04 GMT
  Content-Length: 68095
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Date: Mon, 18 Jul 2022 10:03:35 GMT
  Server: lighttpd/1.4.65
Length: 68095 (66K) [text/javascript]
Saving to: 'moment-with-locales.min.js'

moment-with-locales.min.js         0%[                                                          ]       0  --.-KB/s    in 0s      

2022-07-18 12:03:35 (223 MB/s) - Read error at byte 0 (Invalid argument).Retrying.

Source file size here: 260313 bytes. The corresponding compressed cache file on disk has only binary zeroes as content.

Actions #1

Updated by gstrauss 9 months ago

Able to reproduce. This appears to be an issue with how lighttpd is using mmap and libdeflate. If you build lighttpd ./configure --without-libdeflate, the problem does not occur. As you noted, the problem only occurs with file sizes > 128k. (I may have to come back to this later, so next update may be delayed until tomorrow.)

Actions #2

Updated by gstrauss 9 months ago

Downloading using curl instead of wget, I get the same length of data returned from the call to the libdeflate library function libdeflate_gzip_compress() but the contents it is not valid gzip. ...More later.

Actions #3

Updated by gstrauss 9 months ago

What version of libdeflate is on your system?

Actions #4

Updated by flynn 9 months ago

The version currently in use of libdeflate is 1.12 from debian testing.

I can confirm that without libdeflate using ./configure --without-libdeflate the problem does not occur.

Actions #5

Updated by gstrauss 9 months ago

  • Status changed from New to Patch Pending

I certainly tested this code path when I wrote the code in January, though there is perhaps a chance I changed MAP_SHARED to MAP_PRIVATE for consistency. (On some older filesystems, a read-only mmap MAP_SHARED fails while a read-only mmap MAP_PRIVATE succeeds.) Or perhaps a kernel upgrade changed behavior? I am not sure. In any case, here is the fix so that the mmap'ed pages are reflected on disk:

--- a/src/mod_deflate.c
+++ b/src/mod_deflate.c
@@ -1596,7 +1596,7 @@ static int mod_deflate_using_libdeflate (handler_ctx * const hctx, const plugin_

     /*void *addr = mmap(NULL, sz, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);*/
-    void * const addr = mmap(NULL, sz, PROT_WRITE, MAP_PRIVATE, fd, 0);
+    void * const addr = mmap(NULL, sz, PROT_WRITE, MAP_SHARED, fd, 0);
     if (MAP_FAILED == addr) {
         log_perror(hctx->r->conf.errh, __FILE__, __LINE__, "mmap");
         return mod_deflate_using_libdeflate_err(hctx, fn, fd);

Actions #6

Updated by flynn 9 months ago

Tested and works for me, thank you for the quick fix.

Actions #7

Updated by gstrauss 9 months ago

The Linux man page for mmap says that MAP_PRIVATE is not carried through to underlying file, so it appears that I messed that up.

Actions #8

Updated by gstrauss 9 months ago

  • Status changed from Patch Pending to Fixed

Also available in: Atom