Bug #3161: gzip dynamic compression broken with source size bigger than 128k - Lighttpd - lighty labs

Actions

Copy link

Bug #3161

closed

gzip dynamic compression broken with source size bigger than 128k

Added by flynn over 2 years ago. Updated over 2 years ago.

Status:

Fixed

Priority:

Normal

Category:

mod_deflate

Target version:

1.4.66

ASK QUESTIONS IN Forums:

Description

Dynamic files compressed with gzip bigger than 128kB are broken, consist only of binary zeroes.

Brotli compressed files of the same source file are ok. Small files are also without errors.

To exclude local build problems or the current debian package I built version 1.4.65 from source and used the following minimal configuration:

server.document-root = "/var/www/" 

server.port = 30000

mimetype.assign = ( ".js" => "text/javascript" )

server.modules += ( "mod_deflate" )

deflate.cache-dir = "/usr/local/lighttpd/cache/" 
deflate.mimetypes = ( "text/javascript" )
deflate.allowed-encodings = ( "brotli", "gzip", "deflate" ) # "bzip2" also supported

A request with wget -S --compression=gzip http://1.2.3.4:3000/... to file bigger than 128k results in read errors:

  HTTP/1.1 200 OK
  Content-Type: text/javascript
  ETag: "532812655-gzip" 
  Last-Modified: Wed, 14 Oct 2020 19:29:04 GMT
  Content-Length: 68095
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Date: Mon, 18 Jul 2022 10:03:35 GMT
  Server: lighttpd/1.4.65
Length: 68095 (66K) [text/javascript]
Saving to: 'moment-with-locales.min.js'

moment-with-locales.min.js         0%[                                                          ]       0  --.-KB/s    in 0s      

2022-07-18 12:03:35 (223 MB/s) - Read error at byte 0 (Invalid argument).Retrying.

Source file size here: 260313 bytes. The corresponding compressed cache file on disk has only binary zeroes as content.

Actions

Copy link

Updated by gstrauss over 2 years ago

Able to reproduce. This appears to be an issue with how lighttpd is using mmap and libdeflate. If you build lighttpd ./configure --without-libdeflate, the problem does not occur. As you noted, the problem only occurs with file sizes > 128k. (I may have to come back to this later, so next update may be delayed until tomorrow.)

Actions

Copy link

Updated by gstrauss over 2 years ago

Downloading using curl instead of wget, I get the same length of data returned from the call to the libdeflate library function libdeflate_gzip_compress() but the contents it is not valid gzip. ...More later.

Actions

Copy link

Updated by gstrauss over 2 years ago

What version of libdeflate is on your system?

Actions

Copy link

Updated by flynn over 2 years ago

The version currently in use of libdeflate is 1.12 from debian testing.

I can confirm that without libdeflate using ./configure --without-libdeflate the problem does not occur.

Actions

Copy link

Updated by gstrauss over 2 years ago

Status changed from New to Patch Pending

I certainly tested this code path when I wrote the code in January, though there is perhaps a chance I changed MAP_SHARED to MAP_PRIVATE for consistency. (On some older filesystems, a read-only mmap MAP_SHARED fails while a read-only mmap MAP_PRIVATE succeeds.) Or perhaps a kernel upgrade changed behavior? I am not sure. In any case, here is the fix so that the mmap'ed pages are reflected on disk:

--- a/src/mod_deflate.c
+++ b/src/mod_deflate.c
@@ -1596,7 +1596,7 @@ static int mod_deflate_using_libdeflate (handler_ctx * const hctx, const plugin_
     }

     /*void *addr = mmap(NULL, sz, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);*/
-    void * const addr = mmap(NULL, sz, PROT_WRITE, MAP_PRIVATE, fd, 0);
+    void * const addr = mmap(NULL, sz, PROT_WRITE, MAP_SHARED, fd, 0);
     if (MAP_FAILED == addr) {
         log_perror(hctx->r->conf.errh, __FILE__, __LINE__, "mmap");
         return mod_deflate_using_libdeflate_err(hctx, fn, fd);

Actions

Copy link