Project

General

Profile

[Solved] high latency on 1.4.51 + proxy + deflate

Added by GilGalaad over 5 years ago

Hi everybody, and thank you in advance if you will try to help me.

Configuration rapid description: 2 virtual hosts, one for private network traffic, one for a public application. Private domain serves static files on the server, public domain proxies traffic to a Java application running on the same machine (lighttpd is doing SSL termination for that). All http connections are redirected on https, except ones for certbot/letsencrypt validation. Dynamic compression enabled for all traffic via mod_deflate.

OS: FreeBSD 11.2-RELEASE-p4
Lighttpd version: 1.4.51, binary package from FreeBSD pkg repository
Config file: https://pastebin.com/rVWQjVLE
Browser used: Firefox 63.0.1 on Windows 7 64 bit

Scenario description: when I connect via public domain to the java application, I experience a latency of about 6 seconds on most of the url served.
The debug console shows that the loading time seems to be ALWAYS 6 seconds plus some millisecond (which is the timing I expect for the page to be served). The content is downloaded and displayed properly in the first milliseconds, but then the connection "hangs" for 6 seconds, before the client reports the page as loaded. Always 6 seconds, even for the static content of the application. There is absolutely no load on the machine (I am the only user at the moment).

The issue does not appear while serving static content from the filesystem (in the private domain).
The issue appeared when i upgraded from 1.4.50 to 1.4.51.
The issue disappears downgrading to 1.4.50, or disabling the deflate module in 1.4.51 (just commenting out the deflate.mimetypes directive), so seems to me related to this specific version of lighttpd and this module.

Any support is appreciated, thank you.
Francesco.


Replies (5)

RE: high latency on 1.4.51 + proxy + deflate - Added by gstrauss over 5 years ago

An strace of the lighttpd process when this happens would be useful.

Have you tried checking to see if this is related to browser behavior, or, more precisely, javascript in the browser misbehaving? Please try using curl instead of the browser. Do you see the same behavior?

RE: high latency on 1.4.51 + proxy + deflate - Added by GilGalaad over 5 years ago

Hi gstrauss, thanks for your answer.

I am confident this is not browser related, this happens in multiple browser tested, and always disappears disabling deflate.
I was able to reproduce it with curl too.
This is the command output with timings: https://pastebin.com/5CTQ6M2y

The line "* no chunk, no close, no size. Assume close to signal end" seems related, maybe.
It's like lighttpd used 6 second to gzip a 991 bytes file, before start sending html payload, which is a bit strange (of course there is absolutely no load on the machine, that is a quad core Intel with 32gb ram). I'm just guessing by the way.

Just to provide more information, even if not very low level, here is a firefox console screenshot showing the process dowloading some (compressed) svg files, stuck for 6 seconds before continung on. I don't think there is any javascript here.

I've never used strace in my whole life, so I am a bit embarassed, but i'll try if there is no other way.
Any idea so far?

Thank you so much :)

RE: high latency on 1.4.51 + proxy + deflate - Added by gstrauss over 5 years ago

ick. That's a bug. Thanks for reporting it. This should fix it:

--- a/src/http_header.c
+++ b/src/http_header.c
@@ -69,7 +69,7 @@ void http_header_response_set(connection *con, enum http_header_e id, const char
      * (note: if 0 == vlen, header is still inserted with blank value,
      *  which is used to indicate a "removed" header)
      */
-    con->response.htags |= id;
+    (vlen) ? (con->response.htags |= id) : (con->response.htags &= ~id);
     array_set_key_value(con->response.headers, k, klen, v, vlen);
 }

@@ -121,7 +121,7 @@ void http_header_request_set(connection *con, enum http_header_e id, const char
      * (note: if 0 == vlen, header is still inserted with blank value,
      *  which is used to indicate a "removed" header)
      */
-    con->request.htags |= id;
+    (vlen) ? (con->request.htags |= id) : (con->request.htags &= ~id);
     array_set_key_value(con->request.headers, k, klen, v, vlen);
 }

A more extensive modification is needed for a more complete generic fix

RE: high latency on 1.4.51 + proxy + deflate - Added by GilGalaad over 5 years ago

Glad to be helpful, and thank you so much for your work! :)

    (1-5/5)