Feature #3006: Lighty returns HTTP 411 Length Required with proxy and streaming requests/reponses body - Lighttpd - lighty labs

Actions

Copy link

Feature #3006

closed

Lighty returns HTTP 411 Length Required with proxy and streaming requests/reponses body

Added by aureq almost 5 years ago. Updated over 4 years ago.

Status:

Fixed

Priority:

Normal

Category:

mod_proxy

Target version:

1.4.56

ASK QUESTIONS IN Forums:

Description

Hi lighty labs team,

I use lighty on a Debian x86_64 as a reverse proxy. Lighty acts as a TLS server and as a HTTP request router to my Docker containers (separate host).

One of the traffic passing through is related to my Docker container registry. I push container images to the registry.
In a default configuration as highlighted below, Docker is able to push content to the registry successfully.

However, enabling the lines below appears to be breaking Docker and pushing images fails every single time.

server.stream-request-body = 2
server.stream-response-body = 2

The error message returned is:

error parsing HTTP 411 response body: invalid character '<' looking for beginning of value: "<?xml version=\"1.0\" encoding=\"iso-8859-1\"?>\n<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\"\n         \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">\n<html xmlns=\"http://www.w3.org/1999/xhtml\" xml:lang=\"en\" lang=\"en\">\n <head>\n  <title>411 Length Required</title>\n </head>\n <body>\n  <h1>411 Length Required</h1>\n </body>\n</html>\n"

Sniffing the traffic on the Docker host, I've been able to confirm that:

The Docker registry doesn't return any HTTP 411 error code. I'm assuming this is Lighty because the docker registry host name points at Lighty and there's nothing in between.
The Docker registry doesn't receive the 'PATCH' (HTTP verb) used in uploading blobs to the container registry.

Disabling the configuration directives server.stream-(request|response)-body fix the issue.

Note 1: I have a need for server.stream-(request|response)-body as the Lighty box is a very weak server with slow local storage. These 2 options allow me to efficiently stream data in and out without the need of creating temporary files on the Lighty host.
Note 2: I have a VM that enables me to recompile Debian packages, so I'm happy to test any patches and report back any results.

Here is a small diagram to explain the problem from a different angle.

 +---------------------------------------+
 |  Docker Host                          |
 |     ^    +                            |
 |     |    |   +----------+   +-------+ |
 |     |    |   | Container|   | Other | |
 |     |    |   | Registry |   | Cont. | |
 |     |    |   +--^---+---+   +-------+ |
 +-----|----|------|---|-----------------+
       |    |      |   |
       |    |      |   |
       |    |      |   |
       |    |      |   |
      2|   3|6    4|  5|
       |    |      |   |
  +----+----v----------v----+            +-------------------------+
  |                         |  7         |                         |
  |                         +------------>                         |
  |                         |            |                         |
  |   Lighty server         <------------+    Docker Client        |
  |                         |  1         |                         |
  |                         |            |                         |
  |                         |            |                         |
  +-------------------------+            +-------------------------+

1 - Docker client sends 'push' command to Docker Host (HTTPS)
2 - Lighty relays command to Docker Host (HTTP)
3 - Docker Host initiates 'push' to registry (HTTPS to docker.local.example.net)
4 - Lighty sends HTTP traffic to registry (HTTP to 192.168.2.201:5000/v2/)
5 - Registry acknowledges commands until completion
6 - Docker Host acknowledges completion of 'push'
7 - Lighty relays to Docker client completion of 'push'

My observation make me think the issue happens after step 3 and before step 4 since the 'PATCH' (blob upload) is never received by the registry.

As requested, I'm including my lighty configuration. Maybe I've misconfigured something in there that's causing the problem.

I hope this is enough details included here.

Technical information
OS: Debian 10 (up to date as of 2020/02/14)
Lighttpd version 1.4.53-4 (as provided by Debian)
HTTP client: Docker version 18.09.7, build 18.09.7

Configuration:
/etc/lighttpd/lighttpd.conf

# /etc/lighttpd/lighttpd.conf
server.modules = (
        "mod_indexfile",
        "mod_access",
        "mod_alias",
        "mod_redirect",
)

server.document-root        = "/var/www/html" 
server.upload-dirs          = ( "/var/cache/lighttpd/uploads" )
server.errorlog             = "/var/log/lighttpd/error.log" 
server.pid-file             = "/var/run/lighttpd.pid" 
server.username             = "www-data" 
server.groupname            = "www-data" 
server.port                 = 80

# interactions with backends
# server.stream-request-body = 2
# server.stream-response-body = 2

# strict parsing and normalization of URL for consistency and security
# https://redmine.lighttpd.net/projects/lighttpd/wiki/Server_http-parseoptsDetails
# (might need to explicitly set "url-path-2f-decode" = "disable" 
# if a specific application is encoding URLs inside url-path)
server.http-parseopts = (
  "header-strict"           => "enable",# default
  "host-strict"             => "enable",# default
  "host-normalize"          => "enable",# default
  "url-normalize-unreserved"=> "enable",# recommended highly
  "url-normalize-required"  => "enable",# recommended
  "url-ctrls-reject"        => "enable",# recommended
  "url-path-2f-decode"      => "disable",# recommended highly (unless breaks app)
 #"url-path-2f-reject"      => "enable",
  "url-path-dotseg-remove"  => "enable",# recommended highly (unless breaks app)
 #"url-path-dotseg-reject"  => "enable",
 #"url-query-20-plus"       => "enable",# consistency in query string
)

index-file.names            = ( "index.php", "index.html" )
url.access-deny             = ( "~", ".inc" )
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )

compress.cache-dir          = "/var/cache/lighttpd/compress/" 
compress.filetype           = ( "application/javascript", "text/css", "text/html", "text/plain" )

# default listening port for IPv6 falls back to the IPv4 port
include_shell "/usr/share/lighttpd/use-ipv6.pl " + server.port
include_shell "/usr/share/lighttpd/create-mime.conf.pl" 
include "/etc/lighttpd/conf-enabled/*.conf" 
include "/etc/lighttpd/sites-enabled/*.conf" 

#server.compat-module-load   = "disable" 
server.modules += (
        "mod_compress",
        "mod_dirlisting",
        "mod_staticfile",
)

/etc/lighttpd/conf-enabled/10-core.conf

# /etc/lighttpd/conf-enabled/10-core.conf
server.modules += (
        "mod_openssl",
        "mod_proxy",
        "mod_auth",
        "mod_accesslog",
        "mod_setenv",
        "mod_rewrite",
)

mimetype.assign         += (".dat" => "application/x-ns-proxy-autoconfig" )

#accesslog.format       = "%V %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" 
#accesslog.filename     = "/var/log/lighttpd/access.log" 

accesslog.use-syslog    = "enable" 

# ssl.read-ahead = "disable"

/etc/lighttpd/sites-enabled/10-default.conf

# /etc/lighttpd/sites-enabled/10-default.conf
$HTTP["host"] == "docker.local.example.net" {
        server.document-root    = "/var/www/empty/html" 
        $HTTP["scheme"] == "http" {
                url.redirect = ( "" => "https://${url.authority}${url.path}${qsa}")
        }
}

$SERVER["socket"] == "192.168.2.254:443" {

        ssl.engine      = "enable" 
        ssl.pemfile     = "/etc/ssl/LetsEncrypt/example.net/aurelien.example.net/aurelien.example.net.bundle.pem" 
        ssl.cipher-list = "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384" 
        ssl.openssl.ssl-conf-cmd = ("Protocol" => "ALL, -SSLv2, -SSLv3, -TLSv1, -TLSv1.1")
        ssl.honor-cipher-order = "disable" 
        #ssl.read-ahead = "disable" 

        $HTTP["host"] == "aurelien.example.net" {
                include "/etc/lighttpd/includes.d/aurelien.example.net.conf" 
        }

        $HTTP["host"] == "test1.example.net" {
                include "/etc/lighttpd/includes.d/test1.example.net.conf" 
        }

        $HTTP["host"] == "docker.local.example.net" {
                protocol        = "https://" 

                ssl.cipher-list = "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384" 
                ssl.openssl.ssl-conf-cmd = ("Protocol" => "ALL, -SSLv2, -SSLv3, -TLSv1")
                ssl.honor-cipher-order = "disable" 

                ssl.pemfile     = "/etc/ssl/RocketCloudServicesIntermediateCA/bundle/docker.local.example.net/docker.local.example.net.pem" 

                server.document-root    = "/var/www/empty/html" 
                accesslog.filename      = "/var/log/lighttpd/docker.local.example.net.access.log" 

                setenv.add-response-header  = (
                        "Strict-Transport-Security" => "max-age=63072000" 
                )

                setenv.add-environment = (
                        "HTTPS" => "on" 
                )

                proxy.header = (
                        "upgrade" => "enable" 
                )

                $HTTP["url"] =~ "^/v1/" {
                        proxy.server = (
                                "" => ( (
                                        "host" => "192.168.2.201",
                                        "port" => 5000
                                ) )
                        )
                } else $HTTP["url"] =~ "^/v2/" {
                        proxy.server = (
                                "" => ( (
                                        "host" => "192.168.2.201",
                                        "port" => 5000
                                ) )
                        )
                } else {
                        proxy.server = (
                                "" => ( (
                                        "host" => "192.168.2.201",
                                        "port" => 5080
                                ) )
                        )
                }
        }
}

/etc/lighttpd/sites-enabled/20-public.conf

# /etc/lighttpd/sites-enabled/20-public.conf
$HTTP["host"] =~ "aurelien\.example\.net$" {
        server.document-root    = "/var/www/empty/html" 
        $HTTP["scheme"] == "http" {
                url.redirect = ( "" => "https://${url.authority}${url.path}${qsa}")
        }
}

$HTTP["host"] =~ "test1\.example\.net$" {
        server.document-root    = "/var/www/empty/html" 
        $HTTP["scheme"] == "http" {
                url.redirect = ( "" => "https://${url.authority}${url.path}${qsa}")
        }
}

$HTTP["host"] =~ "test2\.example\.net$" {
        server.document-root    = "/var/www/empty/html" 
        $HTTP["url"] =~ "^/" {
                proxy.server = (
                        "" => ( (
                                "host" => "192.168.2.201",
                                "port" => 11080
                        ) )
                )
        }

}

$SERVER["socket"] == "192.168.1.254:443" {

        ssl.engine      = "enable" 
        ssl.pemfile     = "/etc/ssl/LetsEncrypt/example.net/aurelien.example.net/aurelien.example.net.bundle.pem" 
        ssl.cipher-list = "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384" 
        ssl.openssl.ssl-conf-cmd = ("Protocol" => "ALL, -SSLv2, -SSLv3, -TLSv1, -TLSv1.1")
        ssl.honor-cipher-order = "disable" 

        $HTTP["host"] == "aurelien.example.net" {
                include "/etc/lighttpd/includes.d/aurelien.example.net.conf" 
        }

        $HTTP["host"] == "test1.example.net" {
                include "/etc/lighttpd/includes.d/test1.example.net.conf" 
        }
}

Actions

Copy link

Updated by gstrauss almost 5 years ago

What do the request headers look like? What is the HTTP method used? Is this a POST request?

Temporarily place these in your lighttpd config (and then restart lighttpd)

debug.log-request-header = "enable" 
debug.log-response-header = "enable"

Actions

Copy link

Updated by gstrauss almost 5 years ago

lighttpd mod_proxy makes an HTTP/1.0 request to the backend. Is your request sending Content-Length? It should be able to do so for such a large image. I am going to guess that the request is HTTP/1.1 and sending Transfer-Encoding: chunked. lighttpd supports that on front-side, but not with server.stream-request-body = 2 using mod_proxy to send to a backend.

Can your HTTP request send Content-Length?

Actions

Copy link

Updated by aureq almost 5 years ago

Thank you for your answer.

To clarify, I'm not making the HTTP request, but instead this is the dockerd daemon that's actually doing the image transfer to the image registry.
So, dockerd daemon is communicating to Lighty and Lighty communicates to the image registry service.

Also, my assumption (but I could be wrong) is it's a PATCH request, not a POST. But the way the data is submitted is likely to be a POST (as opposed to a GET) since each blob could be quite large. I have no idea about the Transfer-Encoding though.

Following up on your request for debug logs, I've pasted the logs at https://paste.lighttpd.net/17#sfScrvbZuMKpHqAs9s4TeaXW
At the moment server.stream-request-body is commented out, so it is set to the default value.

Actions

Copy link

Updated by gstrauss almost 5 years ago

Tracker changed from Bug to Feature

The HTTP POST requests in your paste have Content-Length: 0, which is fine.

The HTTP PATCH requests have Transfer-Encoding: chunked, which is very likely what is running into the lighttpd (current) limitation I mentioned previously.

To be pedantic, lighttpd is HTTP/1.1 compliant in returning 411 Length Required; lighttpd is permitted to do so by the HTTP/1.1 specification. While it would be much nicer if lighttpd supported Transfer-Encoding chunked when proxying to backends, the program that is not compliant with the HTTP/1.1 specification is the dockerd daemon, which should handle 411 Length Required, and appears to not even be prepared for an "HTTP/1.1 411 Length Required" HTTP error response.

I am changing this ticket to be a Feature Request, as the missing functionality is a known limitation and is a feature that can be added to lighttpd. It is not a trivial feature to add, so will take some time.

Actions

Copy link

Updated by gstrauss almost 5 years ago

If you're feeling adventurous, see DevelGit and look at my development branch personal/gstrauss/master. I committed some (as yet) completely untested code which might handle this specific case as HTTP/1.1 to the backend. YMMV and the code may change further.

Actions

Copy link