Project

General

Profile

Bug #3006

Lighty returns HTTP 411 Length Required with proxy and streaming requests/reponses body

Added by aureq 6 days ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
mod_proxy
Target version:
Start date:
2020-02-14
Due date:
% Done:

0%

Estimated time:
Missing in 1.5.x:

Description

Hi lighty labs team,

I use lighty on a Debian x86_64 as a reverse proxy. Lighty acts as a TLS server and as a HTTP request router to my Docker containers (separate host).

One of the traffic passing through is related to my Docker container registry. I push container images to the registry.
In a default configuration as highlighted below, Docker is able to push content to the registry successfully.

However, enabling the lines below appears to be breaking Docker and pushing images fails every single time.

server.stream-request-body = 2
server.stream-response-body = 2

The error message returned is:

error parsing HTTP 411 response body: invalid character '<' looking for beginning of value: "<?xml version=\"1.0\" encoding=\"iso-8859-1\"?>\n<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\"\n         \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">\n<html xmlns=\"http://www.w3.org/1999/xhtml\" xml:lang=\"en\" lang=\"en\">\n <head>\n  <title>411 Length Required</title>\n </head>\n <body>\n  <h1>411 Length Required</h1>\n </body>\n</html>\n" 

Sniffing the traffic on the Docker host, I've been able to confirm that:
  • The Docker registry doesn't return any HTTP 411 error code. I'm assuming this is Lighty because the docker registry host name points at Lighty and there's nothing in between.
  • The Docker registry doesn't receive the 'PATCH' (HTTP verb) used in uploading blobs to the container registry.

Disabling the configuration directives server.stream-(request|response)-body fix the issue.

Note 1: I have a need for server.stream-(request|response)-body as the Lighty box is a very weak server with slow local storage. These 2 options allow me to efficiently stream data in and out without the need of creating temporary files on the Lighty host.
Note 2: I have a VM that enables me to recompile Debian packages, so I'm happy to test any patches and report back any results.

Here is a small diagram to explain the problem from a different angle.

 +---------------------------------------+
 |  Docker Host                          |
 |     ^    +                            |
 |     |    |   +----------+   +-------+ |
 |     |    |   | Container|   | Other | |
 |     |    |   | Registry |   | Cont. | |
 |     |    |   +--^---+---+   +-------+ |
 +-----|----|------|---|-----------------+
       |    |      |   |
       |    |      |   |
       |    |      |   |
       |    |      |   |
      2|   3|6    4|  5|
       |    |      |   |
  +----+----v----------v----+            +-------------------------+
  |                         |  7         |                         |
  |                         +------------>                         |
  |                         |            |                         |
  |   Lighty server         <------------+    Docker Client        |
  |                         |  1         |                         |
  |                         |            |                         |
  |                         |            |                         |
  +-------------------------+            +-------------------------+

1 - Docker client sends 'push' command to Docker Host (HTTPS)
2 - Lighty relays command to Docker Host (HTTP)
3 - Docker Host initiates 'push' to registry (HTTPS to docker.local.example.net)
4 - Lighty sends HTTP traffic to registry (HTTP to 192.168.2.201:5000/v2/)
5 - Registry acknowledges commands until completion
6 - Docker Host acknowledges completion of 'push'
7 - Lighty relays to Docker client completion of 'push'

My observation make me think the issue happens after step 3 and before step 4 since the 'PATCH' (blob upload) is never received by the registry.

As requested, I'm including my lighty configuration. Maybe I've misconfigured something in there that's causing the problem.

I hope this is enough details included here.

Technical information
OS: Debian 10 (up to date as of 2020/02/14)
Lighttpd version 1.4.53-4 (as provided by Debian)
HTTP client: Docker version 18.09.7, build 18.09.7

Configuration:
/etc/lighttpd/lighttpd.conf

# /etc/lighttpd/lighttpd.conf
server.modules = (
        "mod_indexfile",
        "mod_access",
        "mod_alias",
        "mod_redirect",
)

server.document-root        = "/var/www/html" 
server.upload-dirs          = ( "/var/cache/lighttpd/uploads" )
server.errorlog             = "/var/log/lighttpd/error.log" 
server.pid-file             = "/var/run/lighttpd.pid" 
server.username             = "www-data" 
server.groupname            = "www-data" 
server.port                 = 80

# interactions with backends
# server.stream-request-body = 2
# server.stream-response-body = 2

# strict parsing and normalization of URL for consistency and security
# https://redmine.lighttpd.net/projects/lighttpd/wiki/Server_http-parseoptsDetails
# (might need to explicitly set "url-path-2f-decode" = "disable" 
# if a specific application is encoding URLs inside url-path)
server.http-parseopts = (
  "header-strict"           => "enable",# default
  "host-strict"             => "enable",# default
  "host-normalize"          => "enable",# default
  "url-normalize-unreserved"=> "enable",# recommended highly
  "url-normalize-required"  => "enable",# recommended
  "url-ctrls-reject"        => "enable",# recommended
  "url-path-2f-decode"      => "disable",# recommended highly (unless breaks app)
 #"url-path-2f-reject"      => "enable",
  "url-path-dotseg-remove"  => "enable",# recommended highly (unless breaks app)
 #"url-path-dotseg-reject"  => "enable",
 #"url-query-20-plus"       => "enable",# consistency in query string
)

index-file.names            = ( "index.php", "index.html" )
url.access-deny             = ( "~", ".inc" )
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )

compress.cache-dir          = "/var/cache/lighttpd/compress/" 
compress.filetype           = ( "application/javascript", "text/css", "text/html", "text/plain" )

# default listening port for IPv6 falls back to the IPv4 port
include_shell "/usr/share/lighttpd/use-ipv6.pl " + server.port
include_shell "/usr/share/lighttpd/create-mime.conf.pl" 
include "/etc/lighttpd/conf-enabled/*.conf" 
include "/etc/lighttpd/sites-enabled/*.conf" 

#server.compat-module-load   = "disable" 
server.modules += (
        "mod_compress",
        "mod_dirlisting",
        "mod_staticfile",
)

/etc/lighttpd/conf-enabled/10-core.conf

# /etc/lighttpd/conf-enabled/10-core.conf
server.modules += (
        "mod_openssl",
        "mod_proxy",
        "mod_auth",
        "mod_accesslog",
        "mod_setenv",
        "mod_rewrite",
)

mimetype.assign         += (".dat" => "application/x-ns-proxy-autoconfig" )

#accesslog.format       = "%V %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" 
#accesslog.filename     = "/var/log/lighttpd/access.log" 

accesslog.use-syslog    = "enable" 

# ssl.read-ahead = "disable" 

/etc/lighttpd/sites-enabled/10-default.conf

# /etc/lighttpd/sites-enabled/10-default.conf
$HTTP["host"] == "docker.local.example.net" {
        server.document-root    = "/var/www/empty/html" 
        $HTTP["scheme"] == "http" {
                url.redirect = ( "" => "https://${url.authority}${url.path}${qsa}")
        }
}

$SERVER["socket"] == "192.168.2.254:443" {

        ssl.engine      = "enable" 
        ssl.pemfile     = "/etc/ssl/LetsEncrypt/example.net/aurelien.example.net/aurelien.example.net.bundle.pem" 
        ssl.cipher-list = "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384" 
        ssl.openssl.ssl-conf-cmd = ("Protocol" => "ALL, -SSLv2, -SSLv3, -TLSv1, -TLSv1.1")
        ssl.honor-cipher-order = "disable" 
        #ssl.read-ahead = "disable" 

        $HTTP["host"] == "aurelien.example.net" {
                include "/etc/lighttpd/includes.d/aurelien.example.net.conf" 
        }

        $HTTP["host"] == "test1.example.net" {
                include "/etc/lighttpd/includes.d/test1.example.net.conf" 
        }

        $HTTP["host"] == "docker.local.example.net" {
                protocol        = "https://" 

                ssl.cipher-list = "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384" 
                ssl.openssl.ssl-conf-cmd = ("Protocol" => "ALL, -SSLv2, -SSLv3, -TLSv1")
                ssl.honor-cipher-order = "disable" 

                ssl.pemfile     = "/etc/ssl/RocketCloudServicesIntermediateCA/bundle/docker.local.example.net/docker.local.example.net.pem" 

                server.document-root    = "/var/www/empty/html" 
                accesslog.filename      = "/var/log/lighttpd/docker.local.example.net.access.log" 

                setenv.add-response-header  = (
                        "Strict-Transport-Security" => "max-age=63072000" 
                )

                setenv.add-environment = (
                        "HTTPS" => "on" 
                )

                proxy.header = (
                        "upgrade" => "enable" 
                )

                $HTTP["url"] =~ "^/v1/" {
                        proxy.server = (
                                "" => ( (
                                        "host" => "192.168.2.201",
                                        "port" => 5000
                                ) )
                        )
                } else $HTTP["url"] =~ "^/v2/" {
                        proxy.server = (
                                "" => ( (
                                        "host" => "192.168.2.201",
                                        "port" => 5000
                                ) )
                        )
                } else {
                        proxy.server = (
                                "" => ( (
                                        "host" => "192.168.2.201",
                                        "port" => 5080
                                ) )
                        )
                }
        }
}

/etc/lighttpd/sites-enabled/20-public.conf

# /etc/lighttpd/sites-enabled/20-public.conf
$HTTP["host"] =~ "aurelien\.example\.net$" {
        server.document-root    = "/var/www/empty/html" 
        $HTTP["scheme"] == "http" {
                url.redirect = ( "" => "https://${url.authority}${url.path}${qsa}")
        }
}

$HTTP["host"] =~ "test1\.example\.net$" {
        server.document-root    = "/var/www/empty/html" 
        $HTTP["scheme"] == "http" {
                url.redirect = ( "" => "https://${url.authority}${url.path}${qsa}")
        }
}

$HTTP["host"] =~ "test2\.example\.net$" {
        server.document-root    = "/var/www/empty/html" 
        $HTTP["url"] =~ "^/" {
                proxy.server = (
                        "" => ( (
                                "host" => "192.168.2.201",
                                "port" => 11080
                        ) )
                )
        }

}

$SERVER["socket"] == "192.168.1.254:443" {

        ssl.engine      = "enable" 
        ssl.pemfile     = "/etc/ssl/LetsEncrypt/example.net/aurelien.example.net/aurelien.example.net.bundle.pem" 
        ssl.cipher-list = "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384" 
        ssl.openssl.ssl-conf-cmd = ("Protocol" => "ALL, -SSLv2, -SSLv3, -TLSv1, -TLSv1.1")
        ssl.honor-cipher-order = "disable" 

        $HTTP["host"] == "aurelien.example.net" {
                include "/etc/lighttpd/includes.d/aurelien.example.net.conf" 
        }

        $HTTP["host"] == "test1.example.net" {
                include "/etc/lighttpd/includes.d/test1.example.net.conf" 
        }
}

No data to display

Also available in: Atom