Project

General

Profile

Bug #3063

closed

websocket proxy fails if 101 Switching Protocols from backend includes Content-Length

Added by daimh 22 days ago. Updated 20 days ago.

Status:
Fixed
Priority:
Normal
Category:
core
Target version:
ASK QUESTIONS IN Forums:
No

Description

After my lighttpd was upgraded from 1.4.56 to 1.4.59, its websocket proxy stopped working. I tried some other versions in between, it seems 1.4.57 and 1.4.58 has the same issue too.

Compared the two versions in firefox web developer console network tab, Firefox send the message below to lighttpd under both versions,

{"method":"init","data":{"bins":30,".clientdata_output_distPlot_width":610,".clientdata_output_distPlot_height":400,".clientdata_output_distPlot_bg":"rgb(255, 255, 255)",".clientdata_output_distPlot_fg":"rgb(51, 51, 51)",".clientdata_output_distPlot_accent":"rgb(51, 122, 183)",".clientdata_output_distPlot_font":{"families":["Helvetica Neue","Helvetica","Arial","sans-serif"],"size":"14px"},".clientdata_output_distPlot_hidden":false,".clientdata_pixelratio":1,".clientdata_url_protocol":"https:",".clientdata_url_hostname":"y.mbni.org",".clientdata_url_port":"",".clientdata_url_pathname":"/dashboard/daimh/test/",".clientdata_url_search":"",".clientdata_url_hash_initial":"",".clientdata_url_hash":"",".clientdata_singletons":"",".clientdata_allowDataUriScheme":true}}

However, only 1.4.56 returns a lot of messages. The one below is the first. {"config":{"workerId":"","sessionId":"fed34dd7d65d17d4e2d0ac3c512de812","user":null}}


Files

lighttpd-proxy.zip (103 KB) lighttpd-proxy.zip daimh, 2021-02-03 13:40
#1

Updated by daimh 22 days ago

the configuration.

$HTTP["url"] =~ "^/dashboard/daimh/test/" {
proxy.server = ( "" => ( ( "host" => "mengf-s1", "port" => "1024" ) ) )
proxy.header = (
"https-remap" => "enable",
"map-urlpath" => ( "/dashboard/daimh/test/" => "/" ),
"upgrade" => "enable",
)
}

The actual web socket server I used for test is an R shiny demo.

echo "library(shiny); runExample('01_hello', host='0.0.0.0', port = 1024, launch.browser=F)" | R -q --no-save

#2

Updated by gstrauss 22 days ago

Please share your lighttpd config: lighttpd -f /etc/lighttpd/lighttpd.conf -p (minus any passwords, if present)

#3

Updated by daimh 22 days ago

Thanks a lot for getting back to me!

There are two config files. One is /etc/lighttpd/lighttpd.conf ################

server.port        = 80
server.username        = "http" 
server.groupname    = "http" 
server.document-root    = "/srv/http" 
server.errorlog        = "/var/log/lighttpd/error.log" 
dir-listing.activate    = "disable" 
index-file.names    = (
                "index.html",
                "index.py",
                "index.sh",
            )
mimetype.assign        = (
                ".html" => "text/html",
                ".txt" => "text/plain",
                ".css" => "text/css",
                ".js" => "application/x-javascript",
                ".jpg" => "image/jpeg",
                ".jpeg" => "image/jpeg",
                ".gif" => "image/gif",
                ".png" => "image/png",
                ".svg" => "image/svg+xml",
                ".mp4" => "video/mp4",
                "" => "application/octet-stream" 
            )
server.max-read-idle = 1800
server.modules        += (
                "mod_accesslog",
                "mod_alias",
                "mod_auth",
                "mod_authn_file",
                "mod_cgi",
                "mod_openssl",
                "mod_proxy",
                "mod_redirect",
            )
accesslog.filename    = "/var/log/lighttpd/access.log" 
server.breakagelog    = "/var/log/lighttpd/breakage.log" 
cgi.assign        = (
                ".py2" => "/usr/bin/python2",
                ".py" => "/usr/bin/python",
                ".sh" => "/usr/bin/bash",
            )
cgi.upgrade            = "enable" #websocket
static-file.exclude-extensions = ( ".py", ".sh" )
$HTTP["scheme"] == "http" {
    $HTTP["host"] =~ ".*" {
        url.redirect = (
            "^/$" => "https://%0$0",
            "^/[^.].*" => "https://%0$0",
        )
    }
}
$SERVER["socket"] == ":443" {
    ssl.engine = "enable" 
    ssl.pemfile = "/etc/ssl/private/certbot-combined.pem" # Combined Certificate
    ssl.ca-file = "/etc/ssl/certs/certbot-chain.crt" # Root CA
    ssl.cipher-list = "FIPS: +3DES:!aNULL" 
    ssl.openssl.ssl-conf-cmd  = ("MinProtocol" => "TLSv1.2")
}
include "/opt/dashboard/var/*/*.conf" 

###And another file under /opt/dashboard/var/<USER>/<USER>.conf

$HTTP["url"] =~ "^/dashboard/daimh/test/" {
    proxy.server = ( "" => ( ( "host" => "shiny-host", "port" => "1024" ) ) )
    proxy.header = (
        "https-remap" => "enable",
        "map-urlpath" => ( "/dashboard/USER/test/" => "/" ),
        "upgrade" => "enable",
    )
}

#4

Updated by gstrauss 22 days ago

A simple websockets test worked for me, but I am continuing to test.

What is the HTTP request being sent? I'd like to confirm if the request is being handled by mod_cgi or mod_proxy.

#5

Updated by gstrauss 22 days ago

BTW, do you really want to enable +3DES in ssl.cipher-list? You probably meant -3DES
See lighttpd TLS cipher recommendations for a stronger cipher list.

#6

Updated by daimh 22 days ago

The http request I tested is an R shiny demo running on another host.
echo "library(shiny); runExample('01_hello', host='0.0.0.0', port = 1024, launch.browser=F)" | R -q --no-save

I just tested it with 1.4.56. If the line mod_proxy is not included in server.modules, the same url is handled by my CGI program with mod_cgi, but if the line mod_proxy is included in server.module, the url request is handled by mod_proxy and sent to R shiny server. This is actually I wanted, and it worked before.

In other words, if the url is /dashboard, it will be handled by my CGI program. But if the url is something like /dashboard/shinydemo, it will be handled by mod_proxy and the request is sent to another host.

Let me know if you have any question, I will check my email tomorrow morning. thanks a million!

#7

Updated by gstrauss 22 days ago

echo "library(shiny); runExample('01_hello', host='0.0.0.0', port = 1024, launch.browser=F)" | R -q --no-save

I think you have made an assumption that everyone knows what shiny is and where to get 01_hello, and has R installed. Those assumptions are all incorrect.

I just tested it with 1.4.56. If the line mod_proxy is not included in server.modules, the same url is handled by my CGI program with mod_cgi, but if the line mod_proxy is included in server.module, the url request is handled by mod_proxy and sent to R shiny server. This is actually I wanted, and it worked before.

That is slightly confusing. Would you please try to be more clear about which scenarios work and which scenarios do not? Please do not assume that I know anything about R.

From your initial post, it seems that your client is Firefox. Is that correct? What is the connection string in the javascript which is used to send an HTTP request? (i.e. what is the HTTP request?)

#8

Updated by daimh 22 days ago

So sorry for the misunderstanding. I picked an R shiny server demo in my test because it is the easiest for me and all my services that need a lighttpd websocket proxy is based on R shiny. Right now I really don't have a quick way to start a web socket server with other language to test the issue, but I will try it tomorrow morning when I get a chance to use my desktop.

And yes, my client is Firefox. But I just tested it with my android chrome, 1.4.56 still works fine, but 1.4.59 still doesn't.

I started a working demo at https://y.mbni.org/dashboard/daimh/test/ , which is running with 1.4.56. You should be able to access the url and see the last request is a websocket GET. If I upgraded lighttpd to 1.4.59, the websocket request still sends the same data, but won't get anything back.

I also did a test by fully disabling mod_cgi. The above demo still works with lighttpd 1.4.56, but not 1.4.59. Thus I believe the problem is not related to mod_cgi.

Feel free to let me know if you have any questions, your help is greatly appreciated!

#9

Updated by gstrauss 22 days ago

This is what an HTTP request looks like, which you can see under Firefox "Tools..Web Developer" console.

GET wss://y.mbni.org/dashboard/daimh/test/websocket/ HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.5
Cache-Control: no-cache
Connection: keep-alive, Upgrade
DNT: 1
Host: y.mbni.org
Origin: https://y.mbni.org
Pragma: no-cache
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: [snip]
Sec-WebSocket-Version: 13
Upgrade: websocket
User-Agent: [snip]

#10

Updated by gstrauss 22 days ago

You can use strace -s 2048 -p <pid> on the lighttpd server pid and, separately, on your websocket backend to see what lighttpd is sending to the backend and what the backend is receiving. Look for read write send recv in the strace when you send a single request that fails. Please share the results.

You said that Firefox is sending requests. Is lighttpd sending those requests to the backend? Is the backend receiving those requests? Is the backend responding to those requests? Is lighttpd receiving the response from the backend? Is lighttpd sending the response back to Firefox? (The response to Firefox is likely encrypted, so while you will not be able to read it, you can see whether or not lighttpd is sending something back to the client.)

#11

Updated by daimh 21 days ago

four strace files are attached. In the file name, 'backend-/lighttpd-' means it is on the R shiny or the lighttpd; "-56/-59" means it is done with lighttpd 1.4.56 or 1.4.59.

I glanced at them, it seems to me that lighttpd-59 doesn't have 'write' after the last line containing 'showcase-src'.

#12

Updated by gstrauss 21 days ago

Thank you for those. I took a quick look at the backend-* and they look similar in syscalls. I'll read more carefully through the lighttpd-* strace later today.

#13

Updated by daimh 21 days ago

FWIW, I just tested a simple Python websocket client/server program does work fine with both lighttpd versions as websocket proxy.

#14

Updated by gstrauss 21 days ago

Please try with the following. Note the addition of server.stream-request-body and server.stream-response-body. Streaming with mode = 2 is automatically enabled in mod_proxy after Connection: upgrade results in 101 Switching Protocols, if streaming is not already enabled. I am curious if changing the streaming mode makes a difference for you, and may help me to track down the issue.

$HTTP["url"] =~ "^/dashboard/daimh/test/" {
    server.stream-response-body = 1
    server.stream-request-body = 1
    proxy.server = ( "" => ( ( "host" => "shiny-host", "port" => "1024" ) ) )
    proxy.header = (
        "https-remap" => "enable",
        "map-urlpath" => ( "/dashboard/USER/test/" => "/" ),
        "upgrade" => "enable",
    )
}

#15

Updated by daimh 21 days ago

The problem persists after I added the two lines below to the configuration file.

server.stream-response-body = 1
server.stream-request-body = 1

#16

Updated by gstrauss 21 days ago

  • Subject changed from websocket proxy to websocket proxy fails if 101 Switching Protocols from backend includes Content-Length
  • Category set to core
  • Status changed from New to Patch Pending
  • Target version changed from 1.4.x to 1.4.60

From the strace, I see that the backend included Content-Length: 0 with the HTTP/1.1 101 Switching Protocols response. Unfortunately, that exposed a bug introduced in commit 903024d7 which fixed #3046 but in the process broke HTTP/1.1 101 Switching Protocols from backends when Content-Length: 0 is included. The Content-Length response header is permitted by the RFCs, but not necessary with HTTP status 101 Switching Protocols. The following patch resets the Content-Length tracking after 101 Switching Protocols.

--- a/src/http-header-glue.c
+++ b/src/http-header-glue.c
@@ -961,6 +961,7 @@ void http_response_upgrade_read_body_unknown(request_st * const r) {
           (FDEVENT_STREAM_RESPONSE_BUFMIN | FDEVENT_STREAM_RESPONSE);
     r->conf.stream_request_body |= FDEVENT_STREAM_REQUEST_POLLIN;
     r->reqbody_length = -2;
+    r->resp_body_scratchpad = -1;
     r->keep_alive = 0;
 }

#17

Updated by gstrauss 21 days ago

  • Status changed from Patch Pending to Fixed
#18

Updated by daimh 20 days ago

I will report back as soon as Arch Linux package is upgraded with this patch! Thanks a million! gstrauss

#19

Updated by gstrauss 20 days ago

lighttpd 1.4.59 was just released 2 Feb (2 days ago). The above patch is on the lighttpd git master branch and will be part of lighttpd 1.4.60, but the release of lighttpd 1.4.60 is not yet scheduled.

Did you submit a request to Arch package maintainers to pick up this patch in the meantime?

#20

Updated by daimh 20 days ago

I believe Arch Linux package maintainer won't do so as the 'source' line in its PKGBUILD file at the link below uses your tar.gz file, instead of your git master branch.

I can live with version 1.4.56 for a while, unless you have concern.

I truly appreciate your help! Lighttpd is an excellent software, please keep up the good work! I will report back as soon as 1.4.60 is released.

https://github.com/archlinux/svntogit-packages/blob/packages/lighttpd/trunk/PKGBUILD

#21

Updated by gstrauss 20 days ago

FYI: pkg supports patches, e.g. for apache https://github.com/archlinux/svntogit-packages/tree/master/apache/trunk

You could also take the arch repo for lighttpd, add a patch, and build your own arch package of lighttpd.

.

lighttpd 1.4.60 will probably be released in a month or two, depending on feedback.

Also available in: Atom