Project

General

Profile

[Solved] Proxy for wss://

Added by vicencb 3 months ago

Hi!
I want to access a server (REMOTE) behind double NAT from the Internet.
In order to solve the issue i've setup a wireguard tunnel to another computer (PUBLIC) with a public address.
From the PUBLIC computer i can access the REMOTE server correctly via http:// and ws://.

Then, on the PUBLIC computer i set up a proxy that allows access to REMOTE,
but as it is connected to the Internet i need encryption and authentication.
With this setup, access to https:// works fine, but wss:// fails due to wrong password.

Here is a way that reproduces the issue i have:

User is u and password is p.

index.html

<!DOCTYPE html><pre id="log"></pre><script>
  const logelt = document.getElementById('log');
  function log(msg) { logelt.textContent += msg + '\n'; }
  let ll = 0;
  function send_msg() { if (++ll <= 5) { log('SEND: '+ll); ws.send(ll+'\n'); } else { ws.close(); } }
  let ws_url = window.location.protocol === 'https:' ? 'wss://' : 'ws://';
  ws_url += window.location.host + "/ws";
  console.log(ws_url);
  let ws = new WebSocket(ws_url);
  ws.onopen    = function(event) { log('CONNECT\n'); send_msg(); };
  ws.onclose   = function(event) { log('DISCONNECT'); };
  ws.onmessage = function(event) { log('RECV: ' + event.data); setTimeout(send_msg, 333); };
  ws.onerror   = function(event) { log('ERROR'); console.log(event); };
</script>

echo.pl

#!/usr/bin/perl -Tw
$SIG{PIPE} = 'IGNORE';
for (my $FH; accept($FH, STDIN); close $FH) {
  select($FH); $|=1;
  print $FH $_ while (<$FH>);
}

server.conf

server.document-root = var.CWD
server.port = 8080
index-file.names = ("index.html")
server.modules = ("mod_wstunnel")
wstunnel.server = ("/ws" => (("socket" => "echo.sock", "bin-path" => "echo.pl")))

proxy.conf

server.modules = ("mod_openssl", "mod_auth", "mod_authn_file", "mod_proxy")
server.port = 44380
server.document-root = "No" 

# Encryption
ssl.engine = 1
ssl.pemfile = "testprox.pem" 
ssl.openssl.ssl-conf-cmd = ("MinProtocol" => "TLSv1.3", "Options" => "-ServerPreference")

# Authentication
auth.backend = "htdigest" 
auth.backend.htdigest.userfile = "testprox.usr" 
auth.require = ("" => ("method" => "digest", "algorithm" => "SHA-256", "realm" => "r", "require" => "valid-user"))

# Removing either Encryption or Authentication makes it work.
# Otherwise, it reports invalid password when trying to connect to wss://

proxy.server = ("" => (("host" => "127.0.0.1", "port" => 8080)))
proxy.header = ("upgrade" => 1, "connect" => 1)

server.errorlog = "proxy.log" 
debug.log-request-handling = 1

run_test

#!/bin/sh -e

if [ ! -f testprox.pem ] ; then
  openssl req -x509 -noenc -keyout testroot.key -out testroot.crt -subj /CN=TestRoot
  openssl req -x509 -noenc -keyout testprox.pem -out testprox.pem -subj /CN=TestServ \
    -CAkey testroot.key -CA testroot.crt -extensions usr_cert -addext "subjectAltName=DNS:localhost,IP:127.0.0.1" 
  rm testroot.key
  echo 'Add testroot.crt to your web browser list of CAs'
fi

if [ ! -f testprox.usr ] ; then
  printf 'u:r:p' | sha256sum | sed 's/^/u:r:/;s/ .*//' > testprox.usr
fi

rm -f proxy.log

lighttpd -D -f server.conf &
lighttpd -D -f proxy.conf  &

# firefox --private-window http://localhost:8080
firefox --private-window http://localhost:44380

killall lighttpd

Please, can somebody review this issue and tell me if this is a misconfiguration or a bug and a fix for it?

Regards,
Vicente.


Replies (26)

RE: Proxy for wss:// - Added by vicencb 3 months ago

In run_test change
http://localhost:44380
to
https://localhost:44380

The http:// was a test i did without encryption.

RE: Proxy for wss:// - Added by vicencb 3 months ago

All files attached and ready to download.

RE: Proxy for wss:// - Added by gstrauss 3 months ago

lighttpd supports HTTP/2 extended CONNECT :protocol: websocket.
lighttpd supports HTTP Digest auth algorithm SHA-256.

Perhaps your client fails to support one or both of these, or in combination.
Try using HTTP Digest auth algorithm MD5: mod_auth
Try temporarily disabling HTTP/2 in lighttpd: server.feature-flags

If you think the problem is with lighttpd, then please capture request and response headers for the requests: DebugVariables

RE: Proxy for wss:// - Added by vicencb 3 months ago

MD5 behaviour is the same.
Disabling HTTP/2 makes it behave differently.

1.- For the case with HTTP/2 enabled:
The attached case reproducer now includes the logs with all the headers.
The interesting part is at proxy.log lines 92 and 104.

2.- For the case with HTTP/2 disabled:
It works sometimes, but in this case i think the issue is with the client.
When the page is first loaded it works fine.
On a second attempt, the client re-uses the cached index.html and then tries to connect to wss:// without providing the authorization credentials, so it fails.

RE: Proxy for wss:// - Added by gstrauss 3 months ago

Why are you allowing that page to be cached? Perhaps you should expicitly send Cache-Control: no-cache

RE: Proxy for wss:// - Added by gstrauss 3 months ago

HTTP/2 extended CONNECT builds on the HTTP/2 protocol and should work with HTTP Digest Authentication. If the client is not sending authentication headers, then that is a client problem, not a problem with lighttpd. On the other hand, if the client is sending authentication headers and lighttpd HTTP/2 extended CONNECT is not handling that, then that is something I can look into further.

From proxy.log:
2024-08-16 00:23:53: (../src/mod_auth.c.1546) digest: auth failed for u: wrong password, IP: 127.0.0.1

That means exactly what it says: that what was sent by the client did not pass digest auth checks.
You might try testing with Basic Auth to narrow down the issue.

I'll try to find some time later to try your reproduction test scripts.


auth.backend.htdigest.userfile = "testprox.usr"
You should prefer a full path to the file instead of assuming current directory. If lighttpd is run as a daemon, then lighttpd changes working directory to /.

RE: Proxy for wss:// - Added by vicencb 3 months ago

The file index.html is static, so caching is beneficial for speed.
Are you suggesting to disable cache as a solution or as a workaround?

RE: Proxy for wss:// - Added by gstrauss 3 months ago

If you're passing through multiple proxies and changing IP addresses, please see mod_extforward if you want to log actual remote IP on a server besides the frontend.

RE: Proxy for wss:// - Added by gstrauss 3 months ago

Are you suggesting to disable cache as a solution or as a workaround?

Neither. I am not making any suggestions about solutions or workarounds until after the problem is identified.

If a request is omitting sending authentication info and failing to handle 401, then you have to figure out why that is happening for the client, and then choose a different approach for a solution/workaround for the client. Do you know for certain whether this is an issue with the server, the client, or both? If not, then please troubleshoot as if any of those could be the case.

RE: Proxy for wss:// - Added by vicencb 3 months ago

Using the basic auth method makes it work:
auth.backend = "plain"
auth.backend.plain.userfile = "testprox-basic.usr"
auth.require = ("" => ("method" => "basic", "realm" => "r", "require" => "valid-user"))

Regarding full paths: yes, i know they should be preferred and i use them in the real case.
I was just trying to make the reproducer as simple as possible.

RE: Proxy for wss:// - Added by gstrauss 3 months ago

Does it work with HTTP/1.1 and HTTP Basic Auth?
Does it work with HTTP/1.1 and HTTP Digest Auth?
Does it work with HTTP/2 and HTTP Basic Auth?
Does it work with HTTP/2 and HTTP Digest Auth?

RE: Proxy for wss:// - Added by gstrauss 3 months ago

Are you testing with Cache-Control: no-cache response header? Or Cache-Control: private ? Or Cache-Control: must-revalidate ?

RE: Proxy for wss:// - Added by gstrauss 3 months ago

2024-08-16 00:23:53: (../src/mod_auth.c.1546) digest: auth failed for u: wrong password, IP: 127.0.0.1

line 1546 of src/mod_auth.c indicates a mismatch of the digest provided compared to the calculated digest.

Unverified: I wonder if Firefox is calculating hash digest using a different HTTP method with HTTP/2 extended CONNECT than that used by lighttpd (CONNECT).

https://www.rfc-editor.org/rfc/rfc8441#section-5 notes that https://www.rfc-editor.org/rfc/rfc6455 uses a GET-based request in WebSockets opening handshake.

RE: Proxy for wss:// - Added by vicencb 3 months ago

Hi, from the 4 tests combinations 1.1/2 basic/digest, the only one that fails is 2/digest.
Attached are the 4 logs.

Regarding Cache-Control, i am not setting anything, so, defaults apply.
Although on the real case, there are some files that are "Cache-Control: no-store".

RE: Proxy for wss:// - Added by gstrauss 3 months ago

Since HTTP Auth Digest works with HTTP/1.1, but not with HTTP/2, the difference might be the HTTP method that the client is using in the hash.

The lighttpd code for HTTP Auth Digest is the same for HTTP/1.1 and HTTP/2; there is no HTTP version check there which might change behavior.

Have you tried testing HTTP/2 and HTTP Auth Digest for a regular HTTP/2 request? That works, right? So it is only the HTTP/2 extended CONNECT with HTTP Auth Digest that is failing? That points to the method used in the hash.

Have you tried a different clients with HTTP/2 and HTTP Auth Digest? Chrome? curl? (https://curl.se/docs/websocket.html)

RE: Proxy for wss:// - Added by gstrauss 3 months ago

If you wanted to temporarily modify the lighttpd code to test Firefox wss with HTTP Digest auth, specifically if Firefox is using GET method in the hash, you can modify
src/mod_auth.c line 1541
mod_auth_digest_mutate(&ai, &dp, http_method_buf(r->http_method));
to be
mod_auth_digest_mutate(&ai, &dp, http_method_buf(HTTP_METHOD_GET));
Recompiling lighttpd is much easier that recompiling Firefox.

RE: Proxy for wss:// - Added by vicencb 3 months ago

Have you tried testing HTTP/2 and HTTP Auth Digest for a regular HTTP/2 request? That works, right? So it is only the HTTP/2 extended CONNECT with HTTP Auth Digest that is failing?

I am not a field expert, i don't understand the question. Please, can you re-phrase?

OK, later i'll try again with recompiled lighttpd with your proposed change for testing.

RE: Proxy for wss:// - Added by gstrauss 3 months ago

Have you tried testing HTTP/2 and HTTP Auth Digest for a regular HTTP/2 request? That works, right? So it is only the HTTP/2 extended CONNECT with HTTP Auth Digest that is failing?

I am not a field expert, i don't understand the question. Please, can you re-phrase?

If you make an HTTP/2 GET request, does the HTTP Digest authentication work?
It's only the wss request over HTTP/2 (using HTTP/2 extended CONNECT) that is failing, right?

Try different clients. I think this may be a bug in Firefox, or at least not specified in https://www.rfc-editor.org/rfc/rfc8441 and possibly an underspecified interaction between HTTP extended CONNECT and HTTP Digest authentication.

RE: Proxy for wss:// - Added by vicencb 3 months ago

Using chromium also fails.

Your test of forcing http_method_buf to HTTP_METHOD_GET makes it work in both firefox and chromium.

RE: Proxy for wss:// - Added by vicencb 3 months ago

If you make an HTTP/2 GET request, does the HTTP Digest authentication work?
It's only the wss request over HTTP/2 (using HTTP/2 extended CONNECT) that is failing, right?

RE: Proxy for wss:// - Added by vicencb 3 months ago

Sorry for the previous post, was unintentional.

If you make an HTTP/2 GET request, does the HTTP Digest authentication work?

Yes, that is the method used to download index.html and it works.

It's only the wss request over HTTP/2 (using HTTP/2 extended CONNECT) that is failing, right?

Correct.

But i am still not sure i've understood correctly the question, as those answers are in the previously attached logs...

RE: Proxy for wss:// - Added by gstrauss 3 months ago

But i am still not sure i've understood correctly the question, as those answers are in the previously attached logs...

You have understood. Yes, it was in the logs, but not explicitly summarized here.

In a previous post, you mentioned things sometimes working, and then not working when the client did not handle and send HTTP Digest authorization. Seems like you have sorted out the client-side code issue, with or without changing caching settings.

I'll post a workaround patch below for wss using HTTP/2 extended CONNECT along with HTTP Digest Authentication, but will be posting to the IETF ietf-http-wg mailing list to ask for clarification on why the client HTTP Digest Auth is misusing HTTP extended CONNECT with wss://, since the HTTP method is CONNECT, not GET. If you get a chance, please try testing with curl.

BTW, thank you for posting instructions how to reproduce.
Sorry, I haven't had time to reproduce this myself, as I am travelling and will soon be offline for a couple weeks.

This patch is untested, but likely more correct than the hard coding test you did earlier:

--- a/src/mod_auth.c
+++ b/src/mod_auth.c
@@ -1538,7 +1538,12 @@ mod_auth_check_digest (request_st * const r, void *p_d, const struct http_auth_r
     if (__builtin_expect( (HANDLER_GO_ON != rc), 0))
         return rc;

+  if (r->h2_connect_ext) {
+    mod_auth_digest_mutate(&ai, &dp, http_method_buf(HTTP_METHOD_GET));
+  }
+  else {
     mod_auth_digest_mutate(&ai, &dp, http_method_buf(r->http_method));
+  }

     if (!ck_memeq_const_time_fixed_len(dp.rdigest, ai.digest, ai.dlen)) {
         /*ck_memzero(ai.digest, ai.dlen);*//*skip clear since mutated*/

I bristle at having to add a special-case. I think Chrome and Firefox are doing it incorrectly, but I am not sure whether or not it can be changed at this point.

RE: Proxy for wss:// - Added by gstrauss 3 months ago

I posted a question to the ietf-http-wg mailing list. This is the start of the thread:
https://lists.w3.org/Archives/Public/ietf-http-wg/2024JulSep/0169.html

RE: Proxy for wss:// - Added by vicencb 3 months ago

Thank you for the patch that works around the issue. I've applied it and it works.
I'll be looking forward for a proper solution...

In a previous post, you mentioned things sometimes working, and then not working when the client did not handle and send HTTP Digest authorization.

I think this is another issue and it is still there, even with the workaround patch applied.
I've tested with both firefox and chromium and both behave the same.
Using --private-window in firefox makes it work, but that is a workaround, not a solution because in this case it doesn't offer the option of saving the password for that site, among other things.
When not using --private-window, the first time the page is visited it asks for the password, the page is loaded correctly and then it offers to save the password, which i consent.
Then, on a second visit to the page, the static part (html, js) is loaded from the client cache and then proceeds with the websocket connection without providing auth credentials.
A refresh of the page F5 then asks again for the password (the form is automatically filled in) and just pressing enter works fine.

Attached are the logs for both firefox and chromium.

Any ideas where to start hunting this second issue?


The very first line of this conversation is about a server behind double NAT.
Is this the easiest way (from the client point of view) of accessing such a server from the Internet?

RE: Proxy for wss:// - Added by vicencb 2 months ago

The issue of the client connecting to the websocket without providing auth credentials is only reproducible with the test case provided here.
It turns out that on the real case (which i have already deployed using Basic Auth) it all works fine.
Thank you for your support!

(1-25/26)