Lighttpd

From reading the documentation, I believe I understand that the only ways of handling websockets are mod_proxy, mod_cgi, and mod_wstunnel. Is that correct? Is there any consideration to having, say, mod_scgi also support websockets?

Currently, yes, I believe that is the case.

Is there any consideration to having, say, mod_scgi also support websockets?

The SCGI specification requires CONTENT_LENGTH as the very first request header in the SCGI request.
https://en.wikipedia.org/wiki/Simple_Common_Gateway_Interface

There are similar requirements for CGI, FastCGI, and AJP13.

In the case of CGI, mod_cgi configured with cgi.upgrade = "enable" (passively) allows CONTENT_LENGTH=-1 with HTTP_UPGRADE=websocket, where -1 is not a valid CONTENT_LENGTH.
The lighttpd src/mod_cgi.c code special-cases HTTP/2 extended connect and adds HTTP_UPGRADE=websocket and HTTP_CONNECTION=upgrade to the CGI environment, even though those technically should not be part of the CGI environment for an HTTP/2 request.

For SCGI, FastCGI, AJP13, the CONTENT_LENGTH is necessary to delineate the end of the request received over the socket by the backend.

It seems to me that a pretty obvious extension to SCGI would work

This comment is off-putting, since you clearly have not highlighted "pretty obvious" portions of the SCGI protocol specification which are incompatible with Upgrade: websocket.

Have you tried extending your SCGI server to special-case an invalid length, such as CONTENT_LENGTH=-1 when HTTP_UPGRADE=websocket is present?
Have you tried modifying the lighttpd code to see what might be necessary to pass HTTP_UPGRADE=websocket to the backend?

The SCGI specification requires CONTENT_LENGTH as the very first request header in the SCGI request.

There are similar requirements for CGI, FastCGI, and AJP13.

In the case of CGI, mod_cgi configured with cgi.upgrade = "enable" (passively) allows CONTENT_LENGTH=-1 with HTTP_UPGRADE=websocket, where -1 is not a valid CONTENT_LENGTH.
The lighttpd src/mod_cgi.c code special-cases HTTP/2 extended connect and adds HTTP_UPGRADE=websocket and HTTP_CONNECTION=upgrade to the CGI environment, even though those technically should not be part of the CGI environment for an HTTP/2 request.

Setting CONTENT_LENGTH to a magic value, like -1, or omitting it altogether (this is an extension to the existing specification, so IMO that would be a possible option, though not necessarily the best one) both seem acceptable to me.

For SCGI, FastCGI, AJP13, the CONTENT_LENGTH is necessary to delineate the end of the request received over the socket by the backend.

For a plain HTTP request, yes. Websocket doesn’t really have the concept of “the request” though, since it’s a bidirectional, potentially endless, stream of frames. It has request headers, but you find the end of those using the netstring length prefix, not the CONTENT_LENGTH header. Once the headers have been sent in both directions, it seems to me that what you’re left with is just a bidirectional bytestream socket (either TCP or UNIX), over which Websocket frames could flow, verbatim, in both directions, at any time as needed, forever. The end of a connection could be signalled with a shutdown(SHUT_WR) call, which TCP and UNIX both support, allowing the proxying to be fully transparent.

Would this be, strictly speaking, an expansion to SCGI, and not exactly traditional SCGI? Yes, of course. But websocket over CGI is also an expansion to traditional CGI—RFC3875 says that the script must read no more than CONTENT_LENGTH bytes from stdin, yet Lighttpd has its -1 thing, and that a Document response must include a Content-Type header, yet a Content-Type header doesn’t make any sense for a 101 response. The point is that it seems like a very small, reasonable expansion. It changes the meaning of one or two header fields. It changes the strict request/response pairing over something that is already a bidirectional stream socket, into a looser use of the same transport to carry frames in either direction at any time. To me, neither of those seems extreme—I’m not proposing changing the key/value-pairs-with-NULs, nor the netstring encoding, nor the HTTP-headers-followed-by-two-CRLFs, big things that to me are most strongly characteristic of SCGI and that make it so easy for a backend application to parse.

This comment is off-putting, since you clearly have not highlighted "pretty obvious" portions of the SCGI protocol specification which are incompatible with Upgrade: websocket.

Sorry if it was worded badly; it certainly wasn’t meant to be off-putting. I also wasn’t trying to suggest that the implementation would be easy in Lighttpd; I was only speaking about the needed extensions to the specification. If I’m missing some specific part of the spec that makes this unreasonable, I’d love to know what it is, but as far as I can tell, so far I’ve addressed each point you’ve brought up, and they all seem like very small variations on SCGI, not big changes.

Have you tried extending your SCGI server to special-case an invalid length, such as CONTENT_LENGTH=-1 when HTTP_UPGRADE=websocket is present? Have you tried modifying the lighttpd code to see what might be necessary to pass HTTP_UPGRADE=websocket to the backend?

No, I haven’t written any code. I thought it made sense to start a discussion first and see if there was any point in doing so, as well as gain some consensus on what would be a sensible specification first. I’ve written an SCGI protocol library in Python and this would be an easy change to add to it (it just receives the netstring length followed by the headers and the splits them on NULs and pairs them into a dictionary, thus it doesn’t care at all whether CONTENT_LENGTH is first; it does currently reject the request if CONTENT_LENGTH is completely absent, but it only looks for that variable when it’s ready to start reading the request body, so checking for HTTP_UPGRADE first and taking a new “switch protocols, there is no request body” path would be easy). I have no idea how hard or easy it would be in Lighttpd; I’ve never looked at Lighttpd’s code before.

Correction to what I said above. lighttpd mod_cgi special-cases -1 and sends CONTENT_LENGTH=0 to the CGI. CGI requests require CONTENT_LENGTH, so a 411 may be returned if CONTENT_LENGTH is not known (and if lighttpd is configured to stream the request body).

In practice, HTTP/1.1 requests which send Upgrade: websocket do not send a request body (and instead send content-length: 0).

Sorry if it was worded badly; it certainly wasn’t meant to be off-putting. I also wasn’t trying to suggest that the implementation would be easy in Lighttpd; I was only speaking about the needed extensions to the specification. If I’m missing some specific part of the spec that makes this unreasonable, I’d love to know what it is, but as far as I can tell, so far I’ve addressed each point you’ve brought up, and they all seem like very small variations on SCGI, not big changes.

• If we're not following the standard, then anything is possible.
• It's just code.
• It's just a small matter of programming

I am not here to correct all your flawed logic and missteps, and no, I am not going to point them all out. Here's one sloppy take:

and that a Document response must include a Content-Type header, yet a Content-Type header doesn’t make any sense for a 101 response.

1xx are intermediate responses and are permitted. Your statement is incorrect and a misunderstanding of intermediate and final responses.
Yet, you make that statement with (false) confidence as if you knew what you were talking about. Great example of just that.

... so IMO that would be a possible option, though not necessarily the best one) both seem acceptable to me.
...
No, I haven’t written any code. I thought it made sense to start a discussion first and see if there was any point in doing so, as well as gain some consensus on what would be a sensible specification first.
...
I have no idea how hard or easy it would be in Lighttpd; I’ve never looked at Lighttpd’s code before.

You're not building consensus. You are clearly saying that you don't know what you're talking about and want someone else to do the work and to educate you.
Try to ask intelligent questions instead of making guesses as statements.

wiki documentation: lighttpd and WebSockets

Well, if you’ve already implemented it, then the discussion about specifications is no longer relevant. Thank you for doing that; I wasn’t expecting it, certainly not so quickly, and I’ll look forward to getting 1.7.74 and trying it out.

I do hope you'll take this advice to heart: Try to ask intelligent questions instead of making guesses as statements.