Lighttpd

Thanks for the thorough info. It looks like h2c has been cleaned up, and the SIGSEGV occurs on a NULL pointer dereference. I need to look into why...

(gdb) bt full
#0  0x00000055555668f4 in connection_state_machine_h2 (h2r=0x555570a520, con=0x555570a520) at connections.c:1209
        h2c = 0x0

So far, nothing obvious has jumped out to me as an error. Given the stack trace you provided, the connection is still in the job queue after being retired (which can happen), but the connection should not have con->request.http_version set to HTTP_VERSION_2 after being retired, and so should not reach connection_state_machine_h2()

connection_handle_response_end_state() is the only place which calls h2_retire_con(), which is the only place which sets con->h2 = NULL. (con->h2 is allocated when con->request.http_version is set to HTTP_VERSION_2.)

Further along in connection_handle_response_end_state(), request_reset() is called (either directly or through the call to connection_handle_shutdown() which calls connection_reset(), which calls request_reset()), and request_reset() changes con->request.http_version to HTTP_VERSION_UNSET.

I'll have to look more into this later. In the meantime, are you able to share your lighttpd config (lighttpd -f /etc/lighttpd/lighttpd.conf -p) so that I can see what other modules might be involved?

While the following might be a workaround (and might need further minor adjustments to be sufficient), I would like to track down why this is occurring, so the following is a bandaid, and not a solution.

--- a/src/connections.c
+++ b/src/connections.c
@@ -1205,6 +1205,7 @@ static void
 connection_state_machine_h2 (request_st * const h2r, connection * const con)
 {
     h2con * const h2c = con->h2;
+    if (NULL == h2c) return;

     if (h2c->sent_goaway <= 0
         && (chunkqueue_is_empty(con->read_queue) || h2_parse_frames(con))

Here is a guess. Would you please try this patch if you can? Perhaps ALPN negotiates h2 in a TLS module before mod_sockproxy picks up the connection.

--- a/src/mod_sockproxy.c
+++ b/src/mod_sockproxy.c
@@ -163,6 +163,7 @@ static handler_t mod_sockproxy_connection_accept(connection *con, void *p_d) {
                hctx->create_env = sockproxy_create_env_connect;
                hctx->response = chunk_buffer_acquire();
                r->http_status = -1; /*(skip HTTP processing)*/
+               r->http_version = HTTP_VERSION_UNSET;
        }

        return HANDLER_GO_ON;

It's been over a day now with the patch applied and no crash so far.
Looks promising :)
Thank you

I'll continue the testing and report back in a few days if sth changes

Yes I only used the patch to mod_sockproxy.
Unfortunately I have no core dumps. However if the patch turns out to be working, I could remove it, wait for a crash and then check this if that's desired.

Unfortunately I have no core dumps. However if the patch turns out to be working, I could remove it, wait for a crash and then check this if that's desired.

Thanks, but not necessary. The simple mod_sockproxy patch fixes something that I overlooked when adding HTTP/2 support to lighttpd.

Bad news. It just happened again with the exact same stacktrace as before. However I think the patch seems to work somehow because it took a lot longer this time.
I also inadvertently closed the debugger so I have to wait for the next crash for a core dump.

Please save the core dump. If I can access the core file and the associated lighttpd executable file, I can examine the program state outside the stack trace.

If the TLS ClientHello is received or processed after lighttpd mod_sockproxy has been initialized for the connection, I think that might explain what you are seeing. I think one answer is to avoid selecting a protocol using TLS ALPN if mod_sockproxy will be handling the connection. However, doing so will break someone using mod_sockproxy to connect to an HTTP/2 backend, and desiring to use HTTP/2. For that, an alternative patch (further below) might be better. I am leaning towards applying the second patch below, and applying a similar patch to other lighttpd TLS modules.

--- a/src/mod_openssl.c
+++ b/src/mod_openssl.c
@@ -1874,6 +1874,8 @@ mod_openssl_alpn_select_cb (SSL *ssl, const unsigned char **out, unsigned char *
     handler_ctx *hctx = (handler_ctx *) SSL_get_app_data(ssl);
     unsigned short proto;
     UNUSED(arg);
+    if (hctx->r->handler_module) /*(e.g. mod_sockproxy)*/
+        return SSL_TLSEXT_ERR_NOACK;

     for (unsigned int i = 0, n; i < inlen; i += n) {
         n = in[i++];

Alternative patch

--- a/src/mod_openssl.c
+++ b/src/mod_openssl.c
@@ -1883,7 +1883,8 @@ mod_openssl_alpn_select_cb (SSL *ssl, const unsigned char **out, unsigned char *
             if (in[i] == 'h' && in[i+1] == '2') {
                 if (!hctx->r->conf.h2proto) continue;
                 proto = MOD_OPENSSL_ALPN_H2;
-                hctx->r->http_version = HTTP_VERSION_2;
+                if (hctx->r->handler_module == NULL)/*(e.g. not mod_sockproxy)*/
+                    hctx->r->http_version = HTTP_VERSION_2;
                 break;
             }
             continue;

It crashed again. I dumped the core but I can't upload it here due to filesize restrictions. So I put it on Google Drive:
https://drive.google.com/file/d/1JmruFxcxVg92QWGT7oY51KWloz94x4Xg/view?usp=drivesdk

I'll also apply the second patch now and see if that works.

Thank you. I can see from the core that there are 4 connections, and the crash occurs on a connection being handled by mod_sockproxy before any non-TLS bytes are read. (I can't as easily look into mod_openssl.so state since you did not include that in the .tar.) Still, the core supports my hunch that the connection is negotiating h2 via ALPN, but that mod_sockproxy has already claimed the connection, so the http_version is not being reset. I think the latest patch will address your issue.

As an additional safeguard, my dev branch contains the following patch, which should (independently) avoid the issue.

--- a/src/connections.c
+++ b/src/connections.c
@@ -1378,11 +1380,11 @@ connection_state_machine_h1 (request_st * const r, connection * const con)
 void
 connection_state_machine (connection * const con)
 {
     request_st * const r = &con->request;
-    if (r->http_version == HTTP_VERSION_2)
+    if (con->h2)
         connection_state_machine_h2(r, con);
     else /* if (r->http_version <= HTTP_VERSION_1_1) */
         connection_state_machine_h1(r, con);
 }

As an aside should others find this, a short-term workaround is to disable h2 support in lighttpd, unless HTTP/2 support is critical to your needs
server.feature-flags += ("server.h2proto" => "disable")

gstrauss wrote in #note-15:

I can't as easily look into mod_openssl.so state since you did not include that in the .tar.

I added all the *.mod files to the archive. You can use the link from comment #note-14

@ultimator: The core contains sensitive info. Please change your TLS certificate(s) and revoke the current one(s), if necessary.

The mod_openssl.c patch you are now testing should address the issue. My guess is that some scanner is connecting to port 853 and trying TLS ALPN h2. Later this weekend or on Mon, I'll try to reproduce this and confirm that it is fixed with either of the mod_openssl.c patches above.

gstrauss wrote in #note-17:

@ultimator: The core contains sensitive info. Please change your TLS certificate(s) and revoke the current one(s), if necessary.

Thanks, already done.