Lighttpd

Update: it's sockproxy not proxy

ss -tupn | grep CLOSE

tcp   CLOSE-WAIT 1      0          internal-ip:853         external-ip:50282 users:(("lighttpd",pid=25677,fd=19))
tcp   CLOSE-WAIT 1      0          internal-ip:853         external-ip:54738 users:(("lighttpd",pid=25677,fd=25))
tcp   CLOSE-WAIT 1      0          internal-ip:853         external-ip:45374 users:(("lighttpd",pid=25677,fd=23))
tcp   CLOSE-WAIT 1      0          internal-ip:853         external-ip:43702 users:(("lighttpd",pid=25677,fd=17))
tcp   CLOSE-WAIT 1      0          internal-ip:853         external-ip:55522 users:(("lighttpd",pid=25677,fd=26))
tcp   CLOSE-WAIT 1      0          internal-ip:853         external-ip:36140 users:(("lighttpd",pid=25677,fd=18))
tcp   CLOSE-WAIT 1      0          internal-ip:853         external-ip:47640 users:(("lighttpd",pid=25677,fd=20))
tcp   CLOSE-WAIT 1      0          internal-ip:853         external-ip:49532 users:(("lighttpd",pid=25677,fd=16))
tcp   CLOSE-WAIT 1      0          internal-ip:853         external-ip:39818 users:(("lighttpd",pid=25677,fd=29))
tcp   CLOSE-WAIT 1      0          internal-ip:853         external-ip:42926 users:(("lighttpd",pid=25677,fd=21))
tcp   CLOSE-WAIT 1      0          internal-ip:853         external-ip:50500 users:(("lighttpd",pid=25677,fd=24))

$SERVER["socket"] == "0.0.0.0:853" {
    ssl.engine    = "enable" 

    # ssl params
    ...

    sockproxy.server = ( "" => ( ( "host" => "my-internal-host", "port" => 9000 ) ) )
}

Bisecting gave me

bc873072ba8e23d5f07c5a9b0e58bde88dada722 is the first bad commit
commit bc873072ba8e23d5f07c5a9b0e58bde88dada722
Author: Glenn Strauss <gstrauss@gluelogic.com>
Date:   Mon Dec 11 01:51:06 2023 -0500

    [core] quicker server graceful shutdown of websockets

 src/connections.c |  9 +++++++++
 src/gw_backend.c  |  4 +++-
 src/h2.c          | 23 +++++++++++++++++++++--
 3 files changed, 33 insertions(+), 3 deletions(-)

I reverted this for now and that fixes the issue.

sockproxy.server = ( "" => ( ( "host" => "my-internal-host", "port" => 9000 ) ) )

Is your backend service receiving TCP_FIN from lighttpd calling shutdown(fd, SHUT_WR)? Is your backend then calling close() on the socket?
(The backend service likely called close() since the socket is in CLOSE_WAIT state)

The socket on the backend is closed.
I did a capture on the backend until I noticed the first socket in close-wait state on the proxy host.
It's like this:

Lighttpd host  --- Backend host
             Syn -> 
                <- Syn Ack
             Ack ->
         Fin Ack ->
                <- Ack
                <- Fin Ack
             Ack ->
------------- 5 minutes later -------------------
             Syn ->
                <- Syn Ack
             Rst ->

Would you please test if this addresses your issue?

--- a/src/gw_backend.c
+++ b/src/gw_backend.c
@@ -1924,7 +1924,11 @@ static void gw_conditional_tcp_fin(gw_handler_ctx * const hctx, request_st * con
     if (r->http_version <= HTTP_VERSION_1_1)
         r->con->is_readable = 0;
     shutdown(hctx->fd, SHUT_WR);
-    fdevent_fdnode_event_clr(hctx->ev, hctx->fdn, FDEVENT_OUT);
+    int events = fdevent_fdnode_interest(hctx->fdn) & ~FDEVENT_OUT;
+    if (!(events & FDEVENT_IN))
+        hctx->read_ts = log_monotonic_secs;
+    events |= FDEVENT_IN|FDEVENT_RDHUP;
+    fdevent_fdnode_event_set(hctx->ev, hctx->fdn, events);
 }

 static handler_t gw_write_refill_wb(gw_handler_ctx * const hctx, request_st * const r) {

gstrauss wrote in #note-7:

Would you please test if this addresses your issue?
[...]

Does not fix the issue unfortunately. Same behavior as before.

Thanks for testing. I'll see if I can find some time to reproduce this. If not today, it may have to wait for next weekend. Sorry.

I was unable to reproduce the issue using a trivial mod_sockproxy backend.

Would you be able to get an strace() of lighttpd while it handles a request to one of your mod_sockproxy backends?

I am still unable to reproduce this.

I have been reviewing commit bc873072

How frequently do you restart lighttpd? Are you sending SIGUSR1 to lighttpd for graceful restart?
Are the requests that are not closing the sockets HTTP/1.x or HTTP/2?

What is the rest of your lighttpd.conf, such as timeout settings?
What is the behavior of your client? Is it actively uploading data?
What is the behavior of your backend behind mod_sockproxy? Is it actively sending data?

Here is a modified patch to the patch above

--- a/src/gw_backend.c
+++ b/src/gw_backend.c
@@ -1920,11 +1920,17 @@ static void gw_conditional_tcp_fin(gw_handler_ctx * const hctx, request_st * con

     /* propagate shutdown SHUT_WR to backend if TCP half-close on con->fd */
     r->conf.stream_request_body |= FDEVENT_STREAM_REQUEST_BACKEND_SHUT_WR;
-    r->conf.stream_request_body &= ~FDEVENT_STREAM_REQUEST_POLLIN;
+    r->conf.stream_request_body &=
+      ~(FDEVENT_STREAM_REQUEST_BUFMIN|FDEVENT_STREAM_REQUEST_POLLIN);
     if (r->http_version <= HTTP_VERSION_1_1)
         r->con->is_readable = 0;
     shutdown(hctx->fd, SHUT_WR);
-    fdevent_fdnode_event_clr(hctx->ev, hctx->fdn, FDEVENT_OUT);
+    int events = fdevent_fdnode_interest(hctx->fdn) & ~FDEVENT_OUT;
+    if (!(events & FDEVENT_IN)) {
+        hctx->read_ts = log_monotonic_secs;
+        events |= FDEVENT_IN|FDEVENT_RDHUP;
+    }
+    fdevent_fdnode_event_set(hctx->ev, hctx->fdn, events);
 }

 static handler_t gw_write_refill_wb(gw_handler_ctx * const hctx, request_st * const r) {

Awaiting feedback.

Please confirm that the issue does not occur with lighttpd 1.4.75 and with the following set to hctx->host->tcp_fin_propagate = 0;.

--- a/src/gw_backend.c
+++ b/src/gw_backend.c
@@ -1852,7 +1852,7 @@ void gw_set_transparent(gw_handler_ctx *hctx) {
             /*(error, but not critical)*/
         }
     }
-    hctx->host->tcp_fin_propagate = 1; /*(force setting enabled for host)*/
+    hctx->host->tcp_fin_propagate = 0; /*(test: force disabled)*/
     hctx->wb_reqlen = -1;
     gw_set_state(hctx, GW_STATE_WRITE);
 }

Please try this patch by itself:

--- a/src/gw_backend.c
+++ b/src/gw_backend.c
@@ -1915,6 +1915,8 @@ static void gw_conditional_tcp_fin(gw_handler_ctx * const hctx, request_st * con
     if (!chunkqueue_is_empty(&hctx->wb))return;
     if (!hctx->host->tcp_fin_propagate) return;
     if (hctx->gw_mode == GW_AUTHORIZER) return;
+    if (hctx->state == GW_STATE_CONNECT_DELAYED)
+        return;
     if (r->conf.stream_request_body & FDEVENT_STREAM_REQUEST_BACKEND_SHUT_WR)
         return;

Sorry for the absensce.

How frequently do you restart lighttpd?

Usually it's running for maybe 2-3 weeks until I restart due to updates.

Are you sending SIGUSR1 to lighttpd for graceful restart?

I'm using the gentoo openrc service of lighttpd. The stop command is

start-stop-daemon --stop --quiet --pidfile "${LIGHTTPD_PID}" 

There is also a graceful stop that does the following but I don't use it

start-stop-daemon --quiet --pidfile "${LIGHTTPD_PID}" --signal INT  

Are the requests that are not closing the sockets HTTP/1.x or HTTP/2?

I'll check this.

What is the rest of your lighttpd.conf, such as timeout settings?

I have no timeouts specified so they should all be set to default values.
I enabled HTTP/2 with

server.feature-flags = (                                                                                                  
    "server.h2proto" => "enable",                                                                                                   
    "server.h2c"     => "enable",                                                                                                   
)

What is the behavior of your client? Is it actively uploading data?
What is the behavior of your backend behind mod_sockproxy? Is it actively sending data?

I can't really tell. I'm basically serving DOT but there is no client yet that uses it (at least none I'm aware of). The backend is DNS resolver. Lighttpd strips the TLS and forwards to the resolver. A proper DOT client would encapsulate a DNS packet but the transmitted packages do not contain data, only tcp metadata. I don't even know if there really is a client or if it's just some check for the upstream server by lighttpd itself. I can provide the capture file if needed.

I'll test the patches now.

Patch from #note-12 does not work. Same behavior as before.
Patch from #note-14 with

hctx->host->tcp_fin_propagate = 0;

Tested with lighttpd version 1.4.75.