Lighttpd

When disabling HTTP/2 there is no problem with Chrome.

When using Chrome 92 my fcgi backend is not called.

The logs suggest otherwise. The logs you shared show
2021-08-10 13:20:28: gw_backend.c.2571) handling it in mod_gw
but do not show responses for the chrome scenario with multiple requests.

This sounds like an issue with your FastCGI backend. However it is hard to say since you have not read How to get help and you have not provided a lighttpd.conf or any hints how to reproduce besides "multiple PUT requests".

The mod_fastcgi doc has a section near the top of the page about debugging.
Try fastcgi.debug = 3

Look into what your fastcgi backed is doing differently between requests from Chrome and requests from Firefox.

Sorry about this. Setting fastcgi.debug = 3 suggests that fcgi is called. The bug have to be in my system.

lighttpd HTTP/2 support limits SETTINGS_MAX_CONCURRENT_STREAMS to 8 and communicates this in the initial SETTINGS sent by the server. Your client should recover if additional streams are rejected with REFUSED_STREAM.

Are you getting REFUSED_STREAM when your client sends too many requests before any in the 8 already-in-flight requests finish?

This have to be a bug in Chrome. In h2_recv_headers, when the 9th header frame is received, h2c->sent_settings is set so it does not send REFUSED_STREAM but silently ignores the error.

When testing I have seen occasionally correct behavior of Chrome, sending 8 header frames + 8 data frames and then 1 header frame and 1 data frame. Then all is working as expected.

I have to try to report a chrome bug.

If you do, please post a link to the issue, so that I can track it, too.

RFC 7540 (on which Google participated) sets SETTINGS_MAX_CONCURRENT_STREAMS to 100 by default, but permits smaller values -- including 0 for short periods of time. lighttpd limits SETTINGS_MAX_CONCURRENT_STREAMS to 8 to put a constraint on memory and other resource use, and I believe that lighttpd is doing so in a way that respects RFC 7540 Hypertext Transfer Protocol Version 2 (HTTP/2). lighttpd attempts to process all the data initially received before lighttpd sends the initial SETTINGS frame from the server, but after that presumes that the client should have received the initial SETTINGS frame from the server.

You can see from the comment in h2_recv_headers that I had tested this with h2load -n 100 -m 100. However, I tested on localhost. With multiple systems, there is a windows with a chance of latency and multiple packets delayed/dropped/resent where the client has sent multiple packets, but lighttpd has only received some of them before lighttpd sends the initial SETTINGS frame from the server, and then more time before the client might receive and process the SETTINGS frame from the server.

I am open to ideas how lighttpd can do this better as long as it does not open lighttpd to additional resource-based attacks.

I tried some changes in lighttpd yesterday and was able to make Chrome work without errors. One of the changes was to always send REFUSED_STREAM, similar to your suggestion. Some more changes was needed. I´m working on a description of the changes and will post this later today.

I´m working on a description of the changes and will post this later today.

Thanks. I'll look for it.

One of the changes was to always send REFUSED_STREAM

At the moment, I don't recall exactly why, but there was a reason why I chose not to always send REFUSED_STREAM, or else I would not have written that extra code. Maybe I was handling it as a protocol violation after SETTINGS_MAX_CONCURRENT_STREAMS should have been processed by the client.

.

Unrelated to this issue, but since you helpfully provided your lighttpd.conf:

Given your cipher list, restricting to TLS1.2 or later, and lighttpd 1.4.59, the following lines are no longer necessary:
ssl.ca-file = "/etc/lighttpd/ssl/.../chain.pem"
ssl.dh-file = "/etc/lighttpd/ssl/dh2048.pem"

If you support mobile devices, you may want to add CHACHA20 to your cipher list, and then replace the line:
ssl.honor-cipher-order = "enable"
with
ssl.openssl.ssl-conf-cmd += ("Options" => "-ServerPreference")

For clearer syntax about supported TLS versions, you might use the syntax
ssl.openssl.ssl-conf-cmd += ("MinProtocol" => "TLSv1.2") to replace "Protocol" => "-TLSv1.1, -TLSv1, -SSLv3".
MinProtocol was added in openssl 1.1.0 and you're using 1.1.x in Ubuntu 18.04

Further doc at lighttpd TLS docs

I tried your purposed change in h2_recv_headers now and it is working. My description of the changes tried yesterday is no longer needed.

To make this work you need one more change. h2_recv_data must not send PROTOCOL_ERROR when receiving data frames on a prior REFUSED_STREAM stream but "ignore" it. I just commented out the lines in h2_recv_data:

if (h2c->h2_cid < id || (!h2c->sent_goaway && 0 != alen))
h2_send_goaway_e(con, H2_E_PROTOCOL_ERROR);

to make it work. You have probably a better solution than commenting this out, that I can test.

To make this work you need one more change. h2_recv_data must not send PROTOCOL_ERROR when receiving data frames on a prior REFUSED_STREAM stream but "ignore" it.

@stenvaag see #3078 and commit 81d18a8e which is on lighttpd git master and will be part of the next lighttpd release.

(Note: 81d18a8e uses log_monotonic_secs, but you can substitute log_epoch_secs if you want to test the patch against the lighttpd 1.4.59 release)

I applied a new patch to the (current) HEAD of my development branch. It is slightly different from the two-line patch I posted further up.
https://git.lighttpd.net/lighttpd/lighttpd1.4/src/branch/personal/gstrauss/master
Some of this code is not yet on lighttpd git master, but the intention is that it all will be in the next month or two.
Also, please be aware that I often rewrite git history of my dev branch during development.
Usual caveats apply: this is a development branch: ok for testing, but at your own risk anywhere else.

I made a test with the first proposed change in #3078. And this worked.

Thanks for testing. FYI: the patch you tested from #3078 is not the patch which ended up being committed, but the idea and result should be the same.

@stenvaag
It appears that my patch to handle the burst of PUT requests on initial connection has caused a regression when Firefox sends a burst of (non-PUT) requests on initial connection. I posted an updated patch in #3100 which hopefully addresses both scenarios.

I downloaded and tested https://git.lighttpd.net/lighttpd/lighttpd1.4/src/branch/master 40 minutes ago. "My" error is still fixed, but I get a random HTTP2 protocol error from Chrome.

I downloaded and tested https://git.lighttpd.net/lighttpd/lighttpd1.4/src/branch/master 40 minutes ago. "My" error is still fixed, but I get a random HTTP2 protocol error from Chrome.

Would you please provide some additional details? What kind of requests are now resulting in random HTTP/2 protocol errors from Chrome? With the current patches on lighttpd git master, the behavior "should" (famous last words) be the same as in lighttpd 1.4.59, except when the connecting client sends more than 8 requests with request bodies (e.g. PUT) before the request bodies for the first 8 have been sent.

It is not easy to get the error. I have to run my application multiple times (reload the page) to possible get the error. I got the error now when navigating to a page (inside my application) that in this case have 12 GETS to different URL's (all that resulting in a call to my fcgi backend). It is the first one that fails. The second has the SSL setup delay. All, but the first one, succeeds. The error from Chrome is "net::ERR_HTTP2_PROTOCOL_ERROR".

Would you test with lighttpd 1.4.59 to check if you saw this behavior with lighttpd 1.4.59, too? I have tested with h2load, which IIRC sends 100 GET requests upon the initial connection.

When this occur my fcgi backend reports that first byte received is not 1 (FCGI_VERSION_1), but a random number. This could be a error in my fcgi backend, but after a decade++, this error only appeared in 1.4.59 when the http/2 protocol was activated.

lighttpd mod_fastcgi always assigns header->version = FCGI_VERSION_1; when creating FCGI headers.
https://git.lighttpd.net/lighttpd/lighttpd1.4/src/branch/master/src/mod_fastcgi.c#L181

If you're seeing something different in the FCGI application, then something might be getting corrupted somewhere (either in lighttpd or in your app, not sure which).

This could be a error in my fcgi backend, but after a decade++, this error only appeared in 1.4.59 when the http/2 protocol was activated.

lighttpd HTTP/2 code currently is used only between lighttpd and the client, not between lighttpd and backends. Use of HTTP/2 can enable faster requests, so maybe this is exposing a race condition that has been present before. ...My next step is to run a load test on a FastCGI backend and see how things go...

FYI: I see 0 failures with h2load running 20000+ request per second against a FastCGI backend written in C (lighttpd tests/fcgi-responder). I tested using h2load against both http and https targets, with lighttpd tests/fcgi-responder handling the requests.

The error from Chrome is "net::ERR_HTTP2_PROTOCOL_ERROR".

BTW, it still concerns me that you're seeing that. If the backend FastCGI fails, lighttpd should send an HTTP response for the error handling the request, not a connection error.

this error only appeared in 1.4.59 when the http/2 protocol was activated.

Perhaps we should treat this as a separate issue from this one ("Chrome 92, HTTP/2, fcgi, mutiple puts no response"). Would you mind opening a new bug, and continuing to try to help me to reproduce these observations of yours? (It could be one or more issues.)

Yes, I will open a new bug. (I was out of office, but will now look into this.)

I tried to get the error with 1.4.59 with HTTP/2 enabled, but with no "luck".