Lighttpd

I'll try to set up an environment later today to test this.

Are there any errors from mod_openssl in your lighttpd error.log?
Your example above indicates that lighttpd is sending a certificate request to the client.
Are the clients sending a certificate back as part of the TLS negotiation?
Try testing with ssl.verifyclient.enforce = "enable"
What is the version of openssl libraries installed on your system?
(Aside: please prefer to test with the latest lighttpd, lighttpd 1.4.59, instead of lighttpd 1.4.57, as there are bugs that have been fixed between lighttpd 1.4.57 and lighttpd 1.4.59)

In lighttpd 1.4.56, mod_openssl certificate management was rewritten to newer OpenSSL interfaces. It is recommended that ssl.pemfile contain the full certificate chain for the server certificate. ssl.ca-file should contain the CA for the client certificates (which may or may not be different from the server certificate CA), or ssl.ca-file should contain the chain of CAs if an intermediate CA signed the client certificates. If an intermediate CA signed client certificates, ssl.ca-dn-file should contain only the CA which signed the client certificates, as the Subject of these certificates are sent to client as a hint for client certificate selection.

There should be trace in your lighttpd error.log.

By any chance, are you using the CA used to sign your client certs as the web server certificate? Or is the web server certificate independent, e.g. a separate certificate signed by the CA?

vesions are
libssl-dev:armhf 1.1.1d-0+deb10u5+rpt1
libssl1.1:armhf 1.1.1d-0+deb10u5+rpt1

I have tried ssl.verifyclient.enforce, it changes nothing.

error.log only shows following, no mod_ssl, the requests don't even fail
FastCGI-stderr:PHP message: PHP Notice: Undefined index: SSL_CLIENT_S_DN_CN in /var/www/example.com/mgmt/login.php on line 7
except when using firefox with ssl.verifyclient.enforce and the request fails
2021-03-13 00:57:16: mod_openssl.c.3095) SSL: 1 error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate

client certs are created and signed using easy rsa with my own CA (directly, no intermediates). Webserver's certificate is from letsencrypt.

ssl.pemfile contains letsencrypt certificate and private key
ssl.ca-file contains letsencrypt chain (1 cert) and my own root CA cert
I will try moving the LE chain into pemfile and leaving only CA in ca-file

Are the clients sending a certificate back as part of the TLS negotiation?

I don't know how to check, but I suppose they are not, since they don't show the dialog and I get Undefined index: SSL_CLIENT_S_DN_CN in PHP

PS:
tested (also) using following browsers:
Windows 10 20H2: Chrome (v88), IE (v20H2), (chromium) MS Edge (v89), Firefox (v86)
Raspbian 10: openssl s_client (OpenSSL 1.1.1d)
Android 11: Chrome (v88)
iPad OS 14.4.1: Safari, Chrome (v87)

chrome on ipad does not work even with 1.4.55, but safari does
sometimes browsers on windows work for a few minutes after after restarting lighttpd, but mobile browsers don't
firefox doesn't work even with 1.4.55, but I haven't imported the cert into firefox

having chain+cert+key in pemfile and ca_cert in ca-file makes no apparent difference in both versions

With Let's Encrypt, it is recommended that you use the files as provided by Let's Encrypt:

ssl.privkey = "/etc/lighttpd/certs/www.example.com/privkey.pem" 
ssl.pemfile = "/etc/lighttpd/certs/www.example.com/fullchain.pem" 

As you are doing, the CA cert (used to sign client certificates) goes in ssl.ca-file

You might want to comment out ssl.verifyclient.depth = 2 during your testing, or increase it to 4 to see if that makes a difference.

I have tested that certificate verification works for me using curl. In doing so, I found that if the server certificate is self-issued and ssl.ca-file is specified, then lighttpd mod_openssl would fail to build the certificate chain, and would issue trace in the lighttpd error log. I have a small patch for that, but that does not seem to be your issue. Once I added the patch to lighttpd for self-issued server certificate (along with ssl.ca-file being specified), I can successfully use mod_openssl with client certificate verification.

If you have an inclination, you can build lighttpd with alternative TLS libraries. In addition to mod_openssl, you can test with one or more of mod_gnutls, mod_mbedtls, mod_nss, and mod_wolfssl.

A client-side error is indicated if lighttpd 1.4.55 and lighttpd 1.4.59 both send identical certificate requests

Acceptable client certificate CA names
CN = example.com

when using firefox with ssl.verifyclient.enforce and the request fails

2021-03-13 00:57:16: mod_openssl.c.3095) SSL: 1 error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate
That client-side error would be:

firefox doesn't work even with 1.4.55, but I haven't imported the cert into firefox

.

This sounds like the a proper config:

having chain+cert+key in pemfile and ca_cert in ca-file makes no apparent difference in both versions

It is strange that you are reporting things work sometimes, but not other times. Have you tried flushing your browser caches?

Let's try to narrow down where the problem might be. If you configure the client to provide the cert, is the certificate accepted and do things work?

If you take the client cert and try it with cURL, does it work? You need the cert in PKCS#12 format.

openssl pkcs12 -export -clcerts -in client_cert.pem -inkey client_key.pem -out client.p12
curl --cert client.p12 --cert-type p12 https://login.example.com/

Have you tried openssl s_client with the -cert option?

When not trying to solve the issue (eg testing configurations and versions) I keep running .55 as a service.
So when I go testing, I stop the service, then run lighttpd in console (SSH).

In desktop browsers (edge, chrome), I use mostly incognito to try login.* (close and reopen between retries), so I guess cache is not relevant
Today, the desktop browsers are almost random, sometimes they ask for certificate, other times they don't.
Safari on iPad and chrome on android seem most "reliable", they do not send the cert on .59 and do on .55.
For clarity, safari on iPad never asks to select certificate, it either sends if requested or doesn't.

curl with cert works, certificate gets sent.

ssl.verifyclient.depth=4 does not help

After an hour of testing, I have at least found something.
For reference, my normal login flow is:
1. open example.com/login
2. click button for certificate authentication (form submit)
3. redirect to login.example.com/login
4. validate certificate and redirect back to example.com

I can navigate directly to login.example.com/login

What I found is:
When I navigate directly to login.example.com, desktop and mobile browsers ask for the cert on first or second load.
Subsequent normal login flow works fine (probably because they remember the cert).
However when I use normal login flow, browsers do not ask for certificate.

So I have started up Wireshark and looked into the packets (with SSLKEYLOGFILE).
I have found that with .55, there are multiple separate connections (as expected), some of them with Certificate Request in Server Hello packet
With .59, there's only one (without Certificate Request)
Then I noticed another difference: .55 uses HTTP and .59 uses HTTP2 (HTTP2 was added in .56)
So I suppose modern browsers reuse connection to one host even when using different hostnames.

My solution for now is to disable HTTP2.
Sorry for bothering you and thanks for your help.

Thank you for your thorough troubleshooting and for your effort.

Are you saying that with lighttpd 1.4.59, server.feature-flags += ("server.h2proto" => "disable") makes things work? Some of the time or all of the time? Do things work better or the same as with lighttpd 1.4.55? (You noted that some browsers did not work all the time with lighttpd 1.4.55)

Have you tried setting "Connection: close" when the client browses to an unrestricted page which redirects to login.example.com/login, so that lighttpd closes the connection, causing the browser to redirect, reconnect, and renegotiate TLS with the server certificate request?

Hmmm. After looking through the code again, I see that lighttpd (intentionally) does not provide a means for a backend to trigger HTTP/2 connection to close. Still, if you are disabling HTTP/2 in lighttpd, then to make HTTP/1.1 client certificate auth more reliable for your clients, you might still test having your backend send Connection: close when you want the client to reconnect and renegotiate TLS with a site configured to send a certificate request to the client from the server.

yes, setting server.feature-flags += ("server.h2proto" => "disable") "fixes" the issue, all the time. AFAIR everything worked fine on .55, the issues started on .57. You probably mean this, from my first post:

When using 1.4.57, 98% of requests, browser does not show the dialog and no client certificate is transmitted.

I have just tried the Connection: close header, but it did not show up in in browser's developer tools. Then I found out it is not supported in HTTP2
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Connection

It is possible to issue 421 Misdirected Request, which was introduced to notify the browser to reconnect for that specific request.
I am not sure how to properly identify individual HTTP connections in PHP (so I can tell when the browser is requesting using the same connection).
First idea is saving $_SERVER['REMOTE_PORT'] in session, but that relies on browser not using the same port number.

The modified login flow:
1. open example.com/login
2. click button for certificate authentication (form submit)
3. save port number and redirect to login.example.com/login
4. compare port number and optionally send 421 Misdirected Request
4.5 browser reconnects
5. validate certificate and redirect back to example.com

For now I'm fine with staying on HTTP1

forgot to add this:
https://httpwg.org/specs/rfc7540.html#reuse

Right. 421 Misdirected Request is the solution for HTTP/2
Try this:
1. open example.com/login
2. click button for certificate authentication (form submit)
3. save port number and redirect to login.example.com/login-cert-check
4. If certificate not provided, send 421 Misdirected Request
4.5 browser reconnects
5. validate certificate and redirect back to example.com

sometimes I login from a device with no certificate, which would result in redirect loop. I could keep track of redirect count and/or do some more (slightly) complex logic, but I don't want to do that for now and will leave it on http1.1

login.example.com/login-cert-check-2 could be a fallback for loop detection. You could set a parameter in the query string, or temporarily set a cookie with a short expiration to indicate that you already sent 421 Misdirected Request