Lighttpd

It looks like:

• lighttpd silently(!!) ignores "older" certificates (Public Key Algorithm: rsaEncryption, beginning with "-----BEGIN PRIVATE KEY-----") for vhosts, when the "main" certificate is a "newer type" (Public Key Algorithm: id-ecPublicKey, beginning with "-----BEGIN EC PRIVATE KEY-----").
• If I change the main certificate to a rsaEncryption-type, everything works well.

Example:
- server containing 3 domains: domain-ec.example, domain-rsa1.example, domain-rsa2.example
- domain-ec.example uses a id-ecPublicKey-certificate, domain-rsa*.example use rsaEncryption-certificates
- There's a $HTTP["host"] =~ ".*domain-ec.example" { like entry for each domain, each containing ssl.pemfile.
- Now: Depending on what's written in the $SERVER["socket"] ":443", the server deliveres different certificates.

- $SERVER["socket"]  ":443": ssl.pemfile = ".../domain-ec.example/full.pem" 
    - domain-ec.example:   uses domain-ec.example-pemfile
    - domain-rsa1.example: uses domain-ec.example-pemfile (!)
    - domain-rsa2.example: uses domain-ec.example-pemfile (!)

- $SERVER["socket"] == ":443": ssl.pemfile = ".../domain-rsa1.example/full.pem" 
    - domain-ec.example:   uses domain-ec.example-pemfile
    - domain-rsa1.example: uses domain-rsa1.example-pemfile
    - domain-rsa2.example: uses domain-rsa2.example-pemfile

mod_simple_vhost: I can understand your desire for such a feature, but this has never worked the way you suggest and has never been documented to work the way you suggest.

Unless you have a huge number of domains, you might consider generating your lighttpd config instead of using mod_simple_vhost. If you do have a huge number of domains, then you might consider leveraging SAN in certs.

[Edit] You're not using mod_simple_vhost. Why did you say simple-vhost in your report?

lighttpd supports SNI. Similar to advice given in some of your historical posts, please consider using the latest version of lighttpd. More importantly, you should upgrade your openssl libraries, as that is more likely the source of the behavior you are seeing.

According to https://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_SSL, that's exactly the way to use SSL/TLS with multiple domains.

So, is it intentional that lighttpd silently (!) -- without any error or warning -- ignores domain-specific rsaEncryption-certificates if the default ssl.pemfile is a id-ecPublicKey-certificate?

If yes: Why?
If no: How can this be fixed?

[Edit] You're not using mod_simple_vhost. Why did you say simple-vhost in your report?

I am using mod_simple_vhost. I don't know if this has anything to do with this problem or with SNI (since I don't know the lighttpd-sourcecode), but I am definitely using it here.

lighttpd supports SNI. Similar to advice given in some of your historical posts, please consider using the latest version of lighttpd. More importantly, you should upgrade your openssl libraries, as that is more likely the source of the behavior you are seeing.

I am using lighttpd 1.4.55, and according to https://redmine.lighttpd.net/projects/lighttpd/wiki, that's the latest release; probably with openssl 1.1.1f. So, upgrading won't help.

lighttpd is not openssl. lighttpd uses openssl (or whatever TLS libraries lighttpd was built with, e.g. LibreSSL, BoringSSL, WolfSSL, etc)

If you comment out ssl.cipher-list then lighttpd defaults to ssl.cipher-list = "HIGH". You might temporarily test with ssl.cipher-list = "LOW" Similarly, you might test inverting ssl.honor-cipher-order

.

FYI: mod_simple_vhost is independent from mod_openssl. mod_simple_vhost does not configure SSL certificates. mod_openssl does, so you have to configure ssl.pemfile in lighttpd.conf for SNI in $HTTP["host"] conditions, which is what it looks like you are doing.

.

Actually, ssl.cipher-list = "LOW" won't work for you, as those ciphers were removed in openssl 1.1.0.
https://www.openssl.org/docs/man1.1.1/man1/ciphers.html
Try ssl.cipher-list = "MEDIUM"

If certificates for your hosts are of different types from that of the default certificate for the socket, then the certificate types for the hosts should be listed in ssl.cipher-list before the certificate type of the default certificate for the socket.

This is the default (truncated) order on my system, and I suspect openssl ciphers will show similar ordering on your system

ssl.cipher-list = "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384" 

but try the following for your desire to prefer the RSA certs for your SNI hosts before the EC cert you defined as the default for the socket:

ssl.cipher-list = "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384" 

For more details, see https://redmine.lighttpd.net/issues/2842?#note-18