Project

General

Profile

Actions

Feature #2842

closed

Lighttpd Returns Wrong Cert In Multi-cert Set-up

Added by cnguyen over 6 years ago. Updated over 3 years ago.

Status:
Fixed
Priority:
Normal
Category:
TLS
Target version:
ASK QUESTIONS IN Forums:
No

Description

lighttpd (v. 1.4.45) running on freebsd 11

  • Set up to use multiple SSL certs: a default set, and one used for a particular hostname sent in the SNI extension.
  • Certs set (i.e., pemfile and ca-file) can be RSA, ECC, or a mix of both.
  • All combinations work except for the case of the default set being RSA and the set used for the hostname being ECC.
  • In this case, even though the client sends the right SNI, lighttpd will always return the RSA one. As such, the TLS/SSL handshake fails.
  • Note: RSA/RSA, ECC/ECC and ECC/RSA combination work as expected. The default one is returned for most connections. The one associated with the hostname in the SNI extension is returned if extension is included.

Files

lighttpd.log1 (16.7 KB) lighttpd.log1 combined configuration and log starr, 2020-05-13 17:33
certs.zip (4.25 KB) certs.zip sample certificates starr, 2020-05-15 15:20

Related issues 1 (0 open1 closed)

Has duplicate Bug #3009: lighttpd uses wrong pem-fileDuplicateActions
Actions #1

Updated by gstrauss over 6 years ago

Would you please attach your configs (lighttpd -p -f /etc/lighttpd/lighttpd.conf) and describe the contents of the ssl.pemfile and ssl.ca-file, but please do not share the secure files?

Actions #2

Updated by cnguyen over 6 years ago

Unfortunately this happened several months ago and we've decided not to support clients that use SNI extensions. This is partly due to the fact that not all the clients we support can be configured to use SNI extensions. So we no longer have the relevant conf file. We're going through our internal lists of issues for resolution and noticed that this has an external dependency. So we decided to raise this issue so that you are aware of it.

Since it was an experiment, we used a default/stock lighttpd configuration using a socket conditional as well as host conditionals within the socket conditional as described by your documentation. https://redmine.lighttpd.net/projects/1/wiki/docs_ssl#Server-Name-Indication-SNI. We used openssl to generate self-signed CA certs as well as the server cert (signed by these "root" CAs).

As stated previously, when the socket conditional/default certs are ECC type, then the host conditional certs can be ECC type or RSA type. If a client sends an SNI extension that matches the host conditional, then the associated cert will be returned. Otherwise, the socket conditional cert is returned.

But if the socket conditional certs are RSA, then the host conditional certs must also be of RSA type for the SNI extension matching to work correctly. If it is ECC type, then the socket conditional cert (of type RSA) is always returned, even though the client sent an SNI extension that matched the host conditional. Obviously, this would not match the client configuration and will fail.

Note that the socket conditional and host conditional certs always use different "root" CAs (even if they are both RSA types or ECC types).

Actions #3

Updated by gstrauss about 6 years ago

  • Status changed from New to Need Feedback

lighttpd accepts both types of certs as long as the version of openssl accepts them. There might have been something specific to your configuration, potentially related to the contents of the CA cert info. Perhaps #2692 which allows setting explicit SSL server certificate chain, part of lighttpd 1.4.48 already provides a means to address the issue you were seeing.

Related, though not the specific issue you reported:

Can I use both RSA and ECC certificates in apache?
https://serverfault.com/questions/665296/can-i-use-both-rsa-and-ecc-certificates-in-apache#704692

Does a CA need to have the same type of key as the certificates it is signing? RSA / Elliptic Curve (EC/ECDH/ECDSA)
https://security.stackexchange.com/questions/14207/does-a-ca-need-to-have-the-same-type-of-key-as-the-certificates-it-is-signing-r

Actions #4

Updated by cnguyen about 6 years ago

Thanks for the response. If I have some free cycles I'll try to reproduce the relevant certs and attach them here. However, I want to make clear that my lightttpd setup can accept both types of certs (RSA and ECC) without issues.

The problem occurs if I have an RSA one (with its RSA root CA) as the default. And an ECC one (with its ECC root CA) for a specific hostname (say, foobar). The RSA one is always returned even if a client specifies "foobar" in the SNI. So authentication fails because the "foobar" client is expecting ECC certs.

If I reverse the order: ECC one (with ECC root CA) for default, and RSA one (with RSA root CA) for host "barfoo", then an SNI with "barfoo" will return the ECC, and no SNI returns RSA, as expected. So correct certs are always returned and authentication proceeds.

Actions #5

Updated by gstrauss about 6 years ago

FYI: I did a quick review of the code in mod_openssl. When SNI is provided, the callback set by SSL_CTX_set_tlsext_servername_callback() is called. This sets the URI authority to that provided in SNI and re-runs configuration to select certs. The code then calls SSL_use_certificate() and SSL_use_PrivateKey() using the values obtained from ssl.pemfile configured for the SNI host. There is nothing special here to distinguish RSA or ECC certificates. I do not think that anything special needs to be done beyond this, but please correct me if I am mistaken in my understanding on how to use the openssl API here.

(Aside: I think you reversed what you meant to say in "If I reverse the order...")

I am hoping that #2692, with patch part of lighttpd 1.4.48 and allows setting explicit SSL server certificate chain already provides a means to address the issue you were seeing.

Actions #6

Updated by gstrauss almost 6 years ago

  • Status changed from Need Feedback to Missing Feedback

Marking as Missing Feedback.

I am still watching this, so if you are able to reproduce the issue, please post about it and we'll take a look.

Please see my notes above about using more recent versions of openssl, and specifying the certificate chain with lighttpd 1.4.48 and later.

Please also check your ssl.cipher-list ordering and try using ssl.honor-cipher-order = "enable"

Actions #7

Updated by starr almost 4 years ago

I'm having the same problem in my system and can assist reproducing it.
I'm using a Yocto "sumo" build, lighttpd 1.4.51 and openssl 1.0.2p.
I have confirmed that openssl was built with "enable-tlsext".

My config (attached below) has two certificates and two HOST["host"] rules. "server1" is a self-signed RSA certificate, "server2" is a self-signed EC certificate. If the default certificate (when neither HOST rule matches) is "server2" or any other EC certificate, the configuration works as expected. If the default certificate is "server1" or any other RSA certificate, the configuration uses that certificate rather than the configured "server2" certificate.

The attached file contains my failing configuration and a combined log from lighttpd and the openssl s_client commands I used for testing.

Actions #8

Updated by starr almost 4 years ago

Same result with 1.4.55:


lighttpd/1.4.55 (ssl) - a light and fast webserver

Event Handlers:

        + select (generic)
        + poll (Unix)
        + epoll (Linux)
        - /dev/poll (Solaris)
        - eventports (Solaris)
        - kqueue (FreeBSD)
        - libev (generic)

Network handler:

        + linux-sendfile
        - freebsd-sendfile
        - darwin-sendfile
        - solaris-sendfilev
        + writev
        + write
        - mmap support

Features:

        + IPv6 support
        + zlib support
        - bzip2 support
        + crypt support
        + SSL support
        + PCRE support
        - MySQL support
        - PgSQL support
        - DBI support
        - Kerberos support
        - LDAP support
        - PAM support
        - memcached support
        - FAM support
        - LUA support
        - xml support
        - SQLite support
        - GDBM support

Actions #9

Updated by gstrauss almost 4 years ago

  • Status changed from Missing Feedback to Reopened
  • ASK QUESTIONS IN Forums set to No

Thanks for the additional information. I'll try to take a closer look today or tomorrow.

Actions #10

Updated by gstrauss almost 4 years ago

  • Has duplicate Bug #3009: lighttpd uses wrong pem-file added
Actions #11

Updated by starr almost 4 years ago

I've confirmed the same behavior with lighttpd 1.4.55 and openssl 1.0.2u. Sorry, I'm not able to upgrade to openssl 1.1.x to test on this system but at least they're the most current releases on their respective branches.

I agree that Bug #3009 is a duplicate. I'm not using vhosts, the RSA and EC private keys I'm using are both in the same format (----- BEGIN PRIVATE KEY -----) and s_client can establish a secure connection (compatible cipher) using either certificate/key, so the other concerns raised in that issue don't apply here.

I can supply copies of my test certificates and keys on request.

Actions #12

Updated by gstrauss almost 4 years ago

I can supply copies of my test certificates and keys on request.

Yes, please. That would help get me going on setting up a test environment more quickly. (Sorry that I didn't get to it today.)

Actions #13

Updated by starr almost 4 years ago

You can use these for testing. You'll have to concatenate the certificates with the keys and change the file names to match the config file I submitted earlier this week.
S Tarr

Actions #14

Updated by gstrauss almost 4 years ago

I was not able to reproduce with my initial attempt, using tip of my development branch and OpenSSL 1.1.1g, nor after building the lighttpd-1.4.55 tag.

I took your test certs (thank you) and put them in /dev/shm/certs/*. Here is the config I used:

server.document-root = "/dev/shm/www" 
server.bind = "127.0.0.1" 
server.port = 8080
mimetype.assign = (".txt" => "text/plain", ".html" => "text/html" )

server.modules += ("mod_openssl")

$SERVER["socket"] == "127.0.0.1:8443" {
    ssl.engine = "enable" 
    # default: self-signed RSA certificate
    ssl.pemfile = "/dev/shm/certs/rsa.pem" 
    ssl.privkey = "/dev/shm/certs/rsa.key" 
}

$HTTP["host"] == "server1" {
    # self-signed RSA certificate
    ssl.pemfile = "/dev/shm/certs/rsa.pem" 
    ssl.privkey = "/dev/shm/certs/rsa.key" 
}

$HTTP["host"] == "server2" {
    # self-signed EC certificate
    ssl.pemfile = "/dev/shm/certs/ec.pem" 
    ssl.privkey = "/dev/shm/certs/ec.key" 
}

I executed lighttpd in the foreground in one terminal, from the src/ directory of my source tree, running under a test account:

./lighttpd -D -f ttt.conf -m $PWD/.libs

I do not have the root CA you used for your self-signed cert, so I executed the following openssl s_client commands in another terminal, without -CAfile lighttpd.d/certs/root.pem

openssl s_client -connect 127.0.0.1:8443 -showcerts -servername "server1"  # expected RSA cert; got RSA cert
openssl s_client -connect 127.0.0.1:8443 -showcerts -servername "server2"  # expected EC cert;  got EC cert
openssl s_client -connect 127.0.0.1:8443 -showcerts -servername "default"  # expected RSA cert; got RSA cert

When I got the RSA cert:

Server certificate
subject=C = US, ST = Oregon, O = Biamp, CN = rsa-server

When I got the EC cert:

Server certificate
subject=C = US, ST = Oregon, O = Biamp, CN = ec-server

Would you please try to reproduce on your system using the same steps I used above? I suspect that you will be able to reproduce it on your system. Then, we get to go look to see if openssl was built and packaged differently between our systems. My system is x86_64 and running Fedora 32.

Actions #15

Updated by gstrauss almost 4 years ago

I noticed that your logs indicated TLSv1.2 was negotiated.

I also ran the commands with openssl s_client -no_tls1_3 with the same result, the EC cert was correctly returned for "server2"

Actions #16

Updated by gstrauss almost 4 years ago

I can reproduce the issue if I define the cipher preference order like so:

    ssl.cipher-list = "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384" 

but the default output of openssl ciphers on my system lists them in this order:

    ssl.cipher-list = "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384" 

Actions #17

Updated by gstrauss almost 4 years ago

I am curious. What curve did you use when you created the EC cert?

I used the following commands to create new test certs:

openssl req -x509 -newkey rsa:4096 -keyout rsa.key -out rsa.pem -days 365 -subj '/CN=rsa-server' -nodes
openssl genpkey -out ec.key -algorithm ec -pkeyopt ec_paramgen_curve:P-256
openssl req -x509 -key ec.key -out ec.pem -days 365 -subj '/CN=ec-server' -nodes

On my development branch, I was able to experiment with mbedTLS and GnuTLS replacing openssl. Both mbedTLS and GnuTLS worked as expected.

Actions #18

Updated by gstrauss almost 4 years ago

  • Tracker changed from Bug to Feature
  • Status changed from Reopened to New

https://www.openssl.org/docs/manmaster/man3/SSL_CTX_use_certificate.html
"All of the functions to set a new certificate will replace any existing certificate of the same type that has already been set."

Your issue appears to be a feature of openssl. The context may contain multiple certificates of different types (RSA, EC, etc). The default certificate configured for the socket in SSL_CTX is inherited by the SSL struct for the connection. If Server Name Indication (SNI) chooses a certificate of a different type, then both the default cert and the cert chosen by SNI will be present in the SSL struct. Cipher ordering preference will determine which certificate is chosen, as we have seen.

You can control cipher ordering preference in lighttpd with ssl.cipher-list, as I posted above. (ssl.honor-cipher-order = "enable" is also needed, but that is already the default in lighttpd, and is one of the requirements for PFS, so it is recommended that you keep the default setting for ssl.honor-cipher-order enabled and not disable it.)

tl;dr: If certificates for your hosts are of different types from that of the default certificate for the socket, then the certificate types for the hosts should be listed in ssl.cipher-list before the certificate type of the default certificate for the socket.

.

Programatically, with openssl 1.1.1, I could defer setting the certificate until the SSL_CTX_set_client_hello_cb. However, in earlier versions SSL_CTX_set_tlsext_servername_callback is called only if SNI is present.

While reviewing the code, I see that the certificate chain from the default certificate is loaded into SSL_CTX and inherited by the SSL struct. lighttpd mod_openssl currently does not load a separate certificate chain when installing the certificate chosen for SNI. It is fortunate that when needed, the chain is typically the same for the default certificate and the certificate chosen for SNI, so people have not noticed this limitation or have not reported it.

openssl 1.0.2 added SSL_CTX_set_cert_cb() so I may look into using that to set the cert and certificate chains, but that will not be a small patch and probably won't be done any time soon. I'll keep this issue open as a feature request, as there is a valid workaround available for those with mixed certificate types.

Actions #19

Updated by starr almost 4 years ago

Re-ordering the cipher list solves the problem on my target when using the self-signed certificates. However, I'm concerned by your statement that "lighttpd mod_openssl currently does not load a separate certificate chain when installing the certificate chosen for SNI." In my case, the real certificates will be generated by different organizations so they won't have the same certificate chain. Is there some way that I can get both certificate chains into the SSL_CTX? For example, if I included both sets of CAs in the file with the default certificate, would that cause them all to be loaded into the context?

Actions #20

Updated by gstrauss almost 4 years ago

For example, if I included both sets of CAs in the file with the default certificate, would that cause them all to be loaded into the context?

I believe so, but have not tested.

I'll address the limitation in the next version of lighttpd, but I do not yet know when that will be.

Actions #21

Updated by gstrauss almost 4 years ago

@starr please report to your upstream package maintainer(s) that openssl 1.0.2 reached end-of-life on 31 Dec 2019, and there will be no more security patches.
https://www.openssl.org/policies/releasestrat.html
https://www.openssl.org/blog/blog/2019/11/07/3.0-update/
Your upstream package maintainers really, really ought to migrate to openssl 1.1.1.

Similarly, lighttpd 1.4.55 was released 31 Jan 2020 and there are numerous important fixes since 1.4.51 (released 14 Oct 2018). Please upgrade.

Actions #22

Updated by gstrauss almost 4 years ago

  • Status changed from New to Patch Pending
  • Target version changed from 1.4.x to 1.4.56
Actions #23

Updated by gstrauss almost 4 years ago

I might be misunderstanding, but there appears to be a mis-feature in openssl which re-uses the trusted CA certificate list (lighttpd ssl.ca-file) from the openssl (SSL_CTX *) to complete the server certificate chain, if the chain has not been set for the openssl (SSL *). In other words, even though the CA list is intended for client certificate verification, it is also used for the server certificate if a chain has not been set for the server certificate.

If my understanding is correct, then your default server certificate can contain the full chain in the ssl.pemfile and be different from that of your SNI hosts. The chain of intermediate CAs for your SNI hosts could be in ssl.ca-file

Actions #24

Updated by gstrauss over 3 years ago

  • Status changed from Patch Pending to Fixed
Actions

Also available in: Atom