mod_openssl: SSL decryption error (MAC error)
Hi,
I'm running lighttpd 1.4.74 on embedded Yocto based system. When downloading bigger (~4MiB) static file via HTTPS, client (tried wget / curl / chromium / FF) asserts TLS alert "bad record mac" in the middle of transfer. It happens at different offsets and rarely it don't happen at all and file is successfuly donwloaded. Interrestingly when doing the same at localhost and not via remote network, the incidence of error is much lower but happens too (in approx. 1 in 10 tries). Happens with both TLSv1.2 and TLSv1.3.
I tried to tweak these server options:
ssl.read-ahead server.network-backend server.chunkqueue-chunk-sz
without any effect.
I've also tried newest version 1.4.79 with the same result. On the other hand with older version 1.4.67 it works completely fine.
Curl log and lighttpd.conf and lighttpd_error.log attached. OpenSSL used: 3.2.3. (but also tried somewhat newer and older without effect).
Not saying this is lighttpd bug, but any idea how to track down the root cause?
| curl_log.txt (2.56 KB) curl_log.txt | |||
| lighttpd.conf (4.55 KB) lighttpd.conf | |||
| lighttpd_error.log (927 Bytes) lighttpd_error.log |
Replies (4)
RE: mod_openssl: SSL decryption error (MAC error) - Added by gstrauss 3 days ago
Not saying this is lighttpd bug, but any idea how to track down the root cause?
Quick response for now:
I took a quick look at the differences in lighttpd mod_openssl between 1.4.67 and 1.4.74.git log lighttpd-1.4.67..lighttpd-1.4.74 mod_openssl.c
There were incremental hardening changes made to the security defaults. Please share your TLS configuration in lighttpd.conf. Also, what version of openssl did you use when testing lighttpd 1.4.67
For initial troubleshooting, try ruling out whether or not KTLS or preadv2() with RWF_NOWAIT are involved.ssl.openssl.ssl-conf-cmd += ( "Options" => "-KTLS" )
If you build lighttpd, you can disable HAVE_PREADV2 by temporarily setting #undef HAVE_PREADV2 after #include "first.h" in src/chunk.c and src/network_write.c
What is the filesystem on your Yocto device underlying the file being served? Is it a static file or is it being served through FastCGI or some other means? If some other means, try disabling lighttpd request offload from backend: server.stream-response-body = 2 in lighttpd.conf
What is the filesystem underlying the lighttpd server.upload-dirs? The default is @/var/tmp. How much space is available that filesystem compared to the size of the download?
If you can build different versions of lighttpd, it would be useful to narrow down which version between 1.4.67 and 1.4.74 introducted the change in behavior you are seeing, and if it is related to openssl version or not.
Another way to troubleshoot is to try using a different lighttpd TLS module, e.g. lighttpd mod_mbedtls or lighttpd mod_wolfssl, which use mbedTLS and WolfSSL, both TLS libraries aimed at embedded systems.
RE: mod_openssl: SSL decryption error (MAC error) - Added by dandys 2 days ago
Thank you. With KTLS disabled it works. Using git bisect I was also able to narrow it down to commit 9dc670db776 which adds initial support of KTLS (all versions before this commit works fine).
I'll probably just keep KTLS disabled for now but wondering why it don't work.
To answer some of your other questions: I'm testing everything using openssl 3.2.3 and Linux 6.6.36. The file being served is static on ext4 (also tried tmpfs).
RE: mod_openssl: SSL decryption error (MAC error) - Added by gstrauss 2 days ago
Thank you for the update. It is fine to leave KTLS disabled, especially if it is causing problems for you.
If lighttpd with KTLS worked with an earlier kernel version, then there might be an issue with the kernel, and the issue might be fixed by a newer kernel.
Alternatively, it could possibly be an issue between OpenSSL and the kernel. If you are ambitious, please test if you can reproduce the issue (with kTLS enabled) using lighttpd mod_gnutls instead of lighttpd mod_openssl.
See Docs_SSL for how to enable kTLS in the GnuTLS system configuration.
RE: mod_openssl: SSL decryption error (MAC error) - Added by gstrauss 2 days ago
Note also that there have been security bugs with kTLS, e.g. https://www.cvedetails.com/cve/CVE-2024-53138/ and 6.6.36 is potentially affected by CVE-2024-53138