Bug #3278
openSevere performance reduction from 1.4.67 to 1.4.74 on legacy embedded device
Description
I am currently upgrading a legacy device from Yocto Kirkstone (lighttpd 1.4.67) to Scarthgap (lighttpd 1.4.74) and encountered an extreme reduction in response times when using HTTPS. From 10s before to now 60-180s to load the Web UI.
The HTTP access is not effected and is still around 3s.
The hardware is quite old with an i.MX257 with 400MHz and 64MB RAM.
One difference I have noticed that in the Chrome network debugging view in version 1.4.67 the access looks like it is not done in parallel:
In version 1.4.74 some requests are in parallel but stay in queued or stalled state for very long time (>60s):
I tried to set server.max-connections = 1 and it actually reduced the HTTPS access back to 10s and the access looks not parallel again in the Chrome networking debugging view.
The problem is that with server.max-connections = 1 HTTP access now takes way longer (>60s). Also I would like to avoid that a single connection can block all other connections.
I was not able to find anything in the release notes that sounds related to my problem.
I have attached my configuration for version 1.4.67 and my migrated one for 1.4.74.
Does anyone have an idea what could cause this issue?
Files
| clipboard-202503050813-epjwb.png (12.5 KB) clipboard-202503050813-epjwb.png | Access Diagram 1.4.67 | ||
| lighttpd.confV1.4.67.txt (1.39 KB) lighttpd.confV1.4.67.txt | |||
| clipboard-202503050849-h4oun.png (14.8 KB) clipboard-202503050849-h4oun.png | Access Diagram 1.4.74 | ||
| lighttpd.confV1.4.74.txt (1.27 KB) lighttpd.confV1.4.74.txt |
Updated by gstrauss 1 day ago · Edited
- Status changed from New to Need Feedback
Basic troubleshooting: what changed?
upgrading a legacy device from Yocto Kirkstone (lighttpd 1.4.67) to Scarthgap (lighttpd 1.4.74)
lighttpd is not the only thing that changed. You changed the entire operating system version, versions of underlying libraries, and more.
Have you tested lighttpd 1.4.77 on Yocto Kirkstone?
Have you tested lighttpd 1.4.67 on Yocto Scarthgap?
Have you tested with different clients (e.g. curl)?
Have you tested with different lighttpd TLS modules? (e.g. mod_openssl, mod_mbedtls, mod_wolfssl)
Have you tested different parts of your site? Plain .html and plain .txt versus performance of .shtml or CGI or FastCGI or other requests?
You filed a bug against lighttpd which is extremely vague. Please provide more evidence why you think this is an issue with lighttpd (hint: it is unlikely a vague issue affecting all of lighttpd). I will not respond further here until you do.
From 10s before to now 60-180s to load the Web UI.
10s to load a Web UI is not a good starting point and suggests room for improvement in your implementation.
For an embedded system, your configuration has many inefficiencies and non-recommended configuration options that have been cargo-cult-copied from older generations of lighttpd configurations.
Configuration: TLS/SSL
Resource Tuning
Performance Tuning
It is recommended that you remove all items from your config which are lighttpd defaults.
It is recommended that you remove all items from your config which you are not using (assuming you have not shared truncated configurations).
Do not list server.modules you are not using.
Do not setenv.add-environment = ( "PATH" => env.PATH) unless you know what it does and have document why you need it. (Perhaps your SSI?)ssl.read-ahead = "disable" is default in lighttpd since lighttpd 1.4.46 (released Oct 2017). No need to list.index-file.names should be as short as possible with only the items you use.
lighttpd 1.4.71 and later include a default set of web mime types, so use include "/etc/lighttpd/mimetypes.conf" only if lighttpd default set does not meet your needs.$HTTP["url"] =~ "\.pdf$" { server.range-requests = "disable" } has been obsolete for well over a decade.server.use-ipv6 = "enable" is not needed when you have specified IPv6 addresses in your config, which you have.protocol = "https://" is not used.ssl.disable-client-renegotiation = "enable" is default.ssl.dh-file = "/etc/lighttpd/ssl/dhparam.pem" should no longer be specified. Let lighttpd and TLS library handle it unless you have a specific requirement not to do so."DHParameters" => "/etc/lighttpd/ssl/dhparam.pem" should not be specified. Let lighttpd and TLS library handle it unless you have a specific requirement not to do so.setenv.add-environment = ("HTTPS" => "on") should be removed. lighttpd sets that for CGI where appropriate.ssl.honor-cipher-order = "disable" should be removed. Prefer lighttpd TLS defaults.ssl.cipher-list = "HIGH" should be removed. Prefer lighttpd TLS defaults.
You have specified "MinProtocol" => "TLSv1.3" and that overrides the above excess ssl.* settings.
You're on an embedded system, so consider lighttpd TLS modules which use TLS libraries aimed at embedded system, such as mbedTLS or WolfSSL
Configuration: TLS/SSL
Updated by meinhard.k 1 day ago
Hello gstrauss,
thank you for your detailed response, especially the analysis of my configuration!
Have you tested lighttpd 1.4.67 on Yocto Scarthgap?
I have just tried lighttpd 1.4.67 on Yocto Scarthgap and this resolved the problem. The performance was back to what I have seen on Kirkstone.
Have you tested with different clients (e.g. curl)?
I have tried time curl https://<IPADDR>/ which only took 1.9s but also only downloaded the index.html.
I tired time wget --no-parent -r https://IPADDR/ which reported acceptable time results:
Total wall clock time: 6.6s
Downloaded: 283 files, 3.4M in 2.5s (1.37 MB/s)
real 0m6.656s
I also tried to use Firefox which was able to load the web UI in around 18s. Which is quite high but it is at least consistently the same loading time.
Firefox also loaded all files in parallel as far as I can tell from the performance recording. Most requests are stuck in "Waiting for socket thread" before they are served.
Edge showed the same long loading time as Chrome, which is not surprising.
Have you tested with different lighttpd TLS modules? (e.g. mod_openssl, mod_mbedtls, mod_wolfssl)
Changing the TLS module to anything but mod_openssl is not that trivial. I have to integrate another meta layer. I will try that and come back to you later this week.
Have you tested different parts of your site?
As far as I can tell the problem happens not when downloading single file but when loading all resources of the web UI at the same time.
For an embedded system, your configuration has many inefficiencies and non-recommended configuration options
I will clean my configuration up. Thank you for all the suggestions!
I will do more investigations on what could be the cause for my issue and update this issue when I find something.
Updated by gstrauss about 21 hours ago
I have just tried lighttpd 1.4.67 on Yocto Scarthgap and this resolved the problem. The performance was back to what I have seen on Kirkstone.
Starting in lighttpd 1.4.68, lighttpd mod_openssl and lighttpd mod_gnutls will try to use KTLS if available. You can disable KTLS and test performance with:ssl.openssl.ssl-conf-cmd += ( "Options" => "-KTLS" )
See lighttps TLS docs kTLS
What Linux kernel version are you using in Scarthgap?
What libc and version are you using in Scarthgap? glibc? musl?
Starting in lighttpd 1.4.74, lighttpd will use preadv2 RWF_NOWAIT when available. You might configure your build to fail to detect preadv2 in order to test without this optimization.
Updated by gstrauss about 16 hours ago · Edited
BTW, please check your build and see if you are building lighttpd without HTTP/2 support. If you are not, then you should try building lighttpd with HTTP/2.
If you are building lighttpd with HTTP/2, then check how curl and Chrome and Firefox are connecting (HTTP/1.1 or HTTP/2) and see server.feature-flags to test with support for the HTTP/2 protocol disabled in lighttpd.
What filesystem type holds your docroot? JFFS2? BTRFS? other?
server.network-backend server.network-backend = "writev", though you should prefer the default, which is sendfile, if that works.
Most of this is mentioned in the doc
Configuration: TLS/SSL
Resource Tuning
Performance Tuning
Also available in: Atom