Lighttpd

Your troubleshooting is all over the place -- or rather your communication of your troubleshooting. Please try to be more methodical in your communication.

No issue in http mode or any other download method with isn't via the web-server.

That's a good start. Everything you said prior to that was confusing to me because you failed to scope the conversation before that point.

For the problem you are having, it is not relevant whether SNMP works or not. That merely establishes that the embedded device is on the network. So let us only talk about HTTP and HTTPS in this thread.

Start with the basics.
Can you download a simple static file, e.g. index.html using HTTP? (i.e. from port 80)
Can you download a simple static file, e.g. index.html using HTTPS? (i.e. from port 443)

The fact that I am asking such simple questions should tell you how well (or not) you have communicated your previous scattershot troubleshooting.

all runs fine except for when downloading a file

Did you mean HTTP? HTTPS? Or did you mean completely separate services running completely separate ports and serviced by completely separate daemons? (and almost completely irrelevant to troubleshooting this service (https) running on this port (443) and serviced by this (lighttpd) process)

Here's a hint: if the problem is the certificate, then all the HTTPS requests should fail. Your "fails for some requests" tell me that you prematurely came to a conclusion and you have not provided sufficient information to justify that conclusion, but perhaps you were just deceptively vague with your "description" that "all runs fine except for when downloading a file"

Don't have much experience in debugging web-server sessions, but happy to try anything.

What is the client you are testing with? For troubleshooting, I would suggest avoiding browser which hide most information from you.

Try 'man curl' and make simple requests like curl -v -k https://your-embedded-device.local/index.txt

Your post above seems to combine and duplicate the lighttpd.conf. What should be there?
RRU_T-10.210.14.49> cat /mmx/conf/lighttpd.conf ?

Please make sure that you are running lighttpd with the configuration you think you are running.

To print the config that lighttpd sees: lighttpd -f /mmx/conf/lighttpd.conf -p
Double check that you are running lighttpd using /mmx/conf/lighttpd.conf and not the more typical /etc/lighttpd/lighttpd.conf.

Agree, my problem description looks like a brain dump. Really sorry for it!

I've run curl -v -k https://10.210.14.49/index.html and got this,

*   Trying 10.210.14.49:443...
* TCP_NODELAY set
* Connected to 10.210.14.49 (10.210.14.49) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=10.210.14.49; O=Mimomax Wireless Ltd; OU=Support; emailAddress=support_services@mimomax.com
*  start date: Sep 19 05:35:54 2022 GMT
*  expire date: Sep 19 05:35:54 2023 GMT
*  issuer: CN=10.210.14.49; O=Mimomax Wireless Ltd; OU=Support; emailAddress=support_services@mimomax.com
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x561238c202f0)
> GET /index.html HTTP/2
> Host: 10.210.14.49
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 8)!
< HTTP/2 404 
< content-type: text/html
< content-length: 341
< date: Tue, 20 Sep 2022 02:29:57 GMT
< server: lighttpd/1.4.66
<
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 Not Found</title>
 </head>
 <body>
  <h1>404 Not Found</h1>
 </body>
</html>
* Connection #0 to host 10.210.14.49 left intact

Some clarity on other points,
1. Client - I use Chrome.
2. We can run http:80 or https:443 on the device, but not at the same time.
3. http:80 has got zero issues. https with self-signed cert has issues but it isn't the certificate. Can share parts of certificate if needed.
4. SNMP has nothing to do with anything here - SNMP MIB is just another https downloadable artefact that our device website supports.

Lets talk HTTPS only.

That looks like HTTPS is working, but that you did not create /mmx/web/index.html, so the result is a 404 Not Found.

> GET /index.html HTTP/2
> Host: 10.210.14.49
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 8)!
< HTTP/2 404 
< content-type: text/html
< content-length: 341
< date: Tue, 20 Sep 2022 02:29:57 GMT
< server: lighttpd/1.4.66
<
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 Not Found</title>
 </head>
 <body>
  <h1>404 Not Found</h1>
 </body>
</html>

You should do some additional troubleshooting and collect some additional data to demonstrate the your issue. We can see from the above that your original premature conclusion that the problem had something (hand-wavy) to do with the certificate is not supported by these observations.

Not saying it is the certificate, just happens in HTTPS mode.

It might be related to the time it waits for source file to be ready. Unlike other ''download'' features, the one that fails requires a shell script to complete preparing files, then files are compressed and the archive is sent back to the client. PHP is like the following snippet (I've removed all the product-specific logic, making it as basic as I could),

   header('Content-type: application/x-tar');
   header('Content-Disposition: attachment; filename="test.tar"');
   shell_exec("./prepare-file.sh");
   # there is a while loop which waits for the shell script to complete...
   readfile($file); 

Other PHP code that works fine doesn't apparently need to wait to readfile($file); - maybe it is related.

Unsure if I could use curl for it.

Not saying it is the certificate, just happens in HTTPS mode.

The title of this post is "SSL handshake issue for some requests in 1.4.66" which is clearly wrong. In the future, you should try to describe the problem with observations in your post title, not your premature conclusions.

It might be related to the time it waits for source file to be ready. Unlike other ''download'' features, the one that fails requires a shell script to complete preparing files, then files are compressed and the archive is sent back to the client.

Are you aware of how long you have spent on this (and how much of my time you have wasted) before providing this more detailed observation?
Did you have the same problem accessing these specific shell scripts via HTTP instead of HTTPS?

Do you have the same problems if you (temporarily) disable mod_deflate? Are you running out of disk space in /tmp or the deflate cache location? Why have you enabled deflate caching for these large results which are prepared by a script and presumably not reusable? (Then again, if the backend is not creating an ETag for the result, then the response should not be stored in the deflate cache.)

Have you read the lighttpd documentation for default timeouts?

Configuration: Options
server.max-read-idle
server.max-write-idle
server.max-keep-alive-idle

or
mod_fastcgi
"connect-timeout"
"read-timeout"
"write-timeout"

Seems that the problem is resolved by setting,

server.max-keep-alive = 60

#######

First, @gstrauss I am truly sorry for wasting your time. Your posts have had very useful guidance in them. I am in charge of a team maintaining millions of LOC here with 30+ OSS packages where lighttpd is one of, and we don't have the time to dive into each project enough, unfortunately. This is the reality of working in a start-up that try to do everything while investing the bare minimum into it.

Second, brilliant titles are hard to come up with. SSL handshake err print was the only error I saw back then, so it had been an observation. Moreover, it never happened in HTTP . The idea, I thought, is that one could adjust the title of the issue once your understanding of it changes. I've not spent more than 2 minutes thinking about the title and invested most of the effort into thinking about the problem.

Third, I've tried all your suggestions and the last one - trying the server.max-keep-alive-idle - was the one I needed.

Cheers!

Your excuses are pathetic and suggest that you are not sorry in the least.

Next time, please RTFM.