Project

General

Profile

[UE] (garbage title) SSL handshake issue for some requests

Added by gopher79 17 days ago

Issue showed up after upgrade from 1.4.55 to 1.4.66. This is a device installed with self-signed cert (valid for a year) running a custom website, all runs fine except for when downloading a file (compressed .bz2 logs) from it - results in a Network error, by the end of the download.

with debug.log-ssl-noise = "enable", I see this message when trying to download a file from a page (bz2 archive),
(mod_openssl.c.3281) SSL: 1 error:14094416:lib(20):func(148):reason(1046) (10.54.210.109).
Not all requests fail but on this device they rarely succeed, however, other downloads don't fail, e.g. SNMP MIB download is always successful. The message in the log still fails for any download attempt, successful or not so maybe it is a red herring. Issue is intermittent, this specific bit happens on this device much more often than on others, higher latency (~30-50ms) of access to Host from the client could play a role.

No issue in http mode or any other download method with isn't via the web-server.

The lighttpd.conf configuration has not changed much except for renaming compress. to deflate..
Shared web server config.

Similar behavior with Chrome (v105.0.5195.102) and recent Edge.

Running on embedded Linux box distro arago/openembedded 4.14.67
OpenSSL 1.1.1q
lighttpd 1.4.66 (ssl)

Configuration is below, update the SSL and some other parts based on instructions in Wiki and some previous advise from @gstrauss.
Don't have much experience in debugging web-server sessions, but happy to try anything.

## See lighttpd documentation for detailed configuration options
# Base configuration
var.server_root = "/mmx/web" 
var.state_dir   = "/var/run" 
## Load the modules.
server.modules += (
   "mod_deflate",
   "mod_fastcgi",
   "mod_openssl" 
)

## Use IPv6?
#server.use-ipv6 = "enable" 

## Run as a different username/groupname.
## This requires root permissions during startup.
server.username  = "nobody" 
server.groupname = "nobody" 

## Document root
server.document-root = server_root

## store a pid file
server.pid-file = state_dir + "/lighttpd.pid" 

# PHP is run separately as FastCGI process
fastcgi.server = (
   ".php" =>
   ((
      "host" => "127.0.0.1",
      "port" => 8080
   ))
)

## If you want to log to syslog you have to unset the
## server.errorlog setting and uncomment the next line.
server.errorlog-use-syslog = "enable" 
debug.log-ssl-noise = "enable" 

# Ideally we want to use epoll but it's not supported by DPS2
server.event-handler = "select" 
server.stat-cache-engine = "simple" 

index-file.names += (
  "index.xhtml", "index.html", "index.htm", "default.htm", "index.php" 
)

## which extensions should not be handle via static-file transfer
##
## .php, .pl, .fcgi are most often handled by mod_fastcgi or mod_cgi
##
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi", ".scgi" )
# etags should be enabled by default but let's explicitly set it anyway
static-file.etags = "enable" 

## mimetype mapping
mimetype.assign = (
  ".html" => "text/html",
  ".css" => "text/css",
  ".js" => "text/javascript",
  ".txt" => "text/plain",
  ".jpg" => "image/jpeg",
  ".png" => "image/png" 
)
## Should lighttpd follow symlinks?
server.follow-symlink = "enable" 

## defaults to /var/tmp as we assume it is a local harddisk
server.upload-dirs = ( "/tmp" )

#Compress static content
deflate.cache-dir = "/tmp/lighttpdcompress/" 
deflate.allowed-encodings = ("gzip", "deflate")
deflate.mimetypes = (
   "text/plain",
   "text/html",
   "text/css",
   "text/javascript",
   "image/jpeg",
   "image/png" 
)

## httpd.sh adds additional configuration after this point

RRU_T-10.210.14.49> cat /mmx/conf/lighttpd.conf
## See lighttpd documentation for detailed configuration options
# Base configuration
var.server_root = "/mmx/web" 
var.state_dir   = "/var/run" 
## Load the modules.
server.modules += (
   "mod_deflate",
   "mod_fastcgi",
   "mod_openssl" 
)

## Use IPv6?
#server.use-ipv6 = "enable" 

## Run as a different username/groupname.
## This requires root permissions during startup.
server.username  = "nobody" 
server.groupname = "nobody" 

## Document root
server.document-root = server_root

## store a pid file
server.pid-file = state_dir + "/lighttpd.pid" 

# PHP is run separately as FastCGI process
fastcgi.server = (
   ".php" =>
   ((
      "host" => "127.0.0.1",
      "port" => 8080
   ))
)

## If you want to log to syslog you have to unset the
## server.errorlog setting and uncomment the next line.
server.errorlog-use-syslog = "enable" 
debug.log-ssl-noise = "enable" 

# Ideally we want to use epoll but it's not supported by DPS2
server.event-handler = "select" 
server.stat-cache-engine = "simple" 

index-file.names += (
  "index.xhtml", "index.html", "index.htm", "default.htm", "index.php" 
)

## which extensions should not be handle via static-file transfer
##
## .php, .pl, .fcgi are most often handled by mod_fastcgi or mod_cgi
##
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi", ".scgi" )
# etags should be enabled by default but let's explicitly set it anyway
static-file.etags = "enable" 

## mimetype mapping
mimetype.assign = (
  ".html" => "text/html",
  ".css" => "text/css",
  ".js" => "text/javascript",
  ".txt" => "text/plain",
  ".jpg" => "image/jpeg",
  ".png" => "image/png" 
)
## Should lighttpd follow symlinks?
server.follow-symlink = "enable" 

## defaults to /var/tmp as we assume it is a local harddisk
server.upload-dirs = ( "/tmp" )

#Compress static content
deflate.cache-dir = "/tmp/lighttpdcompress/" 
deflate.allowed-encodings = ("gzip", "deflate")
deflate.mimetypes = (
   "text/plain",
   "text/html",
   "text/css",
   "text/javascript",
   "image/jpeg",
   "image/png" 
)

## httpd.sh adds additional configuration after this point

server.port = 443
ssl.engine = "enable" 
ssl.pemfile = "/mmx/db/host.pem" 
ssl.openssl.ssl-conf-cmd = ("MinProtocol" => "TLSv1.2")

Replies (9)

RE: SSL handshake issue for some requests in 1.4.66 - Added by gstrauss 17 days ago

Your troubleshooting is all over the place -- or rather your communication of your troubleshooting. Please try to be more methodical in your communication.

No issue in http mode or any other download method with isn't via the web-server.

That's a good start. Everything you said prior to that was confusing to me because you failed to scope the conversation before that point.

For the problem you are having, it is not relevant whether SNMP works or not. That merely establishes that the embedded device is on the network. So let us only talk about HTTP and HTTPS in this thread.

Start with the basics.
Can you download a simple static file, e.g. index.html using HTTP? (i.e. from port 80)
Can you download a simple static file, e.g. index.html using HTTPS? (i.e. from port 443)

The fact that I am asking such simple questions should tell you how well (or not) you have communicated your previous scattershot troubleshooting.

all runs fine except for when downloading a file

Did you mean HTTP? HTTPS? Or did you mean completely separate services running completely separate ports and serviced by completely separate daemons? (and almost completely irrelevant to troubleshooting this service (https) running on this port (443) and serviced by this (lighttpd) process)

Here's a hint: if the problem is the certificate, then all the HTTPS requests should fail. Your "fails for some requests" tell me that you prematurely came to a conclusion and you have not provided sufficient information to justify that conclusion, but perhaps you were just deceptively vague with your "description" that "all runs fine except for when downloading a file"

RE: SSL handshake issue for some requests in 1.4.66 - Added by gstrauss 17 days ago

Don't have much experience in debugging web-server sessions, but happy to try anything.

What is the client you are testing with? For troubleshooting, I would suggest avoiding browser which hide most information from you.

Try 'man curl' and make simple requests like curl -v -k https://your-embedded-device.local/index.txt

RE: SSL handshake issue for some requests in 1.4.66 - Added by gstrauss 17 days ago

Your post above seems to combine and duplicate the lighttpd.conf. What should be there?
RRU_T-10.210.14.49> cat /mmx/conf/lighttpd.conf ?

Please make sure that you are running lighttpd with the configuration you think you are running.

To print the config that lighttpd sees: lighttpd -f /mmx/conf/lighttpd.conf -p
Double check that you are running lighttpd using /mmx/conf/lighttpd.conf and not the more typical /etc/lighttpd/lighttpd.conf.

RE: SSL handshake issue for some requests in 1.4.66 - Added by gopher79 17 days ago

Agree, my problem description looks like a brain dump. Really sorry for it!

I've run curl -v -k https://10.210.14.49/index.html and got this,

*   Trying 10.210.14.49:443...
* TCP_NODELAY set
* Connected to 10.210.14.49 (10.210.14.49) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=10.210.14.49; O=Mimomax Wireless Ltd; OU=Support; emailAddress=support_services@mimomax.com
*  start date: Sep 19 05:35:54 2022 GMT
*  expire date: Sep 19 05:35:54 2023 GMT
*  issuer: CN=10.210.14.49; O=Mimomax Wireless Ltd; OU=Support; emailAddress=support_services@mimomax.com
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x561238c202f0)
> GET /index.html HTTP/2
> Host: 10.210.14.49
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 8)!
< HTTP/2 404 
< content-type: text/html
< content-length: 341
< date: Tue, 20 Sep 2022 02:29:57 GMT
< server: lighttpd/1.4.66
<
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 Not Found</title>
 </head>
 <body>
  <h1>404 Not Found</h1>
 </body>
</html>
* Connection #0 to host 10.210.14.49 left intact

Some clarity on other points,
1. Client - I use Chrome.
2. We can run http:80 or https:443 on the device, but not at the same time.
3. http:80 has got zero issues. https with self-signed cert has issues but it isn't the certificate. Can share parts of certificate if needed.
4. SNMP has nothing to do with anything here - SNMP MIB is just another https downloadable artefact that our device website supports.

Lets talk HTTPS only.

RE: SSL handshake issue for some requests in 1.4.66 - Added by gstrauss 17 days ago

That looks like HTTPS is working, but that you did not create /mmx/web/index.html, so the result is a 404 Not Found.

> GET /index.html HTTP/2
> Host: 10.210.14.49
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 8)!
< HTTP/2 404 
< content-type: text/html
< content-length: 341
< date: Tue, 20 Sep 2022 02:29:57 GMT
< server: lighttpd/1.4.66
<
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 Not Found</title>
 </head>
 <body>
  <h1>404 Not Found</h1>
 </body>
</html>

You should do some additional troubleshooting and collect some additional data to demonstrate the your issue. We can see from the above that your original premature conclusion that the problem had something (hand-wavy) to do with the certificate is not supported by these observations.

RE: SSL handshake issue for some requests in 1.4.66 - Added by gopher79 17 days ago

Not saying it is the certificate, just happens in HTTPS mode.

It might be related to the time it waits for source file to be ready. Unlike other ''download'' features, the one that fails requires a shell script to complete preparing files, then files are compressed and the archive is sent back to the client. PHP is like the following snippet (I've removed all the product-specific logic, making it as basic as I could),

   header('Content-type: application/x-tar');
   header('Content-Disposition: attachment; filename="test.tar"');
   shell_exec("./prepare-file.sh");
   # there is a while loop which waits for the shell script to complete...
   readfile($file); 

Other PHP code that works fine doesn't apparently need to wait to readfile($file); - maybe it is related.

Unsure if I could use curl for it.

RE: SSL handshake issue for some requests in 1.4.66 - Added by gstrauss 17 days ago

Not saying it is the certificate, just happens in HTTPS mode.

The title of this post is "SSL handshake issue for some requests in 1.4.66" which is clearly wrong. In the future, you should try to describe the problem with observations in your post title, not your premature conclusions.

It might be related to the time it waits for source file to be ready. Unlike other ''download'' features, the one that fails requires a shell script to complete preparing files, then files are compressed and the archive is sent back to the client.

Are you aware of how long you have spent on this (and how much of my time you have wasted) before providing this more detailed observation?
Did you have the same problem accessing these specific shell scripts via HTTP instead of HTTPS?

Do you have the same problems if you (temporarily) disable mod_deflate? Are you running out of disk space in /tmp or the deflate cache location? Why have you enabled deflate caching for these large results which are prepared by a script and presumably not reusable? (Then again, if the backend is not creating an ETag for the result, then the response should not be stored in the deflate cache.)

Have you read the lighttpd documentation for default timeouts?

Configuration: Options
server.max-read-idle
server.max-write-idle
server.max-keep-alive-idle

or
mod_fastcgi
"connect-timeout"
"read-timeout"
"write-timeout"

RE: SSL handshake issue for some requests in 1.4.66 - Added by gopher79 16 days ago

Seems that the problem is resolved by setting,

server.max-keep-alive = 60
anyway, 5s default was too short.

#######

First, @gstrauss I am truly sorry for wasting your time. Your posts have had very useful guidance in them. I am in charge of a team maintaining millions of LOC here with 30+ OSS packages where lighttpd is one of, and we don't have the time to dive into each project enough, unfortunately. This is the reality of working in a start-up that try to do everything while investing the bare minimum into it.

Second, brilliant titles are hard to come up with. SSL handshake err print was the only error I saw back then, so it had been an observation. Moreover, it never happened in HTTP . The idea, I thought, is that one could adjust the title of the issue once your understanding of it changes. I've not spent more than 2 minutes thinking about the title and invested most of the effort into thinking about the problem.

Third, I've tried all your suggestions and the last one - trying the server.max-keep-alive-idle - was the one I needed.

Cheers!

RE: SSL handshake issue for some requests in 1.4.66 - Added by gstrauss 16 days ago

Your excuses are pathetic and suggest that you are not sorry in the least.

Next time, please RTFM.

    (1-9/9)