Project

General

Profile

[UE] Unable to access server - ERR_SSL_PROTOCOL_ERROR

Added by maxentry about 1 month ago

Hi - I have a strange issue - I rebooted my ubuntu server running lighty 1.4.65 (after updating some app including tzdata) and now I cannot access the site through edge or chrome (fired up explorer 11 and it reaches the site!). After some googling it was suggested this could be due to the date (and I had updated tzdata before reboot) so I checked again and re-configured tzdata (which hadn't changed) but still cannot use edge or chrome.
I then downloaded, compiled and installed lighttpd 1.4.67 using the same config as before and still cannot access the site (see images)


I also tried to access the site through curl and get the following output (the error is at the bottom)

  • TCP_NODELAY set
  • Connected to **** (******) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: /etc/ssl/certs
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
  • TLSv1.3 (IN), TLS handshake, Unknown (8):
  • TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Client hello (1):
  • TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: CN=*.*************
  • start date: Sep 22 10:29:09 2022 GMT
  • expire date: Dec 21 10:29:08 2022 GMT
  • issuer: C=US; O=Let's Encrypt; CN=R3
  • SSL certificate verify ok.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • TLSv1.3 (OUT), TLS Unknown, Unknown (23):
  • TLSv1.3 (OUT), TLS Unknown, Unknown (23):
  • TLSv1.3 (OUT), TLS Unknown, Unknown (23):
  • Using Stream ID: 1 (easy handle 0x56247afe3540)
  • TLSv1.3 (OUT), TLS Unknown, Unknown (23):

GET /index.html HTTP/2
Host: ******
User-Agent: curl/7.58.0
Accept: */*

  • TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • TLSv1.3 (OUT), TLS Unknown, Unknown (21):
  • TLSv1.3 (OUT), TLS alert, Server hello (2):
  • OpenSSL SSL_read: error:1416E09F:SSL routines:tls_process_new_session_ticket:length mismatch, errno 0
  • Failed receiving HTTP2 data
  • Connection #0 to host ** left intact

Replies (15)

RE: Unable to access server - ERR_SSL_PROTOCOL_ERROR - Added by gstrauss about 1 month ago

Providing pictures of what Microsoft calls "friendly" error messages is just noise. Don't bother. It is useless.

now I cannot access the site through edge or chrome (fired up explorer 11 and it reaches the site!)

That suggests a client issue.

Your issue might be some virus "protection" or "safe-web" or "VPN" third-party garbage running on Windows.

Did you check the lighttpd error log?
Has this ever worked for you with lighttpd 1.4.65 or lighttpd 1.4.67? Did it last work for you with lighttpd 1.4.64?
See recent forum post: https://redmine.lighttpd.net/boards/2/topics/10682

curl 7.58 is old. curl 7.58 was released Jan 2018, almost 5 years ago.
Ubuntu "stable" releases are sparsely updated, and so fall into the category of largely "unmaintained".

Have your tried a current version of curl? Have you tried running curl on localhost (on the same Ubuntu machine as the lighttpd server)?

RE: Unable to access server - ERR_SSL_PROTOCOL_ERROR - Added by maxentry about 1 month ago

Your issue might be some virus "protection" or "safe-web" or "VPN" third-party garbage running on Windows

I literally rebooted the VPN via SSH and couldn't access the site after it came back up - didn't close any browsers (since have rebooted local machine though still no access to site)
Did you check the lighttpd error log

I did and cannot see any errors related to my specific accesses, though I did notice yesterday (when the site was still accessible) there was an increase in errors like:
SSL: 1 error:140940F5:SSL routines:ssl3_read_bytes:unexpected record
SSL: 1 error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol

but these seem to be from clients polling the site

Has this ever worked for you with lighttpd 1.4.65 or lighttpd 1.4.67? Did it last work for you with lighttpd 1.4.64?

It's been working OK on 1.4.65 since a couple of days when it was released (jumped from 1.4.63, which worked as well so never used 64). As for 1.4.67 - I only compiled and installed it today after the issue started.
Have your tried a current version of curl? Have you tried running curl on localhost (on the same Ubuntu machine as the lighttpd server)?

Not tried later curl - I thought the error being there as well would be sufficient (I'll try and update it). And yes, I ran the curl command on the same server's SSH as well as from WSL

RE: Unable to access server - ERR_SSL_PROTOCOL_ERROR - Added by maxentry about 1 month ago

See recent forum post: https://redmine.lighttpd.net/boards/2/topics/10682

I've applied those changes but still cannot access site. (old) curl also still spits out same error with patch

RE: Unable to access server - ERR_SSL_PROTOCOL_ERROR - Added by gstrauss about 1 month ago

lighttpd 1.4.67 was released 17 Sep 2022. Again, IMO there is nothing "new" about software running in Ubuntu. The reason I say this is that if there was a widespread issue in lighttpd 1.4.65 or lighttpd 1.4.66 or lighttpd 1.4.67, it probably would have been reported before you noticed it. Since that is not the case, I am approaching this as probably not an issue with lighttpd, or at least is unlikely to be widespread.

What version of openssl is running on the system? Is that ancient, too?
Have you tried running lighttpd with an alternative TLS module, e.g. lighttpd mod_gnutls?

RE: Unable to access server - ERR_SSL_PROTOCOL_ERROR - Added by maxentry about 1 month ago

lighttpd 1.4.67 was released 17 Sep 2022. Again, IMO there is nothing "new" ...

That's reasonable - infact, I think that is the case. This error reared it's ugly head nearly 8hrs ago now, and my first thought was exactly as you put it - must be a client issue (though that I'd ran updates on the affected Ubuntu server which included tzdata before rebooting it made me suspicious it may be server related).
What version of openssl is running on the system? Is that ancient, too?

Looks like it is just as ancient - OpenSSL 1.1.1 11 Sep 2018
Have you tried running lighttpd with an alternative TLS module, e.g. lighttpd mod_gnutls?

No I have not.

RE: Unable to access server - ERR_SSL_PROTOCOL_ERROR - Added by maxentry about 1 month ago

I forgot to add - I have just run:
(Invoke-WebRequest 'https://**************').Headers
(Invoke-WebRequest 'https://**************').Content

in PowerShell and both are successful! Still can't access through chrome/edge, even after resetting the data for domain via settings.

RE: Unable to access server - ERR_SSL_PROTOCOL_ERROR - Added by maxentry about 1 month ago

Right - upgraded to Openssl 3.0.5, recompiled lighttpd 1.4.67 and now get the error:

lighttpd[20241]: 2022-10-27 23:34:33: (plugin.c.202) dlopen() failed for: /usr/local/sbin/1.4.67/lib/mod_openssl.so /usr/local/sbin/1.4.67/lib/mod_openssl.so: undefined symbol: ERR_new

RE: Unable to access server - ERR_SSL_PROTOCOL_ERROR - Added by gstrauss about 1 month ago

Have you tried a different client machine? Or is this an issue with a single client machine?

If you upgrade openssl to an incompatible major version bump, you need to recompile lighttpd against the development headers for the new openssl version.

RE: Unable to access server - ERR_SSL_PROTOCOL_ERROR - Added by maxentry about 1 month ago

No I have not tried another client machine - though I've just tried edge on iphone (I hope that counts) and get the same error.

RE: Unable to access server - ERR_SSL_PROTOCOL_ERROR - Added by gstrauss about 1 month ago

Your troubleshooting does not appear to be very methodical. Why Edge on iPhone? Why not Safari? Do you really only have 2 different clients? You have discovered that some applications work (powershell Invoke-WebRequest, IE 11), and yet you are making large changes on your server which is running very old software? As Microsoft does, they make monthly changes, and sometimes more often. Do you have any Mac or slightly older Windows clients that did not get patched this week? Have you tried Windows or Mac or iPhone clients connecting directly to the server, and not through a proxy or transparent proxy?

Are you in a corporate environment or a home environment? If you are in a corporate environment, why are you wasting my time with changes made on your corporate network? If you are in a corporate environment, I expect that you should have tested lighttpd in a non-corporate environment and found that lighttpd works, so you should have managed to conclude by now that the problem is your corporate environment intercepting and possibly corrupting the TLS, and not lighttpd.

RE: Unable to access server - ERR_SSL_PROTOCOL_ERROR - Added by gstrauss about 1 month ago

curl 7.82 works fine with my lighttpd 1.4.67 server.

$ curl -v https://......./
*   Trying .......:443...
* Connected to ...... (.......) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* ......
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  .......
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: .......]
* h2h3 [user-agent: curl/7.82.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x55dff161a2f0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: .......
> user-agent: curl/7.82.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 8)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200
< ....... 

By comparison, your trace using (very old) curl seems to have gotten lost:

TLSv1.3 (OUT), TLS Unknown, Unknown (23):
TLSv1.3 (OUT), TLS Unknown, Unknown (23):
TLSv1.3 (OUT), TLS Unknown, Unknown (23):

To see if some changes make a difference for the corporate malware intercepting your connection:
  • try temporarily setting forcing lighttpd to use TLSv1.2 with "MaxProtocol" => "TLSv1.2" in ssl.openssl.ssl-conf-cmd
  • try temporarily disabling HTTP/2 in lighttpd using server.feature-flags

RE: Unable to access server - ERR_SSL_PROTOCOL_ERROR - Added by maxentry about 1 month ago

 Why not Safari?

Actually, safari works!
Do you really only have 2 different clients?

Yes
Do you have any Mac or slightly older Windows clients that did not get patched this week? 

No
Are you in a corporate environment or a home environment?

Home

Like I said (and maybe I wasn't that detailed), I literally connected to the server via SSH and was doing sone changes to the site (had sone pages open), then decided to check for ubuntu updates then was advised to reboot. Since it was one of the quieter times, I decided to reboot and when I went to check the updates to the site I'd made, I was met by the error mentioned here.

PS: the delay in responding was that I had an internet outage ... not kidding! so no, not wasting your or anyone's time. Issue persists!

RE: Unable to access server - ERR_SSL_PROTOCOL_ERROR - Added by maxentry about 1 month ago

Somehow, this has resolved (itself?).
Like I mentioned earlier, after upgrading openssl to version 3.0.5 and recompiling lighttpd 1.4.67, it couldn't start because I hadn't linked to dev headers. So I reverted to 1.4.65 (which was compiled before upgrading openssl) and now everything works OK.

curl 7.82 works fine with my lighttpd 1.4.67 server.

Just to add, old curl also works (even with the errors pointed out!), so that was a red herring.
*   Trying ***.***.***.***...
* TCP_NODELAY set
* Connected to *********.*** (***.***.***.***) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.********.***
*  start date: Sep 22 10:29:09 2022 GMT
*  expire date: Dec 21 10:29:08 2022 GMT
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* Using Stream ID: 1 (easy handle 0x555bf9471540)
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> GET /index.html HTTP/2
> Host: *************
> User-Agent: curl/7.58.0
> Accept: */*
>
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/2 200

Not sure what happened here. I wasn't getting any meaningful error reporting via the Expect-CT / Report-To headers (other than network.error), so it may be something to do with the old openssl and chromium based browsers and possibly nothing to do with lihttpd (but I don't know).

RE: [UE] Unable to access server - ERR_SSL_PROTOCOL_ERROR - Added by gstrauss about 1 month ago

Hint: it had nothing to do with the official lighttpd. It probably had something to do with your environment.

RE: [UE] Unable to access server - ERR_SSL_PROTOCOL_ERROR - Added by maxentry about 1 month ago

Hint: We can't know for sure what we don't know.

    (1-15/15)