Project

General

Profile

[Solved] [1.4.59] Main process exited, code=dumped, status=11/SEGV

Added by finn@lesueur.nz about 2 months ago

Kia ora all!

Thank you for the very cool webserver. I am in the process of migrating from Apache and am enjoying it very much so far. Unfortunately I am running into a problem where lighttpd crashes occasionally but I can't seem to find too much in the logs.

Server Version

root@webserver:/etc/lighttpd# lighttpd -v
lighttpd/1.4.59 (ssl) - a light and fast webserver

Error Logs

I woke up this morning to a non-running lighttpd and on inspecting the journalctl logs I see this:

root@webserver:/etc/lighttpd# journalctl -e -u lighttpd.service --since today
Jul 28 02:41:45 webserver systemd[1]: Starting Lighttpd Daemon...
Jul 28 02:41:45 webserver systemd[1]: Started Lighttpd Daemon.
Jul 28 02:42:01 webserver systemd[1]: lighttpd.service: Main process exited, code=dumped, status=11/SEGV
Jul 28 02:42:01 webserver systemd[1]: lighttpd.service: Failed with result 'core-dump'.
Jul 28 02:42:01 webserver systemd[1]: lighttpd.service: Scheduled restart job, restart counter is at 915.
Jul 28 02:42:01 webserver systemd[1]: Stopped Lighttpd Daemon.
Jul 28 02:42:01 webserver systemd[1]: Starting Lighttpd Daemon...
Jul 28 02:42:01 webserver systemd[1]: Started Lighttpd Daemon.
Jul 28 02:44:05 webserver systemd[1]: lighttpd.service: Main process exited, code=dumped, status=11/SEGV
Jul 28 02:44:05 webserver systemd[1]: lighttpd.service: Failed with result 'core-dump'.
Jul 28 02:44:05 webserver systemd[1]: lighttpd.service: Scheduled restart job, restart counter is at 916.
Jul 28 02:44:05 webserver systemd[1]: Stopped Lighttpd Daemon.
Jul 28 02:44:05 webserver systemd[1]: Starting Lighttpd Daemon...
Jul 28 02:44:05 webserver systemd[1]: Started Lighttpd Daemon.
Jul 28 02:45:07 webserver systemd[1]: lighttpd.service: Main process exited, code=dumped, status=11/SEGV
Jul 28 02:45:07 webserver systemd[1]: lighttpd.service: Failed with result 'core-dump'.
Jul 28 02:45:07 webserver systemd[1]: lighttpd.service: Scheduled restart job, restart counter is at 917.
Jul 28 02:45:07 webserver systemd[1]: Stopped Lighttpd Daemon.

This simply repeats on an on. If I filter for err logs I get:

root@webserver:/etc/lighttpd# journalctl -e -u lighttpd.service --since today -p err
Jul 28 05:46:25 webserver systemd[1]: Failed to start Lighttpd Daemon.
Jul 28 08:46:28 webserver systemd[1]: Failed to start Lighttpd Daemon.

If I check out my /var/log/lighttpd/error.log I see hundreds of lines of the following on repeat. I have attached the full log.

2021-07-28 02:20:50: server.c.1513) server started (lighttpd/1.4.59)
2021-07-28 02:21:43: mod_openssl.c.3095) SSL: 1 error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported
protocol
2021-07-28 02:23:27: server.c.1513) server started (lighttpd/1.4.59)
2021-07-28 02:24:38: server.c.1513) server started (lighttpd/1.4.59)
2021-07-28 02:24:45: server.c.1513) server started (lighttpd/1.4.59)
2021-07-28 02:24:46: mod_openssl.c.3095) SSL: 1 error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported
protocol
2021-07-28 02:24:46: mod_openssl.c.3095) SSL: 1 error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported
protocol
2021-07-28 02:25:50: server.c.1513) server started (lighttpd/1.4.59)
2021-07-28 02:26:25: server.c.1513) server started (lighttpd/1.4.59)
2021-07-28 02:26:35: server.c.1513) server started (lighttpd/1.4.59)
2021-07-28 02:26:53: server.c.1513) server started (lighttpd/1.4.59)
2021-07-28 02:26:54: mod_openssl.c.3095) SSL: 1 error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported
protocol
2021-07-28 02:26:54: mod_openssl.c.3095) SSL: 1 error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported
protocol
2021-07-28 02:29:01: server.c.1513) server started (lighttpd/1.4.59)
2021-07-28 02:30:50: server.c.1513) server started (lighttpd/1.4.59)
2021-07-28 02:31:28: server.c.1513) server started (lighttpd/1.4.59)

System Configuration

I am running a Linode Nanode server and regularly have 400mb of RAM available an am running at about 60% disk capacity.

system info:
  Distro......: Ubuntu 21.04
  Kernel......: Linux 5.11.0-25-generic

  Uptime......: up 1 day, 3 hours, 55 minutes
  Load........: 0.02 (1m), 0.06 (5m), 0.04 (15m)
  Processes...: 107 (root), 22 (user), 129 (total)

  CPU.........: AMD EPYC 7601 32-Core Processor (1 vCPU)
  Memory......: 355Mi used, 395Mi avail, 972Mi total

disk usage:
  /                              59% used out of  26G
  [==================================================]

lighttpd Configuration

I have attached my full lighttpd.conf file generated with lighttpd -p -f /etc/lighttpd/lighttpd.conf > ~/lighttpd.conf to this post. I will also copy here a couple of the vhosts I have set up:

root@webserver:~# cat /etc/lighttpd/vhosts.d/putaiao.nz.conf
$HTTP["host"] =~ "(^|.)putaiao.nz$" {
    server.document-root = "/srv/science.lesueur.nz/public/" 
    accesslog.filename = "/var/log/lighttpd/access.log" 

    $SERVER["socket"] == ":443" {
        ssl.engine = "enable" 
        ssl.privkey = "/etc/letsencrypt/live/putaiao.nz/privkey.pem" 
        ssl.pemfile = "/etc/letsencrypt/live/putaiao.nz/fullchain.pem" 
    }
}

$HTTP["host"] =~ "(^|.)science.lesueur.nz$" {
    $SERVER["socket"] == ":443" {
        ssl.engine = "enable" 
        ssl.privkey = "/etc/letsencrypt/live/science.lesueur.nz/privkey.pem" 
        ssl.pemfile = "/etc/letsencrypt/live/science.lesueur.nz/fullchain.pem" 
        url.redirect = ( "^/(.*)" => "https://www.putaiao.nz/$1" )
    }
    url.redirect = ( "^/(.*)" => "https://www.putaiao.nz/$1" )
}

$HTTP["host"] =~ "(^|.)putaiao.lesueur.nz$" {
    $SERVER["socket"] == ":443" {
        ssl.engine = "enable" 
        ssl.privkey = "/etc/letsencrypt/live/putaiao.lesueur.nz/privkey.pem" 
        ssl.pemfile = "/etc/letsencrypt/live/putaiao.lesueur.nz/fullchain.pem" 
        url.redirect = ( "^/(.*)" => "https://www.putaiao.nz/$1" )
    }
    url.redirect = ( "^/(.*)" => "https://www.putaiao.nz/$1" )
}
root@webserver:~# cat /etc/lighttpd/vhosts.d/adventurelog.nz.conf
$HTTP["host"] =~ "(^|.)adventurelog.nz$" {
    server.document-root = "/srv/adventurelog.nz/public/" 
    accesslog.filename = "/var/log/lighttpd/access.log" 

    url.rewrite-if-not-file = (
        "\.(?:js|ico|gif|jpg|jpeg|png|css|ttf)$" => "",
        ".*" => "/index.php?_url=$0" 
    )

    $SERVER["socket"] == ":443" {
        ssl.engine = "enable" 
        ssl.privkey = "/etc/letsencrypt/live/adventurelog.nz/privkey.pem" 
        ssl.pemfile = "/etc/letsencrypt/live/adventurelog.nz/fullchain.pem" 
    }
}

Thank you for your help and please do let me know if there is anything else I can provide to help!

error.log (243 KB) error.log /var/log/lighttpd/error.log
lighttpd.conf (88.8 KB) lighttpd.conf

Replies (6)

RE: [1.4.59] Main process exited, code=dumped, status=11/SEGV - Added by gstrauss about 2 months ago

You have not defined a default certificate for port 443.

$SERVER["socket"] belongs at the top level of the configuration. TLS negotiation happens before the host is known (or host is discovered during TLS ClientHello with SNI), but the socket is known when the connection is established.

(Aside, you also do not appear to have a default accesslog.filename set to log accesses for requests which do not match any subsection of the config)

    $HTTP["scheme"] == "http" {
        url.redirect = (
            "" => "https://${url.authority}${url.path}${qsa}",
        )
    }

    $SERVER["socket"] == ":443" {
        ssl.engine  = "enable" 
        ssl.privkey = "/etc/letsencrypt/live/putaiao.nz/privkey.pem" 
        ssl.pemfile = "/etc/letsencrypt/live/putaiao.nz/fullchain.pem" 
    }

    $HTTP["host"] =~ "(^|.)adventurelog.nz$" {
        server.document-root    = "/srv/adventurelog.nz/public/" 
        accesslog.filename      = "/var/log/lighttpd/access.log" 
        url.rewrite-if-not-file = (
            "\.(?:js|ico|gif|jpg|jpeg|png|css|ttf)$" => "",
            ".*"                                     => "/index.php?_url=$0",
        )
        ssl.privkey = "/etc/letsencrypt/live/adventurelog.nz/privkey.pem" 
        ssl.pemfile = "/etc/letsencrypt/live/adventurelog.nz/fullchain.pem" 
    }

    $HTTP["host"] =~ "(^|.)canterburyplumbing.co.nz$" {
        server.document-root    = "/srv/canterburyplumbing.co.nz/public/" 
        accesslog.filename      = "/var/log/lighttpd/access.log" 
        url.rewrite-if-not-file = (
            "\.(?:js|ico|gif|jpg|jpeg|png|css|ttf)$" => "",
            ".*"                                     => "/index.php?_url=$0",
        )
        ssl.privkey = "/etc/letsencrypt/live/canterburyplumbing.co.nz/privkey.pem" 
        ssl.pemfile = "/etc/letsencrypt/live/canterburyplumbing.co.nz/fullchain.pem" 
    }

    $HTTP["host"] =~ "(^|.)finn.lesueur.nz$" {
        server.document-root = "/srv/finn.lesueur.nz/public/" 
        accesslog.filename   = "/var/log/lighttpd/access.log" 
        ssl.privkey = "/etc/letsencrypt/live/finn.lesueur.nz/privkey.pem" 
        ssl.pemfile = "/etc/letsencrypt/live/finn.lesueur.nz/fullchain.pem" 
    }

    $HTTP["host"] =~ "(^|.)fitlytics.lesueur.nz$" {
        server.document-root    = "/srv/fitlytics.lesueur.nz/public/" 
        accesslog.filename      = "/var/log/lighttpd/access.log" 
        url.rewrite-if-not-file = (
            "\.(?:js|ico|gif|jpg|jpeg|png|css|ttf)$" => "",
            ".*"                                     => "/index.php?_url=$0",
        )
        ssl.privkey = "/etc/letsencrypt/live/fitlytics.lesueur.nz/privkey.pem" 
        ssl.pemfile = "/etc/letsencrypt/live/fitlytics.lesueur.nz/fullchain.pem" 
    }

    $HTTP["host"] =~ "(^|.)pcd.lesueur.nz$" {
        server.document-root = "/srv/pcd.lesueur.nz/public/" 
        accesslog.filename   = "/var/log/lighttpd/access.log" 
        ssl.privkey = "/etc/letsencrypt/live/pcd.lesueur.nz/privkey.pem" 
        ssl.pemfile = "/etc/letsencrypt/live/pcd.lesueur.nz/fullchain.pem" 
    }

    $HTTP["host"] =~ "(^|.)putaiao.nz$" {
        server.document-root = "/srv/science.lesueur.nz/public/" 
        accesslog.filename   = "/var/log/lighttpd/access.log" 
        ssl.privkey = "/etc/letsencrypt/live/putaiao.nz/privkey.pem" 
        ssl.pemfile = "/etc/letsencrypt/live/putaiao.nz/fullchain.pem" 
    }

    $HTTP["host"] =~ "(^|.)science.lesueur.nz$" {
        url.redirect = (
            "^/(.*)" => "https://www.putaiao.nz/$1",
        )
        ssl.privkey  = "/etc/letsencrypt/live/science.lesueur.nz/privkey.pem" 
        ssl.pemfile  = "/etc/letsencrypt/live/science.lesueur.nz/fullchain.pem" 
    }

    $HTTP["host"] =~ "(^|.)putaiao.lesueur.nz$" {
        url.redirect = (
            "^/(.*)" => "https://www.putaiao.nz/$1",
        )
        ssl.privkey  = "/etc/letsencrypt/live/putaiao.lesueur.nz/privkey.pem" 
        ssl.pemfile  = "/etc/letsencrypt/live/putaiao.lesueur.nz/fullchain.pem" 
    }

    $HTTP["host"] =~ "(^|.)sisscc.lesueur.nz$" {
        server.document-root    = "/srv/sisscc.lesueur.nz/public/" 
        accesslog.filename      = "/var/log/lighttpd/access.log" 
        url.rewrite-if-not-file = (
            "\.(?:js|ico|gif|jpg|jpeg|png|css|ttf)$" => "",
            ".*"                                     => "/index.php?_url=$0",
        )
        ssl.privkey = "/etc/letsencrypt/live/sisscc.lesueur.nz/privkey.pem" 
        ssl.pemfile = "/etc/letsencrypt/live/sisscc.lesueur.nz/fullchain.pem" 
    }

RE: [1.4.59] Main process exited, code=dumped, status=11/SEGV - Added by finn@lesueur.nz about 2 months ago

Pō mārie - many thanks for your feedback gstrauss!

I have defined accesslog.filename at the global level, and thus removed the now redundant lines in each vhost.

I have also gone through and edited the SSL configurations as you suggested. I was unaware that I could enable the SSl engine once - that makes a lot of sense though. My new lighttpd.conf is now attached.

With regards to the default SSL certificate, I have just used the certificate for the website that will get the most traffic (putaiao.nz), but I don't think I understand what repercussions that will have if somehow that certificate gets used for a website it is not authorized for. I imagine it will just not load?

I will wait and see if lighttpd crashes overnight and how the logs look in the morning.

Thank you again!

RE: [1.4.59] Main process exited, code=dumped, status=11/SEGV - Added by gstrauss about 2 months ago

With regards to the default SSL certificate, I have just used the certificate for the website that will get the most traffic (putaiao.nz), but I don't think I understand what repercussions that will have if somehow that certificate gets used for a website it is not authorized for. I imagine it will just not load?

FYI: If the client validates the certificate (as the client should) against the requested host, the client will report a certificate mismatch to the user if the requested host does not match the certificate subject (or server alternative names (SAN) in the certificate). For example curl will reject a mismatched certificate unless the certificate validation is disabled with curl -k

RE: [1.4.59] Main process exited, code=dumped, status=11/SEGV - Added by finn@lesueur.nz about 2 months ago

Okay, that's excellent, thanks again gstrauss! I believe the changes have fixed my crashing issue - I've had 24hr uptime just now.

My only query that is left is about these error.log lines:

hello:unsupported protocol
2021-07-30 08:35:26: mod_openssl.c.3095) SSL: 1 error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol
2021-07-30 08:35:26: mod_openssl.c.3095) SSL: 1 error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol
2021-07-30 08:40:21: mod_openssl.c.3095) SSL: 1 error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol
2021-07-30 08:41:06: mod_openssl.c.3095) SSL: 1 error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol
2021-07-30 08:45:27: mod_openssl.c.3095) SSL: 1 error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol
2021-07-30 08:46:26: mod_openssl.c.3095) SSL: 1 error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol
2021-07-30 08:46:58: mod_openssl.c.3095) SSL: 1 error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol
2021-07-30 08:48:16: mod_openssl.c.3095) SSL: 1 error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol
2021-07-30 08:48:16: mod_openssl.c.3095) SSL: 1 error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol
2021-07-30 08:48:17: mod_openssl.c.3095) SSL: 1 error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol

I imagine that someone is trying to connect using an older unsupported (disabled) TLS protocol. Do you have any suggestions about what I could do to improve this?

New lighttpd.conf attached.

RE: [1.4.59] Main process exited, code=dumped, status=11/SEGV - Added by gstrauss about 2 months ago

Looking at your lighttpd.conf, I see that you added "MinProtocol" => "TLSv1.3". There are probably clients trying to connect to you that do not support TLSv1.3. You should probably test with "MinProtocol" => "TLSv1.2", which is the default in lighttpd. If you do not care about older clients, then leave "MinProtocol" => "TLSv1.3" and ignore the error trace.

See lighttpd TLS docs for more details.

RE: [Solved] [1.4.59] Main process exited, code=dumped, status=11/SEGV - Added by finn@lesueur.nz about 2 months ago

Okay, got it! Too aggressive of a requirement, that makes a lot of sense.

My thanks again. I hit you up on PayPal with a little thanks for your time. Best of luck with your future development!

Mā te wā,
Finn

    (1-6/6)