Project

General

Profile

[Solved] lighttpd dies with "buffer.c assertion failed" - probably my config error

Added by thatseattleguy about 1 month ago

OS: RHEL 8.9 / lighttpd/1.4.67 (ssl)

Our internal configuration rewrites URLs from certain on-premise devices, based on their IPs, to keep them from seeing things they shouldn't. This worked fine for a long time:

$HTTP["remoteip"] =~ "^192\.168\.1\d*\.(..)$" {
   url.rewrite-once = ( "^/R..\.html$" => "/R%1.html" )
}

Today, I tried to modify lighttpd.conf to add a "safe" range of IPs that would NOT be rewritten, as follows:

$HTTP["remoteip"] =~ "^192\.168\.1\d*\.(..)$" {
## allow 192.168.1.10-192.168.1.19 to go anywhere; all other 192.168.1.x trigger rewrite
   $HTTP["remoteip"] !~ "^192\.168\.1\.1.$" {
      url.rewrite-once = ( "^/R..\.html$" => "/R%1.html" )
   }
}

So the ONLY difference is wrapping the rewrite-once in an additional conditional block - everything else the same. And this appeared to work at first: it passed the normal "lighttpd -tt -f lighttpd.conf" test without any errors, and started up as a daemon from systemctl, and even seemed to serve requests properly.

The problem occurred a few hours later, in production use, when the daemon kept dying and respawning. In /var/log/messages I found:

*Jan 29 18:44:13 field lighttpd[46214]: buffer.c.58: assertion failed: ((void *)0) != b->ptr*
Jan 29 18:44:13 field systemd[1]: lighttpd.service: Main process exited, code=killed, status=6/ABRT
Jan 29 18:44:13 field systemd[1]: lighttpd.service: Failed with result 'signal'.
Jan 29 18:44:13 field systemd[1]: lighttpd.service: Service RestartSec=100ms expired, scheduling restart.

I know this is probably user error re: the config file, but any pointers as to what I might have done wrong would be so appreciated.


Replies (11)

RE: lighttpd dies with "buffer.c assertion failed" - probably my config error - Added by gstrauss about 1 month ago

Jan 29 18:44:13 field lighttpd[46214]: buffer.c.58: assertion failed: ((void *)0) != b->ptr

That line in lighttpd buffer.c indicates that the call to realloc() (above that line) failed, which should not happen unless you have set memory limits on lighttpd and lighttpd is hitting those memory limits.

Is your system running out of memory?

Does the crash happen with lighttpd 1.4.73, the latest current lighttpd release?


Your use of %1 is incorrect in the regexes. I think you meant this (with your comment):

$HTTP["remoteip"] !~ "^192\.168\.1\.1.$" {
  ## allow 192.168.1.10-192.168.1.19 to go anywhere; all other 192.168.1.x trigger rewrite
  $HTTP["remoteip"] =~ "^192\.168\.1\.(\d+)$" {
    url.rewrite-once = ( "^/R..\.html$" => "/R%1.html" )
  }
}


Separately, if you do not need to capture part of the IP address for substitution in url.rewrite-once, then for a cleaner and slightly faster config, you should use CIDR masks where you can, e.g. (if appropriate)

$HTTP["remoteip"] == "192.168.0.0/16" {
  ## allow 192.168.1.10-192.168.1.19 to go anywhere; all other 192.168.1.x trigger rewrite
  $HTTP["remoteip"] !~ "^192\.168\.1\.1.$" {
    url.rewrite-once = ( "^/R..\.html$" => "/Rxx.html" )
  }
}

RE: lighttpd dies with "buffer.c assertion failed" - probably my config error - Added by thatseattleguy about 1 month ago

Thanks for the quick reply!

(1) I actually do need to capture part of the IP address; that's the basis of the rewrite rule (the final two digits of the incoming IP address determines the allowable files for the device - strange but this is what the client says they need...)

(2) Your version of the rules (test first for !~ the "universal" address, then =~ for the "restricted" address - makes much more sense and I'm sure it will correct the %1 problem. Thanks again.

(3) FWIW, the system is nowhere near running out of memory - cat /proc/meminfo shows 10GB MemFree and 11GB MemAvailable (out of 12GB physical memory on the system). And there's nothing else running on the machine besides Apache and lighttd, really.

(4) I will attempt to upgrade lighttpd to the latest version (I generally use only what the vendor supplies as the latest version on my client machines, to make support easier, so 1.4.67 is just what Rocky Linux happens to be providing right now for 8.9 systems.

(5) I will go back and review my old (two years+) postings on this same subject - sorry!

Thank you for the help and I will let you know if your version works and/or if 1.4.73 has different behavior. And of course thank you for Lightly!

RE: lighttpd dies with "buffer.c assertion failed" - probably my config error - Added by gstrauss about 1 month ago

If you have the space for logging, you might try to capture the requests preceding the crash by adding these in lighttpd.conf and restarting lighttpd
debug.log-request-header = "enable"
debug.log-response-header = "enable"

RE: lighttpd dies with "buffer.c assertion failed" - probably my config error - Added by thatseattleguy about 1 month ago

Sure thing! I do have plenty of space for logging - this is a very low volume server - and I'm happy to do that if it would help. (I already had some debugging turned on earlier for tracing why my conditionals weren't working, so I'm fine enabling that again.)

I just need to time the logging when I can run it with my earlier code that had the problem - so that if/when it crashes, it doesn't impact the users too much. Can likely do that later in the week when I'm back at the customer site.

RE: lighttpd dies with "buffer.c assertion failed" - probably my config error - Added by gstrauss 21 days ago

Any update? I am planning to release lighttpd 1.4.74 soon and if there is something to be fixed in lighttpd to avoid the crash, I would like to fix it. As you noted in your title, there is probably some configuration you are using that is triggering the ABRT, so it would be useful to see your entire config
lighttpd -f /etc/lighttpd/lighttpd.conf -p
with any sensitive information redacted with XXXXXX.

Also, there is a strong possibility that whatever your config is tripping over has already been fixed in lighttpd 1.4.73, which is 6 full lighttpd releases and over a year newer than OS: RHEL 8.9 / lighttpd/1.4.67 (ssl)

RE: lighttpd dies with "buffer.c assertion failed" - probably my config error - Added by thatseattleguy 21 days ago

GS - I'm sorry; I replied a few weeks ago to the email that I received with your reply, but now I realize that my response bounced from your mail server:

<redmine@lighttpd.net>: Recipient address rejected: User unknown in local recipient table (in reply to RCPT TO command)

....so I expect you never saw it.

Give me a moment to retrieve what I wrote and make sure it all applies, and post that directly here.

RE: lighttpd dies with "buffer.c assertion failed" - probably my config error - Added by thatseattleguy 21 days ago

Here is the -p output for the original .conf file, the one that failed with the buffer.c message, per your request. (attached)

Noting this is a production machine and I've not received permission from the client to try your suggested change (to fix my incorrect use of %1 as seen in this original conf) nor to try to upgrade to use 1.4.73 (which I would need to compile from sources).

If I can get them to allow me to do one or both of those, I will report back on the results. In the interim, let me know if any more information would be helpful to you.

lighttpd-p.rtf (101 KB) lighttpd-p.rtf output from "lighttpd -f /etc/lighttpd/lighttpd.conf -p"

RE: lighttpd dies with "buffer.c assertion failed" - probably my config error - Added by gstrauss 21 days ago

I would think the client would let you make changes if you were able to explain that the current syntax you are using with %1 does not work as intended. The misuse of %1 when the enclosing condition is not a capturing regex is probably what is resulting in the crash. I am digging further into this.

I see that mod_magnet is used
magnet.attract-physical-path-to = ("/etc/lighttpd/xxxxx.lua")
FYI: all of the logic around string matching IP addresses and rewriting the URL could be done in that lua script as an alternative to using lighttpd.conf syntax. See ModMagnetExamples

Besides those, nothing immediately jumps out at me in the syntax, though it is also possible for bad code in /etc/lighttpd/xxxxx.lua to change lighttpd internals and cause a crash. I would be interested in seeing the contents of that script, but only if it is possible for you share that without sharing sensitive client info.


Some general feedback on the config:

This should be removed. Back in the 200x, there was a bug in some PDF viewers which mishandled Range responses. That is ancient. Supporting Range requests for PDFs should be permitted.

    $HTTP["url"] =~ "\.pdf$" {
        server.range-requests = "disable" 
    }

When using lighttpd defaults (recommended), it is also recommended not to repeat those values in the configs.
These are all defaults in lighttpd (when run on Linux):

    server.event-handler            = "linux-sysepoll" 
    server.network-backend          = "sendfile" 
    server.stat-cache-engine        = "simple" 
    server.follow-symlink           = "enable" 
    server.upload-dirs              = ("/var/tmp")

A production server should allow more fds. server.max-fds = 16384 is recommended.
server.max-fds must be at least twice server.max-connections

    server.max-fds                  = 1024
    server.max-connections          = 1024

As reported in the lighttpd error log: 2024-02-11 04:47:48: (server.c.1996) can't have more connections than fds/2: 1024 1024

RE: lighttpd dies with "buffer.c assertion failed" - probably my config error - Added by gstrauss 21 days ago

Sorry. The crash is due to a repeat of the same bug as in #2892. It is triggered by your invalid syntax, but lighttpd still should not crash.
The same bug was fixed in lighttpd 1.4.50 in commit f4f13745, but reintroduced in lighttpd 1.4.62 in commit 7db817c59

Patch against lighttpd 1.4.73

--- a/src/keyvalue.c
+++ b/src/keyvalue.c
@@ -198,7 +198,7 @@ static void pcre_keyvalue_buffer_append_match(buffer *b, const pcre_keyvalue_ctx
 static void pcre_keyvalue_buffer_append_ctxmatch(buffer *b, const pcre_keyvalue_ctx *ctx, unsigned int num, int flags) {
     const struct cond_match_t * const cache = ctx->cache;
     if (!cache) return; /* no enclosing match context */
-    if (num < (unsigned int)cache->captures) {
+    if ((int)num < cache->captures) {
       #ifdef HAVE_PCRE2_H
         const PCRE2_SIZE *ovec = (PCRE2_SIZE *)cache->matches;
       #elif defined(HAVE_PCRE_H)

There is no longer a need to enable extra debugging on the production machine, but you should fix your invalid use of %1 in lighttpd.conf

RE: [Solved] lighttpd dies with "buffer.c assertion failed" - probably my config error - Added by gstrauss 19 days ago

@thatseattleguy

You need to modify url.rewrite-once for security. See mod_rewrite. The url.rewrite-once match is against the URL, which includes both url-path and query-string, if present. The rule as you have written it will not match if someone makes a request GET /Rxx.html?query-string HTTP/1.1, so someone can bypass the url.rewrite-once rule to read any Rxx.html file they like if they add a query-string (/Rxx.html?query-string) or path-info (/Rxx.html/path-info) or both. It seems to me that a prefix-match is sufficient:

$HTTP["remoteip"] !~ "^192\.168\.3\.1.$" {
    $HTTP["remoteip"] =~ "^192\.168\.3\d*\.[12]?(..)$" {
        url.rewrite-once = ( "^/R..\.html" => "/R%1.html" )
    }
    $HTTP["remoteip"] =~ "^192\.168\.2\d*\." {
        url.rewrite-once = ( "^/R..\.html" => "/index.html" )
    }
}

The prefix match is the better solution, but you can consider globally disabling path-info on all files to reject requests to /Rxx.html/path-info
server.feature-flags += ("server.http-pathinfo" => "disable")
Globally disabling path-info on files is not a good idea if path-info is leveraged elsewhere on the site. Also note that path-info is separate from query-string and query-string is still allowed.

Whatever you choose, you should test that it works as desired and prevents the behavior that is undesired!

    (1-11/11)