Lighttpd

That is another message:

lighttpd: unbind.c:46: ldap_unbind_ext: Assertion `LDAP_VALID( ld )' failed.

From what version did you update from? (i.e. did this previously work?) Thx.

I updated from 1.4.55.3 (Arch Linux).

Forgot - it was working well until 1.4.56. I did not update anything else on this system before the update until now.

I don't have a test environment set up for LDAP at the moment, so this may take a bit of time for me to troubleshoot.

It appears that assert() is being called in the underlying LDAP library. lighttpd is possibly passing in a NULL value, but I don't immediately see where that might be (after a quick review of the code changes since lighttpd 1.4.55).

If this is easily reproducible for you, would you please post a stack trace? Omitting any sensitive info, would you share your lighttpd.conf? lighttpd -f /etc/lighttpd/lighttpd.conf -p

Are there any other possibly relevant changes that have been made to the system since it last worked? Maybe an upgraded LDAP library?

I see an inconsistency in mod_authn_ldap.c, but this inconsistency existed in lighttpd 1.4.55, too, and wouldn't cause a crash.

--- a/src/mod_authn_ldap.c
+++ b/src/mod_authn_ldap.c
@@ -507,7 +507,7 @@ static LDAP * mod_authn_ldap_host_init(log_error_st *errh, plugin_config_ldap *s
         /* if no CA file is given, it is ok, as we will use encryption
          * if the server requires a CAfile it will tell us */
         if (s->auth_ldap_cafile) {
-            ret = ldap_set_option(NULL, LDAP_OPT_X_TLS_CACERTFILE,
+            ret = ldap_set_option(ld, LDAP_OPT_X_TLS_CACERTFILE,
                                   s->auth_ldap_cafile);
             if (LDAP_OPT_SUCCESS != ret) {
                 mod_authn_ldap_err(errh, __FILE__, __LINE__,

Ok I integrated this patch - but it did not solve the problem as you predicted.

I did indeed update libldap on November 29 and I updated lighttpd on December 13.
I guess it was compiled with a previous version before - I'm going to check this first. (pacman log: upgraded libldap (2.4.53-1 -> 2.4.54-1) )

I will try to strace later (I lack experience in this topic), but I need to block traffic first - there are a lot of permanently connected clients producing lots of noise and people just went home - I'm going to wait a little longer.

This part of the config I can share:

    server.tag                       = "********" 
    server.username                  = "http" 
    server.groupname                 = "http" 
    server.document-root             = "/srv/http/default" 
    server.errorlog                  = "/var/log/lighttpd/error.log" 
    server.feature-flags             = (
        "server.h2proto" => "enable",
        "server.h2c"     => "enable",
        # 2
    )
    server.pid-file                  = "/run/lighttpd/lighttpd.pid" 
    dir-listing.activate             = "disable" 
    dir-listing.encoding             = "utf-8" 
    accesslog.filename               = "/var/log/lighttpd/access.log" 
    deflate.allowed-encodings        = ("brotli", "gzip", "deflate")
    deflate.cache-dir                = "/srv/lighttpd-cache" 
    deflate.compression-level        = 9
    deflate.mimetypes                = (
        "text/html",
        "text/plain",
        "text/javascript",
        "application/javascript",
        "text/x-js",
        "text/json",
        "application/json",
        "application/x-javascript",
        "text/css",
        "text/xml",
        "application/xml",
        "text/xml+rss",
        "application/xml+rss",
        # 13
    )
    index-file.names                 = ("index.php", "index.html", "index.lighttpd.html")
    server.modules                   = (
        "mod_auth",
        "mod_authn_ldap",
        "mod_access",
        "mod_accesslog",
        "mod_alias",
        "mod_deflate",
        "mod_fastcgi",
        "mod_openssl",
        "mod_proxy",
        "mod_redirect",
        "mod_rewrite",
        "mod_setenv",
        "mod_status",
        # 13
    )
    static-file.exclude-extensions   = (".php", ".inc")
    fastcgi.server                   = (
        ".php" => (
            "localhost" => (
                "socket"                => "/run/php-fpm/php-fpm.sock",
                "broken-scriptfilename" => "enable",
                # 2
            ),
        ),
    )
    ssl.disable-client-renegotiation = "enable" 
    ssl.cipher-list                  = "HIGH:!TLSv1.1:!TLSv1:!SSLv3:!SSLv2" 
    ssl.honor-cipher-order           = "enable" 
    auth.backend                     = "ldap" 
    auth.backend.ldap.hostname       = "dc.example.local" 
    auth.backend.ldap.starttls       = "enable" 
    auth.backend.ldap.ca-file        = "/etc/ssl/certs/CA.pem" 
    auth.backend.ldap.allow-empty-pw = "disable" 
    auth.backend.ldap.base-dn        = "OU=Users,DC=example,DC=local" 
    auth.backend.ldap.bind-dn        = "CN=lookup,DC=example,DC=local" 
    auth.backend.ldap.bind-pw        = "*******" 

    auth.backend.ldap.filter = "(&(objectclass=user)(objectclass=person)(objectclass=top)(sAMAccountName=$)(memberOf:1.2.840.113556.1.4.1941:=cn=XYZ,ou=Groups,dc=example,dc=local)(!(UserAccountControl:1.2.840.113556.1.4.803:=2)))" 
    auth.require             = (
        "/download/protected" => (
            "method"  => "basic",
            "realm"   => "Protected Area",
            "require" => "valid-user",
            # 3
        ),
    )

I tried compiling with older versions - same error...

I tried compiling with older versions - same error...

I presume you mean older versions of libldap

Getting a stack trace is different from strace. To create a basic test environment, take your existing lighttpd.conf and change the port on which lighttpd is listening so that it does not interfere with the existing listening interfaces. Then change all the filesystem paths to logs and pid files so that the test-lighttpd.conf does not overwrite the same paths. In the config that you shared, those are:

    server.errorlog                  = "/var/log/lighttpd/error.log" 
    server.pid-file                  = "/run/lighttpd/lighttpd.pid" 
    accesslog.filename               = "/var/log/lighttpd/access.log" 

Well, this is a start. Now to find out why.

(gdb) where
#0  0x00007ffff7d4a9d5 in raise () from /lib64/libc.so.6
#1  0x00007ffff7d338a4 in abort () from /lib64/libc.so.6
#2  0x00007ffff7d33789 in __assert_fail_base.cold () from /lib64/libc.so.6
#3  0x00007ffff7d43026 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff7b3db0e in ldap_pvt_search () from /lib64/libldap-2.4.so.2
#5  0x00007ffff7b4078d in ldap_pvt_search_s () from /lib64/libldap-2.4.so.2
#6  0x00007ffff7b40860 in ldap_search_ext_s () from /lib64/libldap-2.4.so.2
#7  0x00007ffff7b94eab in mod_authn_ldap_search (errh=0x45b490, s=0x46cf10, base=0x45e100 "dc=example,dc=com", filter=0x4b3800 "(&(objectClass=person)(sn=any))") at mod_authn_ldap.c:581
#8  0x00007ffff7b951be in mod_authn_ldap_get_dn (filter=0x4b3800 "(&(objectClass=person)(sn=any))", base=<optimized out>, s=0x46cf10, errh=0x45b490) at mod_authn_ldap.c:630
#9  mod_authn_ldap_basic (r=<optimized out>, p_d=0x46b940, require=0x46c690, username=0x4b3700, pw=0x4b37b4 "any") at mod_authn_ldap.c:737
#10 0x00007ffff7fb78ed in mod_auth_check_basic (r=0x46ca20, p_d=<optimized out>, require=0x46c690, backend=0x45a000 <http_auth_backends>) at mod_auth.c:810
...

lighttpd is not passing NULL for ld. I'll try to bisect the lighttpd git history to see if I can find the change which caused this.

(gdb) up 7
#7  0x00007ffff7b94eab in mod_authn_ldap_search (errh=0x45b490, s=0x46cf10, base=0x45e100 "dc=example,dc=com", filter=0x4b3800 "(&(objectClass=person)(sn=any))") at mod_authn_ldap.c:581
581            ret = ldap_search_ext_s(s->ldap, base, LDAP_SCOPE_SUBTREE, filter,
(gdb) print s->ldap
$1 = (LDAP *) 0x7ffff7ed1a60 <main_arena+96>
(gdb) print base
$2 = 0x45e100 "dc=example,dc=com" 
(gdb) print filter
$3 = 0x4b3800 "(&(objectClass=person)(sn=any))" 
(gdb) print attrs
$4 = {0x7ffff7b960f2 "1.1", 0x0}

Yes, I meant older versions of libldap.

Thanks for explaining and your help - this is new territory.
Do you still want me to stack trace my setup?

Not needed. Your stack would likely show the same.

It does.
(What an easy tool - I will do better next time.)

I have tracked the issue back near a03afc90. I have a hunch that there is a memory error before this point, but the changes here perturb the system enough that it shows up when libldap allocates memory. This may take me a few more hours to find the actual bug. Time to break out valgrind.

Compiled and working well - excellent :-)
Thank you very much for this rapid solution!

Sorry about the bug.

Thank you for your quick and helpful responses, and, of course, for reporting the bug.