Lighttpd

Thank you for taking the time to try to narrow this down.

lighttpd is single-threaded and mod_authn_ldap is blocking. It is also independent of mod_cgi. However, mod_authn_ldap holds open the connection to the ldap server for reuse. My first thought is that ldap is not setting FD_CLOEXEC on its connection fd, or lighttpd should be doing something additional when mod_cgi calls fork().

bjornfor: my apologies for taking so long to get to this after you put in a substantial effort to provide instructions how to reproduce the issue. Thank you very much.

Leveraging your instructions, I have reproduced the issue.

When lots of requests are made in parallel, the "ldap_sasl_bind_s(): Can't contact LDAP server" failure occurs when ldap_sasl_bind_s() aborts after being interrupted by SIGCHLD from exiting CGI processes.

Here is a short-term workaround while I look around to see if there are better solutions to get the desired behavior from ldap_sasl_bind_s():

--- a/src/mod_authn_ldap.c
+++ b/src/mod_authn_ldap.c
@@ -8,6 +8,7 @@
 #include "plugin.h" 

 #include <errno.h>
+#include <signal.h>
 #include <stdlib.h>
 #include <string.h>

@@ -444,11 +445,18 @@ static int mod_authn_ldap_bind(server *srv, LDAP *ld, const char *dn, const char

     /* RFE: add functionality: LDAP_SASL_EXTERNAL (or GSS-SPNEGO, etc.) */

+    sigset_t nsigs, osigs;
+    sigemptyset(&nsigs);
+    sigaddset(&nsigs, SIGCHLD);
+    sigprocmask(SIG_BLOCK, &nsigs, &osigs);
+
     ret = ldap_sasl_bind_s(ld,dn,LDAP_SASL_SIMPLE,&creds,NULL,NULL,NULL);
     if (ret != LDAP_SUCCESS) {
         mod_authn_ldap_err(srv, __FILE__, __LINE__, "ldap_sasl_bind_s()", ret);
     }

+    sigprocmask(SIG_SETMASK, &osigs, NULL);
+
     return ret;
 }

Adding a caching layer (#2805) would likely reduce the occurrence of interruption by SIGCHLD, but would not eliminate it.

Another option: if I manually repeat the attempt to ldap_sasl_bind_s() for one retry if it fails, then the problem did not recur for me. However, there is still a chance that it would fail on a busy server handling lots of CGI.

--- a/src/mod_authn_ldap.c
+++ b/src/mod_authn_ldap.c
@@ -446,7 +446,10 @@ static int mod_authn_ldap_bind(server *srv, LDAP *ld, const char *dn, const char

     ret = ldap_sasl_bind_s(ld,dn,LDAP_SASL_SIMPLE,&creds,NULL,NULL,NULL);
     if (ret != LDAP_SUCCESS) {
-        mod_authn_ldap_err(srv, __FILE__, __LINE__, "ldap_sasl_bind_s()", ret);
+        ret = ldap_sasl_bind_s(ld,dn,LDAP_SASL_SIMPLE,&creds,NULL,NULL,NULL);
+        if (ret != LDAP_SUCCESS) {
+            mod_authn_ldap_err(srv, __FILE__, __LINE__, "ldap_sasl_bind_s()", ret);
+        }
     }

     return ret;

I am not against doing both of the above, though I would prefer if there were a way for Cyrus SASL to be configured to handle EINTR. I am sure there are good reasons for the way the blocking Cyrus SASL routines handle EINTR, e.g. so that it can be interrupted/cancelled by Ctrl-C in a client program, but in this case, that behavior is not desirable. lighttpd is configured to set SIGCHLD handler with SA_RESTART but ldap_sasl_bind_s() must be in a non-restartable system call when the SIGCHLD is received.

gstrauss: Thanks for the update and preliminary patches!

Thank you!

(I've tested lighttpd v1.4.54 in my bug-reproduction-framework (https://github.com/bjornfor/lighttpd-auth-ldap-issue) and cannot provoke the failure anymore.)