[Solved] CGI fork fails after some days
Added by foobar0815 3 months ago
Hi,
I updated lighttpd from lighttpd/1.4.45 to ighttpd/1.4.76. The server runs fine for a while but fails to start new cgi processes after a while.
2024-08-23 14:16:29: (server.c.1939) server started (lighttpd/1.4.76) 2024-09-02 09:36:00: (mod_cgi.c.990) fork/spawn /srv/www/cgi/wait: Resource temporarily unavailable 2024-09-02 09:36:00: (mod_cgi.c.990) fork/spawn /srv/www/cgi/wait: Resource temporarily unavailable 2024-09-02 09:36:00: (mod_cgi.c.990) fork/spawn /srv/www/cgi/wait: Resource temporarily unavailable 2024-09-02 09:36:00: (mod_cgi.c.990) fork/spawn /srv/www/cgi/wait: Resource temporarily unavailable ...
Sep 02 09:36:00 pc lighttpd[15061]: 192.168.178.66 192.168.178.30 - [02/Sep/2024:09:36:00 +0200] "POST /visu/wait?0 HTTP/1.1" 200 45 "http://192.168.178.30/visu/index.fcgi?22&lang=de" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" Sep 02 09:36:00 pc lighttpd[15061]: 192.168.178.66 192.168.178.30 - [02/Sep/2024:09:36:00 +0200] "POST /visu/wait?0 HTTP/1.1" 500 182 "http://192.168.178.30/visu/index.fcgi?22&lang=de" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
Obviously old processes are not cleaned up:
ps aux | grep www-data | grep defunct| wc -l 3541 ps aux | grep www-data | grep defunct| head -n 1 www-data 365 0.0 0.0 0 0 ? Z< Aug31 0:00 [controlDaemon] <defunct>
Any ideas if this is a bug or a configuration issue?
/usr/local/sbin/lighttpd -V
lighttpd/1.4.76 (ssl) - a light and fast webserver
Event Handlers:
- select (generic)
+ poll (Unix)
+ epoll (Linux)
- /dev/poll (Solaris)
- eventports (Solaris)
- kqueue (FreeBSD)
Network handler:
+ linux-sendfile
- freebsd-sendfile
- darwin-sendfile
- solaris-sendfilev
+ writev
+ write
+ mmap support
Features:
+ IPv6 support
+ zlib support
- zstd support
+ bzip2 support
- brotli support
+ crypt support
+ OpenSSL support
- mbedTLS support
- NSS crypto support
- GnuTLS support
- WolfSSL support
- Nettle support
+ PCRE support
- MySQL support
- PgSQL support
- DBI support
- Kerberos support
- LDAP support
- PAM support
+ inotify support
- LUA support
- xml support
- SQLite support
- Y2038 support (unsafe 32-bit signed time_t)
config {
var.CWD = "/root"
var.PID = 10960
mimetype.assign = (
...
)
server.document-root = "/srv/www/htdocs/"
server.upload-dirs = ("/tmp/lighttpd-uploads")
server.errorlog = "/tmp/lighttpd-error.log"
server.pid-file = "/var/run/lighttpd.pid"
server.username = "www-data"
server.groupname = "www-data"
server.port = 80
index-file.names = ("index.php", "index.html", "index.lighttpd.html")
url.access-deny = ("~", ".inc")
static-file.exclude-extensions = (".php", ".pl", ".fcgi")
accesslog.use-syslog = 1
alias.url = (
"/visu/" => "/srv/www/cgi/",
)
deflate.allowed-encodings = ("gzip", "deflate")
deflate.mimetypes = ("text/plain", "text/html", "text/javascript", "text/css", "text/xml")
deflate.cache-dir = "/tmp/lighttpd-cache"
server.modules = (
"mod_access",
"mod_alias",
"mod_redirect",
"mod_accesslog",
"mod_deflate",
"mod_fastcgi",
"mod_cgi",
"mod_proxy",
"mod_openssl",
)
if $SERVER["socket"] == "[::]:80" {
# block 1
} # end of $SERVER["socket"] == "[::]:80"
if $HTTP["url"] =~ "/visu/" {
# block 2
fastcgi.server = (
".fcgi" => (
(
"socket" => "/tmp/visu-cgi.socket",
"max-procs" => 1,
"mode" => "responder",
"check-local" => "enable",
"bin-path" => "/srv/www/cgi/index.fcgi",
),
),
)
cgi.assign = (
"" => "",
)
} # end of $HTTP["url"] =~ "/visu/"
if $HTTP["url"] =^ "/webrtc/" {
# block 3
proxy.header = (
"map-urlpath" => (
"/webrtc/" => "/",
),
)
proxy.server = (
"" => (
(
"host" => "127.0.0.1",
"port" => 8889,
),
),
)
} # end of $HTTP["url"] =^ "/webrtc/"
if $SERVER["socket"] == "0.0.0.0:443" {
# block 4
ssl.engine = "enable"
ssl.pemfile = "/etc/ssl/private/http.pem"
ssl.ca-file = "/etc/ssl/certs/ca.crt"
} # end of $SERVER["socket"] == "0.0.0.0:443"
}
Replies (5)
RE: CGI fork fails after some days - Added by foobar0815 3 months ago
If I restart the lighttpd process, everything looks fine. No zombies remain after CGI processes are forked.
RE: CGI fork fails after some days - Added by foobar0815 3 months ago
The process which started lighttpd blocks several signals, including SIGCHLD. Lighttpd and mod_cgi seem to not enable the signal themselves, so the cleanup handler was never called.
When I restarted lighttpd by hand, the default signal mask did not clock SIGCHLD.
Is there a reason why lighttpd does not enable SIGCHLD?
RE: CGI fork fails after some days - Added by gstrauss 2 months ago
The process which started lighttpd blocks several signals, including SIGCHLD.
It should be considered a bug in your process which starts lighttpd if important signals are not reset to defaults in child processes.
Lighttpd and mod_cgi seem to not enable the signal themselves, so the cleanup handler was never called.
When I restarted lighttpd by hand, the default signal mask did not clock SIGCHLD.
Is there a reason why lighttpd does not enable SIGCHLD?
Some ways of using lighttpd include one-shot mode, so excessive system calls are avoided. If you want SIGCHLD to work as expected in lighttpd, you should unblock it in the calling (child) process before executing lighttpd. lighttpd is a daemon and it is reasonable to expect that important signals are already set to the defaults for daemon processes.
Could lighttpd unblock all signals that lighttpd expects to handle? Yes, lighttpd could call sigprocmask. While that is portable nowadays, sigprocmask was less portable when lighttpd was first written. Also, not every use of lighttpd uses child processes, so unblocking SIGCHLD might add extraneous system calls for those uses.
RE: CGI fork fails after some days - Added by gstrauss 2 months ago
You can try this patch if you like:
--- a/src/server.c
+++ b/src/server.c
@@ -325,6 +325,7 @@ static BOOL WINAPI ConsoleCtrlHandler(DWORD dwType)
static void server_main_setup_signals (void) {
#ifdef HAVE_SIGACTION
+
struct sigaction act;
memset(&act, 0, sizeof(act));
sigemptyset(&act.sa_mask);
@@ -368,7 +369,24 @@ static void server_main_setup_signals (void) {
/* it should be safe to restart syscalls after SIGCHLD */
act.sa_flags |= SA_RESTART | SA_NOCLDSTOP;
sigaction(SIGCHLD, &act, NULL);
+
+ #ifdef SIG_UNBLOCK
+ /* paranoia; unblock signals just in case inherited from calling process */
+ /* INT and HUP might be blocked for various reasons, and if blocked, the
+ * features associated with those signals will not be available in lighttpd.
+ * lighttpd child processes will inherit the blocked signal mask, and that
+ * may include any signals, including SIGINT, SIGHUP, SIGPIPE, etc. */
+ sigset_t * const sigs = &act.sa_mask; /*(prev initialized to empty set)*/
+ sigaddset(sigs, SIGCHLD);
+ sigaddset(sigs, SIGALRM);
+ sigaddset(sigs, SIGTERM);
+ sigaddset(sigs, SIGUSR1);
+ sigprocmask(SIG_UNBLOCK, sigs, NULL);
+ /*(multithreaded process must use pthread_sigmask instead of sigprocmask)*/
+ #endif
+
#elif defined(HAVE_SIGNAL)
+
#ifndef _WIN32
/* ignore the SIGPIPE from sendfile() */
signal(SIGPIPE, SIG_IGN);
@@ -388,6 +406,7 @@ static void server_main_setup_signals (void) {
#ifndef _MSC_VER
signal(SIGBUS, sys_setjmp_sigbus);
#endif
+
#endif
}