How to debug lighttpd could not be started by start-stop-daemon.
Added by zhipengl over 6 years ago
After we upgraded lighttpd from 1.4.45 to 1.4.47 or 49 or 50 version, we found an issue.
it could not be started by start-sop-daemon when we execute /etc/init.d/lighttpd start
It shows error "daemonized server failed to start"
However, it could be started by systemctl start lighttpd
Our OS is CentOS7
Below is our init script
#!/bin/sh ### BEGIN INIT INFO # Provides: Web Server # Required-Start: networking # Required-Stop: networking # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Lighttpd Web Server # Description: Web service to serve static files and proxy ### END INIT INFO PATH=/sbin:/bin:/usr/sbin:/usr/bin DAEMON=/usr/sbin/lighttpd NAME=lighttpd DESC="Lighttpd Web Server" OPTS="-f /etc/lighttpd/lighttpd.conf" PIDFILE="/var/run/$NAME.pid" PORT="80" start() { if lsof -t -i:${PORT} 1> /dev/null 2>&1; then kill $(lsof -t -i:${PORT}) > /dev/null 2>&1 fi if [ -e $PIDFILE ]; then PIDDIR=/proc/$(cat $PIDFILE) if [ -d ${PIDDIR} ]; then echo "$DESC already running." return else echo "Removing stale PID file $PIDFILE" rm -f $PIDFILE fi fi echo -n "Checking scratch filesystem..." let -i COUNT=0 while [ ! -e /scratch ] do if [ $COUNT -ge 15 ] then echo "failed to find /scratch." exit -1 fi let COUNT=COUNT+1 sleep 1 done echo -n "Mounting scratch filesystem to chroot tmp..." umount /www/tmp >& /dev/null rm -r /scratch/lighttpd >& /dev/null mkdir -p /scratch/lighttpd mount --bind /scratch/lighttpd /www/tmp/ chown www /www/tmp/ echo -n "Starting $DESC..." start-stop-daemon --start --pidfile ${PIDFILE} -x "$DAEMON" -- $OPTS RETVAL=$? if [ $RETVAL -eq 0 ]; then echo "done." else echo "failed." fi } stop() { if [ ! -e $PIDFILE ]; then return; fi echo -n "Stopping $DESC..." start-stop-daemon --stop --quiet --pidfile ${PIDFILE} -x "$DAEMON" RETVAL=$? if [ $RETVAL -eq 0 ]; then echo "done." else echo "failed." fi rm -f $PIDFILE echo -n "Unmounting scratch filesystem from chroot tmp..." umount /www/tmp } status() { pid=`cat $PIDFILE 2>/dev/null` if [ -n "$pid" ]; then if ps -p $pid &>/dev/null ; then echo "$DESC is running" RETVAL=0 return else RETVAL=1 fi fi echo "$DESC is not running" RETVAL=3 } case "$1" in start) start ;; stop) stop ;; restart|force-reload|reload) stop start ;; status) status ;; *) echo "Usage: $0 {start|stop|force-reload|restart|reload|status}" RETVAL=1 ;; esac exit $RETVAL
Replies (17)
RE: How to debug lighttpd could not be started by start-stop-daemon. - Added by jbaans over 6 years ago
Hi fellow lighttpd user,
You might want to check your logs for related error messages, and check your config with
sudo lighttpd -tt -f /etc/lighttpd/lighttpd.conf
RE: How to debug lighttpd could not be started by start-stop-daemon. - Added by zhipengl over 6 years ago
Thanks for you kind help first!
Actually, in our project, I just upgrade lighttpd from 1.4.45 to 47, 49 or 50 version.
Then find this issue.
However, I find the log is very limited in below place.
1) var/log/lighttpd/ this folder is empty
2) var/log/lighttpd-access.log & var/log/lighttpd-error.log
Do you know how to enable more detail log of lighttpd, so that I can debug why it failed to start when I used latest version instead of old one?
Actually, we have some patch based on lighttpd.spec file, I also rebase it to latest version.
(It can work in 1.4.45)
such as:
+Source102: index.html.lighttpd +Source103: lighttpd.conf +Source104: lighttpd-inc.conf +Source105: lighttpd.logrotate +Source106: lighttpd-csr.conf %configure \ --libdir='%{_libdir}/lighttpd' \ %{confswitch mysql} \ - %{confswitch ldap} \ + --without-ldap \ %{confswitch attr} \ - %{confswitch openssl} \ + --with-openssl \ %{confswitch kerberos5} \ - %{confswitch pcre} \ + --with-pcre \ %{confswitch fam} \ - %{?with_webdavprops:--with-webdav-props} \ - %{?with_webdavlocks:--with-webdav-locks} \ + --without-webdav-props \ + --without-webdav-locks \ %{confswitch gdbm} \ - %{confswitch memcache} \ - %{confswitch lua} \ + --without-memcache \ + --without-lua \ + --without-bzip2 \ + --disable-static \ +CONFDIR=%{buildroot}%{_sysconfdir}/lighttpd +ROOTDIR=%{buildroot}/www + +install -d ${CONFDIR}/ssl +install -d ${ROOTDIR}/pages/dav +install -m 0640 %{SOURCE103} ${CONFDIR}/lighttpd.conf +install -m 0644 %{SOURCE104} ${CONFDIR}/lighttpd-inc.conf +install -m 0644 %{SOURCE102} ${ROOTDIR}/pages/index.html + +install -d %{buildroot}%{_sysconfdir}/logrotate.d +install -m 644 %{SOURCE105} %{buildroot}%{_sysconfdir}/logrotate.d/lighttpd + +chmod 02770 %{buildroot}%{_sysconfdir}/lighttpd @@ -301,11 +326,8 @@ fi %config %{_sysconfdir}/lighttpd/conf.d/mod.template %config %{_sysconfdir}/lighttpd/vhosts.d/vhosts.template %config(noreplace) %{_sysconfdir}/logrotate.d/lighttpd -%if %{with systemd} %{_unitdir}/lighttpd.service -%else %{_sysconfdir}/rc.d/init.d/lighttpd -%endif +%dir /www/pages/ +/www/pages/* +
Thanks again!
Zhipeng
RE: How to debug lighttpd could not be started by start-stop-daemon. - Added by jbaans over 6 years ago
try
server.modules += ( "mod_accesslog" ) accesslog.use-syslog = "enable"
restart lighttpd and check out syslog (for systemd) with
journalctl -r
RE: How to debug lighttpd could not be started by start-stop-daemon. - Added by zhipengl over 6 years ago
I have changed lighttpd.conf as you mentioned.
Unfortunately, it did not work. I could not find any information related to lighttpd by Journalctl -r
In our project, we need start lighttpd in init.d/lighttpd script
In this script, it starts lighttpd as below.
start-stop-daemon --start --pidfile ${PIDFILE} -x "$DAEMON" -- $OPTS
BTW, I can start it through
systemctl start lighttpd
RE: How to debug lighttpd could not be started by start-stop-daemon. - Added by zhipengl over 6 years ago
/etc/init.d# sudo ./lighttpd start
Checking scratch filesystem...Mounting scratch filesystem to chroot tmp...Starting Lighttpd Web Server...daemonized server failed to start; check error log for details
failed.
Still could not find where I can check valid log.
RE: How to debug lighttpd could not be started by start-stop-daemon. - Added by jbaans over 6 years ago
It seems your line
start-stop-daemon --start --pidfile ${PIDFILE} x "$DAEMON" - $OPTS
is malformed, double check this.
Note: This may be because of copy pasting here, please use < pre > tags.
Secondly, I don't understand why you are using start-stop-daemon, as your system has systemd installed. Or why both services are installed simultaneously anyway. Systemd replaces the former. Did you recently do a large upgrade?
https://unix.stackexchange.com/questions/236084/how-do-i-create-a-service-for-a-shell-script-so-i-can-start-and-stop-it-like-a-d
Lastly, your logs should be in /var/log/. Try /var/log/messages. Or run dmesg.
RE: How to debug lighttpd could not be started by start-stop-daemon. - Added by zhipengl over 6 years ago
no valid log find in /var/log actually.
Now I know that it failed in server.c
static int daemonize(void) { if (bytes <= 0) { /* closed fd (without writing) == failure in grandchild */ fputs("daemonized server failed to start; check error log for details\n", stderr); exit(-1); } }
I see too big gap between 1.4.45 and 1.4.50 for server.c
Not sure what can make it will fail here.
I just doubt, for 1.4.45, may be we not enter this function for daemonize. I'm still checking on this point.
Thanks!
RE: How to debug lighttpd could not be started by start-stop-daemon. - Added by zhipengl over 6 years ago
Hi jbaans,
I found some cue now.
Actually, the log is in /www/var/log/lighttpd-error.log
From log, it always shows below, this is the direct cause!!
2018-09-09 01:48:37: (server.c.790) opening /dev/null failed: No such file or directory 2018-09-09 01:48:37: (server.c.1447) Opening errorlog failed. Going down.
However, I still need to check why this error take place in 1.4.50. below is the related code for both.
In 1.4.50
in log_error_open() ==> else if (!srv->srvconf.dont_daemonize) { /* move STDERR_FILENO to /dev/null */ if (-1 == (errfd = fdevent_open_devnull())) { log_error_write(srv, __FILE__, __LINE__, "ss", "opening /dev/null failed:", strerror(errno)); return -1; } } ==> fdevent_open_cloexec("/dev/null", O_RDWR, 0); ==> int fd = open(pathname, flags | O_NOCTTY, mode);
In 1.4.45
in log_error_open() ==> openDevNull(STDERR_FILENO); ==> open("/dev/null", O_RDWR);
Thanks for your help!
Zhipeng
RE: How to debug lighttpd could not be started by start-stop-daemon. - Added by jbaans over 6 years ago
Please post your lighttpd config.
This says you have /dev/null defined for log location (discards) which should work.
Also /dev/null is not accessible. That's an OS problem. Perhaps bad permissions?
Does
echo Hello > /dev/null
work without throwing errors?
Does
ls -la /dev/null
show rw permissions for everyone?
RE: How to debug lighttpd could not be started by start-stop-daemon. - Added by zhipengl over 6 years ago
This 2 commands can work.
As we can access /dev/null
controller-0:/www$ ls pages tmp var
When I copy dev/null to
/www/dev/null and chmod 777
no issue found anymore. lighttpd can started normally
However, I still not find the root cause and fix.
why it not take place in 1.4.45.
I attached several configuration files
lighttpd.conf (12.6 KB) lighttpd.conf | |||
lighttpd.init (2.5 KB) lighttpd.init | |||
lighttpd45.spec (27.3 KB) lighttpd45.spec |
RE: How to debug lighttpd could not be started by start-stop-daemon. - Added by zhipengl over 6 years ago
In server.c of 1.4.50
I can see that in
static int server_main (server * const srv, int argc, char **argv) { around line 1115, here open devnull should be OK, /* close stdin and stdout, as they are not needed */ { struct stat st; int devnull; int errfd; do { /* coverity[overwrite_var : FALSE] */ devnull = fdevent_open_devnull(); //call for the first time #ifdef __COVERITY__ __coverity_escape__(devnull); #endif } while (-1 != devnull && devnull <= STDERR_FILENO); then, go to around line 1439 if (!srv->srvconf.preflight_check) { if (-1 == log_error_open(srv)) { //call fdevent_open_devnull for the second time .... }log_error_open will call below. My confusion is why here fdevent_open_devnull function called for the second time, and failed.
else if (!srv->srvconf.dont_daemonize) { /* move STDERR_FILENO to /dev/null */ if (-1 == (errfd = fdevent_open_devnull())) { log_error_write(srv, __FILE__, __LINE__, "ss", "opening /dev/null failed:", strerror(errno)); return -1; } }
RE: How to debug lighttpd could not be started by start-stop-daemon. - Added by zhipengl over 6 years ago
can I remove the code that call fdevent_open_devnull() for the first time?
I see it is not called in 1.4.45
Not sure if the close function is not called after first open function was called, then call open function for the second time.
Or do you have any other proposal?
Thanks!
Zhipeng
RE: How to debug lighttpd could not be started by start-stop-daemon. - Added by zhipengl over 6 years ago
Hi,
I have found the root cause, not sure if it is potential issue for lighttpd.
Actually, this issue exist in lighttpd1.4.45 as well.
1) For 1.4.45
in log_error_open(), it will call below
else if (!srv->srvconf.dont_daemonize) {
/* move stderr to /dev/null */
openDevNull(STDERR_FILENO); //It will not check return value
}
2) For 1.4.47+
in log_erorr_open(),
else if (!srv->srvconf.dont_daemonize) {
/* move STDERR_FILENO to /dev/null */
if (-1 == (errfd = fdevent_open_devnull())) { //here, it will check return value.
log_error_write(srv, FILE, LINE, "ss",
"opening /dev/null failed:", strerror(errno));
return -1;
}
}
Actually, for both versions, it will return -1 for fdevent_open_devnull() from debug log.But old version ,it will has no impact
Not sure why this call here could not return OK. Need lighttpd expert further check if there is any potential bug here .
zhipeng
RE: How to debug lighttpd could not be started by start-stop-daemon. - Added by gstrauss over 6 years ago
Actually, the log is in /www/var/log/lighttpd-error.log
2018-09-09 01:48:37: (server.c.790) opening /dev/null failed: No such file or directory
When I copy dev/null to
/www/dev/null and chmod 777
no issue found anymore. lighttpd can started normally
echo -n "Mounting scratch filesystem to chroot tmp..." umount /www/tmp >& /dev/null rm -r /scratch/lighttpd >& /dev/null mkdir -p /scratch/lighttpd mount --bind /scratch/lighttpd /www/tmp/ chown www /www/tmp/
Pretty obvious that there is a chroot involved somewhere, perhaps to /www since the lighttpd error log showed up in /www/var/log/lighttpd-error.log, and you have not configured /dev/null in that chroot environment, hence the error "No such file or directory".
What I do not understand is why, after almost 3 weeks, you think it might be a bug in lighttpd, instead of the very clear error message: "No such file or directory". "No such file or directory" actually means "No such file or directory".
RE: How to debug lighttpd could not be started by start-stop-daemon. - Added by zhipengl over 6 years ago
Hi gstrauss,
Thanks for your reply!I can see that we set chroot in config file.
- chroot() to directory (default: no chroot() )
server.chroot = "/www"
By default, no chroot. However, if customer set chroot to their own one, do they still need to configure dev/null by themselves?
Is it possible to let lighttpd itself to configure dev/null under server.chroot, if server.chroot configured by customer.
Thanks!
RE: How to debug lighttpd could not be started by start-stop-daemon. - Added by gstrauss over 6 years ago
lighttpd supports doing chroot after lighttpd initial setup with server.chroot
, if lighttpd is started as root.
If your init script chroots prior to starting lighttpd, then you are responsible for providing the contents of the chroot environment.
You do not seem to know much about chroot environments. I suggest you do some reading on the internet.
RE: How to debug lighttpd could not be started by start-stop-daemon. - Added by zhipengl over 6 years ago
As you can see, in my attached init script, it do not do chroot, right?
Zhipeng