Lighttpd

Hi fellow lighttpd user,

You might want to check your logs for related error messages, and check your config with

sudo lighttpd -tt -f /etc/lighttpd/lighttpd.conf

Thanks for you kind help first!
Actually, in our project, I just upgrade lighttpd from 1.4.45 to 47, 49 or 50 version.
Then find this issue.

However, I find the log is very limited in below place.
1) var/log/lighttpd/ this folder is empty
2) var/log/lighttpd-access.log & var/log/lighttpd-error.log

Do you know how to enable more detail log of lighttpd, so that I can debug why it failed to start when I used latest version instead of old one?

Actually, we have some patch based on lighttpd.spec file, I also rebase it to latest version.
(It can work in 1.4.45)

such as:

+Source102: index.html.lighttpd
+Source103: lighttpd.conf
+Source104: lighttpd-inc.conf
+Source105: lighttpd.logrotate
+Source106: lighttpd-csr.conf

 %configure \
     --libdir='%{_libdir}/lighttpd' \
     %{confswitch mysql} \
-    %{confswitch ldap} \
+    --without-ldap \
     %{confswitch attr} \
-    %{confswitch openssl} \
+    --with-openssl \
     %{confswitch kerberos5} \
-    %{confswitch pcre} \
+    --with-pcre \
     %{confswitch fam} \
-    %{?with_webdavprops:--with-webdav-props} \
-    %{?with_webdavlocks:--with-webdav-locks} \
+    --without-webdav-props \
+    --without-webdav-locks \
     %{confswitch gdbm} \
-    %{confswitch memcache} \
-    %{confswitch lua} \
+    --without-memcache \
+    --without-lua \
+    --without-bzip2 \
+    --disable-static \

+CONFDIR=%{buildroot}%{_sysconfdir}/lighttpd
+ROOTDIR=%{buildroot}/www
+
+install -d ${CONFDIR}/ssl
+install -d ${ROOTDIR}/pages/dav
+install -m 0640 %{SOURCE103} ${CONFDIR}/lighttpd.conf
+install -m 0644 %{SOURCE104} ${CONFDIR}/lighttpd-inc.conf
+install -m 0644 %{SOURCE102} ${ROOTDIR}/pages/index.html
+
+install -d %{buildroot}%{_sysconfdir}/logrotate.d
+install -m 644 %{SOURCE105} %{buildroot}%{_sysconfdir}/logrotate.d/lighttpd
+
+chmod 02770 %{buildroot}%{_sysconfdir}/lighttpd

@@ -301,11 +326,8 @@ fi
 %config %{_sysconfdir}/lighttpd/conf.d/mod.template
 %config %{_sysconfdir}/lighttpd/vhosts.d/vhosts.template
 %config(noreplace) %{_sysconfdir}/logrotate.d/lighttpd
-%if %{with systemd}
 %{_unitdir}/lighttpd.service
-%else
 %{_sysconfdir}/rc.d/init.d/lighttpd
-%endif

+%dir /www/pages/
+/www/pages/*
+

Thanks again!
Zhipeng

try

server.modules         += ( "mod_accesslog" )
accesslog.use-syslog    = "enable" 

restart lighttpd and check out syslog (for systemd) with

journalctl -r

I have changed lighttpd.conf as you mentioned.
Unfortunately, it did not work. I could not find any information related to lighttpd by Journalctl -r

In our project, we need start lighttpd in init.d/lighttpd script
In this script, it starts lighttpd as below.

start-stop-daemon --start --pidfile ${PIDFILE} -x "$DAEMON" -- $OPTS

BTW, I can start it through

systemctl start lighttpd

/etc/init.d# sudo ./lighttpd start
Checking scratch filesystem...Mounting scratch filesystem to chroot tmp...Starting Lighttpd Web Server...daemonized server failed to start; check error log for details
failed.

Still could not find where I can check valid log.

It seems your line

start-stop-daemon --start --pidfile ${PIDFILE} x "$DAEMON" - $OPTS

is malformed, double check this.
Note: This may be because of copy pasting here, please use < pre > tags.

Secondly, I don't understand why you are using start-stop-daemon, as your system has systemd installed. Or why both services are installed simultaneously anyway. Systemd replaces the former. Did you recently do a large upgrade?
https://unix.stackexchange.com/questions/236084/how-do-i-create-a-service-for-a-shell-script-so-i-can-start-and-stop-it-like-a-d

Lastly, your logs should be in /var/log/. Try /var/log/messages. Or run dmesg.

no valid log find in /var/log actually.

Now I know that it failed in server.c

static int daemonize(void)
{
   if (bytes <= 0) {
            /* closed fd (without writing) == failure in grandchild */
            fputs("daemonized server failed to start; check error log for details\n", stderr);
            exit(-1);
        }  
}

I see too big gap between 1.4.45 and 1.4.50 for server.c
Not sure what can make it will fail here.

I just doubt, for 1.4.45, may be we not enter this function for daemonize. I'm still checking on this point.

Thanks!

Hi jbaans,

I found some cue now.
Actually, the log is in /www/var/log/lighttpd-error.log
From log, it always shows below, this is the direct cause!!

2018-09-09 01:48:37: (server.c.790) opening /dev/null failed: No such file or directory
2018-09-09 01:48:37: (server.c.1447) Opening errorlog failed. Going down.

However, I still need to check why this error take place in 1.4.50. below is the related code for both.

In 1.4.50

  in log_error_open() ==> 
    else if (!srv->srvconf.dont_daemonize) {
        /* move STDERR_FILENO to /dev/null */
        if (-1 == (errfd = fdevent_open_devnull())) {               
            log_error_write(srv, __FILE__, __LINE__, "ss",
                            "opening /dev/null failed:", strerror(errno));
            return -1;
        }
    }
        ==> fdevent_open_cloexec("/dev/null", O_RDWR, 0);
        ==> int fd = open(pathname, flags | O_NOCTTY, mode);

In 1.4.45

   in log_error_open() ==> openDevNull(STDERR_FILENO); ==> open("/dev/null", O_RDWR);

Thanks for your help！
Zhipeng

Please post your lighttpd config.
This says you have /dev/null defined for log location (discards) which should work.

Also /dev/null is not accessible. That's an OS problem. Perhaps bad permissions?
Does

echo Hello > /dev/null

work without throwing errors?

Does

ls -la /dev/null

show rw permissions for everyone?

This 2 commands can work.
As we can access /dev/null

controller-0:/www$ ls
pages  tmp  var

When I copy dev/null to
/www/dev/null and chmod 777
no issue found anymore. lighttpd can started normally

However, I still not find the root cause and fix.
why it not take place in 1.4.45.

I attached several configuration files

In server.c of 1.4.50
I can see that in

static int server_main (server * const srv, int argc, char **argv) {

     around line 1115, here open devnull should be OK,

          /* close stdin and stdout, as they are not needed */
    {
        struct stat st;
        int devnull;
        int errfd;
        do {
            /* coverity[overwrite_var : FALSE] */
            devnull = fdevent_open_devnull();   //call for the first time
              #ifdef __COVERITY__
            __coverity_escape__(devnull);
              #endif
        } while (-1 != devnull && devnull <= STDERR_FILENO);   

     then, go to around line  1439
             if (!srv->srvconf.preflight_check) {
        if (-1 == log_error_open(srv)) {     //call fdevent_open_devnull for the second time
  ....
}

log_error_open will call below. My confusion is why here fdevent_open_devnull function called for the second time, and failed.

   else if (!srv->srvconf.dont_daemonize) {
        /* move STDERR_FILENO to /dev/null */
        if (-1 == (errfd = fdevent_open_devnull())) {    
            log_error_write(srv, __FILE__, __LINE__, "ss",
                            "opening /dev/null failed:", strerror(errno));
            return -1;
        }
    }

can I remove the code that call fdevent_open_devnull() for the first time?
I see it is not called in 1.4.45

Not sure if the close function is not called after first open function was called, then call open function for the second time.
Or do you have any other proposal？

Thanks!
Zhipeng

Hi，
I have found the root cause, not sure if it is potential issue for lighttpd.
Actually, this issue exist in lighttpd1.4.45 as well.
1) For 1.4.45
in log_error_open(), it will call below
else if (!srv->srvconf.dont_daemonize) {
/* move stderr to /dev/null */
openDevNull(STDERR_FILENO); //It will not check return value
}

2) For 1.4.47+
in log_erorr_open(),
else if (!srv->srvconf.dont_daemonize) {
/* move STDERR_FILENO to /dev/null */
if (-1 == (errfd = fdevent_open_devnull())) { //here, it will check return value.
log_error_write(srv, FILE, LINE, "ss",
"opening /dev/null failed:", strerror(errno));
return -1;
}
}
Actually, for both versions, it will return -1 for fdevent_open_devnull() from debug log.But old version ,it will has no impact
Not sure why this call here could not return OK. Need lighttpd expert further check if there is any potential bug here .

zhipeng

Actually, the log is in /www/var/log/lighttpd-error.log

2018-09-09 01:48:37: (server.c.790) opening /dev/null failed: No such file or directory

When I copy dev/null to
/www/dev/null and chmod 777
no issue found anymore. lighttpd can started normally

    echo -n "Mounting scratch filesystem to chroot tmp..." 
    umount /www/tmp >& /dev/null
    rm -r /scratch/lighttpd >& /dev/null
    mkdir -p /scratch/lighttpd
    mount --bind /scratch/lighttpd /www/tmp/
    chown www /www/tmp/

Pretty obvious that there is a chroot involved somewhere, perhaps to /www since the lighttpd error log showed up in /www/var/log/lighttpd-error.log, and you have not configured /dev/null in that chroot environment, hence the error "No such file or directory".

What I do not understand is why, after almost 3 weeks, you think it might be a bug in lighttpd, instead of the very clear error message: "No such file or directory". "No such file or directory" actually means "No such file or directory".

Hi gstrauss,

Thanks for your reply!
I can see that we set chroot in config file.

1. chroot() to directory (default: no chroot() )
   server.chroot = "/www"

By default, no chroot. However, if customer set chroot to their own one, do they still need to configure dev/null by themselves?
Is it possible to let lighttpd itself to configure dev/null under server.chroot, if server.chroot configured by customer.

Thanks!

lighttpd supports doing chroot after lighttpd initial setup with server.chroot, if lighttpd is started as root.

If your init script chroots prior to starting lighttpd, then you are responsible for providing the contents of the chroot environment.

You do not seem to know much about chroot environments. I suggest you do some reading on the internet.

As you can see, in my attached init script, it do not do chroot, right?

Zhipeng