Lighttpd

because lighttpd didn't exit with a proper error

I do not understand. A non-zero exit value is an error. Full stop. A zero exit value is success. A non-zero exit value indicates an error. An error is an error. What do you mean by "proper error"?

systemd didn't know

Yeah, your problem may be systemd. lighttpd exited non-zero. If systemd is not distinguishing between rc == 0 vs rc != 0, then that would be a bug in systemd, though that would surprise me.

so this change in behavior seems to be new.

Yes. lighttpd 1.4.73 was released 30 Oct 2023. commit 0c5d7096

why don't config errors cause lighttpd to exit with an error code? Is this intentional?

Please validate your assertion. If you have the error in your config and run lighttpd -f /etc/lighttpd/lighttpd.conf -tt, does lighttpd exit 0 or non-zero? lighttpd should exit non-zero for errors (and may exit 0 if only warnings are issued).

Just looking for a sanity check before I go putting in a bug report or something. It's also possible this issue is unique to my distro (Arch), and should instead go to the aur comments section for this package.

As above, please validate that lighttpd exits non-zero when that error is in your config. lighttpd exits 255 when I test server.kbytes-per-second= 700000 in my config.

Then, we can see if this is a problem for systemd or something else.

So upon further investigation thanks to your extremely helpful questions and responses, it appears the error code returned differs depending on whether lighttpd is ran via the command lighttpd vs the command lighttpd-angel, as the latter is how the Arch Linux package's systemd service file has been configured.

To clarify:

• lighttpd -tt -f /etc/lighttpd/lighttpd.conf correctly produces the exit code (255)
• lighttpd -f /etc/lighttpd/lighttpd.conf also correctly produces the exit code (as should be obvious)
• lighttpd-angel -f /etc/lighttpd/lighttpd.conf does not produce the error code, it only warns about it in the terminal (lighttpd-angel.c.107: child (pid=562352) exited normally with exitcode: 255)
• The systemd service file for lighttpd (from the Extra repos, not the AUR, as I mistakenly said before) runs lighttpd with the command /usr/bin/lighttpd-angel -D -f /etc/lighttpd/lighttpd.conf, hence no error returned back to systemd to indicate failure

Thank you for being so thorough. I'll have to review the history of lighttpd-angel to understand why it does that. I won't have time for a few days.

With systemd managing service restart, maybe Arch should consider having systemd call lighttpd without lighttpd-angel. Right now, systemd and lighttpd-angel are two stacked service managers.

Got up early and found a few mins to look into this.

lighttpd-angel is behaving as intended. If the lighttpd process exits normally, with an exit value, then lighttpd-angel does not restart lighttpd, since either lighttpd shutdown was intentional (e.g. due to having been sent a SIGTERM), or there is an error, e.g. a syntax error in the config. Either way, lighttpd-angel should not restart lighttpd. If there is a syntax error, then lighttpd-angel restarting lighttpd would cause a hard loop on the CPU as lighttpd was continually restarted, and continually exited due to syntax error in config.

On the other hand, if lighttpd dies from a signal, e.g. SIGSEGV, then lighttpd-angel will restart lighttpd, though I am unaware of any crashing bugs in current lighttpd that would cause this.

As a process monitoring which is monitoring the lighttpd process, lighttpd-angel exits 0 when there is not an error in lighttpd-angel. Again, if there is a syntax error, then would you want systemd to continually restart lighttpd-angel to restart lighttpd? Well, systemd has backoff logic if systemd finds it is restarting a process too frequently.

Thus, my conclusion that since Arch is using systemd, Arch should have systemd call lighttpd, skipping lighttpd-angel, as there is no benefit that I see from having both systemd and lighttpd-angel process monitors.

Separately, Arch systemd profile for lighttpd ought to be testing the lighttpd config prior to restarting a running lighttpd instance, and this would properly error out for systemd. Does Arch lighttpd.service contain something like this?
ExecStartPre=/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf

==> Arch should consider removing its use of lighttpd-angel and/or Arch lighttpd.service should contain something like
ExecStartPre=/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf

Arch lighttpd.service is ancient, last updated over 8 1/2 years ago.
https://gitlab.archlinux.org/archlinux/packaging/packages/lighttpd/-/blob/main/lighttpd.service?ref_type=heads
Not only is it missing ExecStartPre=/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf, it should use SIGUSR1 for reload.

lighttpd ships a sample lighttpd.service
https://git.lighttpd.net/lighttpd/lighttpd1.4/src/branch/master/doc/systemd/lighttpd.service

Would you please file an issue with AUR to have Arch updated?

I appreciate your insight! Unfortunately, Arch's lighttpd package is not hosted in the AUR, but in one of the main/official repos, so an account on the gitlab link you provided (https://gitlab.archlinux.org/archlinux/packaging/packages/lighttpd) is necessary to submit bugs and I don't currently have one there. I'll see if I can get one set up and submit a bug/issue to the maintainer to have this corrected with a link to this forum thread for further details.

I was curious where that sample lighttpd.service file is on my system but I don't see it anywhere, so I'm guessing during the install/build phase, the Arch package just tosses that out. Not sure though.

Anyway, thank you for all your help!! I'll update here with a link to the bug/issue when I get it opened, or if follow-up questions arise.

Issue https://gitlab.archlinux.org/archlinux/packaging/packages/lighttpd/-/issues/6 opened with the maintainers of the Arch linux version of lighttpd (for visibility).