https://redmine.lighttpd.net/https://redmine.lighttpd.net/favicon.ico?13667327412016-02-09T13:13:22Zlighty labsLighttpd - Bug #2711: Cronolog Broken pipehttps://redmine.lighttpd.net/issues/2711?journal_id=86902016-02-09T13:13:22Zgstrauss
<ul></ul><p>I am curious if the problem persists or goes away if you use /bin/nohup in the piped log command:<br /><pre>
accesslog.filename = "|/bin/nohup /opt/cronolog/sbin/cronolog /srv/htlogs/lighttpd/lighttpd14_access.%d.%H00"
</pre><br />Send the running server a couple SIGHUP signals and then check the logs. Would you test and report back?</p> Lighttpd - Bug #2711: Cronolog Broken pipehttps://redmine.lighttpd.net/issues/2711?journal_id=86912016-02-09T18:58:24Zstbuehler
<ul><li><strong>Target version</strong> set to <i>1.4.x</i></li></ul> Lighttpd - Bug #2711: Cronolog Broken pipehttps://redmine.lighttpd.net/issues/2711?journal_id=87152016-02-10T17:05:57Zatmnxw
<ul></ul><p>The nohup worked</p>
<p>Thanks a lot!</p> Lighttpd - Bug #2711: Cronolog Broken pipehttps://redmine.lighttpd.net/issues/2711?journal_id=87162016-02-10T17:59:22Zgstrauss
<ul></ul><p>Excellent! Glad that works for you.</p>
<p>So that I might suggest a long-term solution, which modules are you using that require you to send SIGHUP to lighttpd?</p>
<p>Some explanation:<br />When lighttpd receives a SIGHUP signal, it propagates the signal to all children in its process group (including piped loggers). I think this done so that lighttpd workers receive SIGHUP. Receipt of SIGHUP triggers module hup handlers, <strong>but</strong> lighttpd does not restart piped loggers, so those they might die and not be restarted.</p>
<p><a class="user active user-mention" href="https://redmine.lighttpd.net/users/7">@stbuehler</a>:</p>
<p>The propagation of SIGHUP to the entire process group is not very graceful, as it might kill piped loggers, any CGI in progress, or any of a slew of other subprocesses. Is the propagation of SIGHUP done only so that lighttpd workers receive SIGHUP, or are there other reasons? If no other reason, then we might reduce the impact by not propagating SIGHUP unless multiple workers are configured. A more proper solution might be to keep track of lighttpd worker pids and to propagate SIGHUP only to those pids.</p>
<p>Separately, is there a reason that piped loggers are not restarted upon receipt of SIGHUP? In a hup handler, mod_accesslog could start a new piped logger, and if successful, close the write end of the pipe to the old piped logger.</p> Lighttpd - Bug #2711: Cronolog Broken pipehttps://redmine.lighttpd.net/issues/2711?journal_id=87232016-02-10T22:11:13Zstbuehler
<ul></ul><p>gstrauss wrote:</p>
<blockquote>
<p>Some explanation:<br />When lighttpd receives a SIGHUP signal, it propagates the signal to all children in its process group (including piped loggers). I think this done so that lighttpd workers receive SIGHUP. Receipt of SIGHUP triggers module hup handlers, <strong>but</strong> lighttpd does not restart piped loggers, so those they might die and not be restarted.</p>
</blockquote>
<p>I'm just not sure what changed; the <code>server.max-workers</code> options is officially not supported (which would be the "forward <code>SIGHUP</code> to ourself" case), and otherwise it should not get forwarded.</p>
<p>Maybe CentOS switched to systemd in that upgrade? That might explain a different process environment. (I think the standard "daemonize" has the session leader exit, while in systemd we use the non-daemonize start, which probably makes lighttpd the session leader. If then the reload sends <code>SIGHUP</code> to the "negative" session leader all processes in the group will get the signal).</p>
<blockquote>
<p><a class="user active user-mention" href="https://redmine.lighttpd.net/users/7">@stbuehler</a>:</p>
<p>The propagation of SIGHUP to the entire process group is not very graceful, as it might kill piped loggers, any CGI in progress, or any of a slew of other subprocesses. Is the propagation of SIGHUP done only so that lighttpd workers receive SIGHUP, or are there other reasons? If no other reason, then we might reduce the impact by not propagating SIGHUP unless multiple workers are configured. A more proper solution might be to keep track of lighttpd worker pids and to propagate SIGHUP only to those pids.</p>
</blockquote>
<p>A proper max-worker solution would do a lot of things differently. I don't think it's worth our time.</p>
<blockquote>
<p>Separately, is there a reason that piped loggers are not restarted upon receipt of SIGHUP? In a hup handler, mod_accesslog could start a new piped logger, and if successful, close the write end of the pipe to the old piped logger.</p>
</blockquote>
<p>It probably would be nice to have a proper <code>SIGCHLD</code> handling (it would need to be integrated into the <code>fdevent</code> loop handling probably, as libev for example already has a <code>SIGCHLD</code> handler. Then the pipe loggers would need to get restarted when they exit (and we would also need a way to limit restart to detect permanent failures and so on...).</p>
<p>I see no reason not to restart pipe loggers on <code>SIGHUP</code> though, although I think we probably should first let the old logger exit before starting a new one to avoid race conditions when writing to the same file.</p> Lighttpd - Bug #2711: Cronolog Broken pipehttps://redmine.lighttpd.net/issues/2711?journal_id=87242016-02-11T09:44:39Zgstrauss
<ul></ul><p>Since mod_accesslog is currently the only standard module hooking handle_sighup in lighttpd 1.4, it is likely that those using piped-loggers do not bother sending SIGHUP to lighttpd. (Unlike other servers, lighttpd does not reload its configuration when it receives SIGHUP.)</p>
<p>I created <a class="external" href="https://github.com/lighttpd/lighttpd1.4/pull/17">https://github.com/lighttpd/lighttpd1.4/pull/17</a> with a simple change to avoid forwarding SIGHUP if server.max-workers is not configured. I also noted: For those configuring server.max-workers, it is recommended that piped loggers be used to avoid log corruption, and then admins can avoid sending lighttpd SIGHUP as there is currently no benefit to doing so with the standard modules (beyond that of log rotation of non-piped access and error logs).</p>
<p>(I also agree that introducing pid-management for workers and piped loggers is a larger project, with a larger scope than this problem report.)</p>
<blockquote>
<p>the server.max-workers options is officially not supported</p>
</blockquote>
<p>cross-references:<br /><a class="external" href="https://redmine.lighttpd.net/projects/lighttpd/wiki/Server_max-workerDetails">https://redmine.lighttpd.net/projects/lighttpd/wiki/Server_max-workerDetails</a><br /><a class="external" href="https://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_MultiProcessor">https://redmine.lighttpd.net/projects/lighttpd/wiki/Docs_MultiProcessor</a></p> Lighttpd - Bug #2711: Cronolog Broken pipehttps://redmine.lighttpd.net/issues/2711?journal_id=87702016-02-13T10:46:53Zstbuehler
<ul><li><strong>Target version</strong> changed from <i>1.4.x</i> to <i>1.4.40</i></li></ul> Lighttpd - Bug #2711: Cronolog Broken pipehttps://redmine.lighttpd.net/issues/2711?journal_id=87802016-02-14T10:45:05Zstbuehler
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Fixed</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>Applied in changeset r3076.</p>