https://redmine.lighttpd.net/https://redmine.lighttpd.net/favicon.ico?13667327412017-03-09T23:48:34Zlighty labsLighttpd - Bug #2796: mod_fastcgi can fail to read entire response from serverhttps://redmine.lighttpd.net/issues/2796?journal_id=109312017-03-09T23:48:34Zgstrauss
<ul></ul><p>Thanks for the details on how to reproduce. I won't have a chance to look into this further for a few days, but will update when I do.</p>
<p>FYI: your suggested patch makes <code>server.stream-response-body = 2</code> equivalent to <code>server.stream-response-body = 1</code>, so that is not the correct fix for <code>server.stream-response-body = 2</code>. Curious: why are you using <code>server.stream-response-body = 2</code> ?</p> Lighttpd - Bug #2796: mod_fastcgi can fail to read entire response from serverhttps://redmine.lighttpd.net/issues/2796?journal_id=109322017-03-10T00:29:46Zhorgh
<ul></ul><p>Thanks for the reply!</p>
<p>That is interesting that my patch makes the behaviour the same as stream-response-body = 1. I did not realize that.</p>
<p>I started using stream-response-body = 2 as I found the streaming behaviour changed when I updated from 1.4.39. The default of 0 broke an app I have that streams continuously. I actually did not try stream-response-body = 1, primarily because 2 sounded more like what I wanted ("minimal buffering"). Given what you said, it sounds like server-response-body = 1 would work just fine for me.</p>
<p>I suppose I was/am not clear about the intended difference of the two settings (1 vs. 2). Is it supposed to be that setting it to 2 avoids writing temporary files, and that is all?</p>
<p>Thanks again. I'd be pleased to help more if I can!</p> Lighttpd - Bug #2796: mod_fastcgi can fail to read entire response from serverhttps://redmine.lighttpd.net/issues/2796?journal_id=109332017-03-10T07:17:01Zgstrauss
<ul></ul><p>Thanks to your detailed description of the issue, I was able to quickly narrow in on what I think is the problem.<br />This is what I see in the code around line 3678:<br /><pre>
/* optimistic read from backend, which might re-enable FDEVENT_IN */
handler_t rc = fcgi_recv_response(srv, hctx); /*(might invalidate hctx)*/
</pre><br />which can theoretically result in the subsequent ioctl() returning 0 (success), while reporting bytes-ready of 0 bytes if none are ready to be read. Normally, if lighttpd receives FDEVENT_IN from the backend, ioctl() reporting 0 bytes ready would likely indicate EOF. In the optimistic read case, we have not been checking for FDEVENT_IN, so we do not know if any bytes are ready to be read. Here is an <strong>untested</strong> patch against lighttpd 1.4.45 which might handle the optimistic read case for you:<br /><pre>
--- a/src/mod_fastcgi.c
+++ b/src/mod_fastcgi.c
@@ -2421,6 +2421,10 @@ static int fcgi_demux_response(server *srv, handler_ctx *hctx) {
}
}
if (0 == r) {
+ if (!(fdevent_event_get_interest(srv->ev, hctx->fd) & FDEVENT_IN)) {
+ fdevent_event_add(srv->ev, &(hctx->fde_ndx), hctx->fd, FDEVENT_IN);
+ return 0;
+ }
log_error_write(srv, __FILE__, __LINE__, "ssdsb",
"unexpected end-of-file (perhaps the fastcgi process died):",
"pid:", proc->pid,
</pre></p>
<p>(If this works for you, then a similar patch also needs to be applied in mod_proxy)</p> Lighttpd - Bug #2796: mod_fastcgi can fail to read entire response from serverhttps://redmine.lighttpd.net/issues/2796?journal_id=109342017-03-10T07:29:52Zgstrauss
<ul></ul><p>To answer your question about the difference between <code>server.stream-response-body = 1</code> and <code>server.stream-response-body = 2</code>:</p>
<p>lighttpd is typically faster than the client and typically faster than the backend. Users who stream a large response will most likely want <code>server.stream-response-body = 1</code> so that lighttpd buffers (to tempfiles) the response from the backend as quickly as the backend produces it, even if lighttpd was unable to send the response as quickly to a (far away) client. Buffering the response from the backend allows the backend to finish sending its response quickly, and then to go on serving other requests. This is quite advantageous when the backend is a fat PHP client compared to the slim lighttpd.</p>
<code>server.stream-response-body = 2</code> is more appropriate when the backend is producing what might be an extremely large stream and it is desired that the backend be <strong>blocked</strong> from sending more response data when lighttpd is unable to send response data just as quickly to the client. While this avoids the tempfiles, it also blocks the backend producer from sending more data, which might be desirable for some use cases, though probably not for the most common use cases. This might be appropriate for large generated responses from a backend to lighttpd on a small embedded system which does not have a large amount of disk space for temporary files.
<code>server.stream-response-body = 0</code> just means that lighttpd should buffer the entire response from the backend before starting to send the response to the client. This currently allows mod_deflate (if configured) to compress the response before sending it to the client. In most cases, the response from the backend is small enough, the backend is located on the same machine as the lighttpd server, and the backend produces the response quickly enough that there is not a big perceived difference between <code>server.stream-response-body = 0</code> and <code>server.stream-response-body = 1</code>. However, the tunable exists for those who do have use cases which suggest a different tuning from the default.
<p>In your case, it would seem that <code>server.stream-response-body = 1</code> will be best for you, though I would very much appreciate some feedback if you get a chance to test the above patch with <code>server.stream-response-body = 2</code>.</p>
<p>I'll be off the grid for the next few days, so won't be able to respond to further posts until next week.</p> Lighttpd - Bug #2796: mod_fastcgi can fail to read entire response from serverhttps://redmine.lighttpd.net/issues/2796?journal_id=109352017-03-10T07:51:39Zgstrauss
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Patch Pending</i></li><li><strong>Target version</strong> changed from <i>1.4.x</i> to <i>1.4.46</i></li></ul> Lighttpd - Bug #2796: mod_fastcgi can fail to read entire response from serverhttps://redmine.lighttpd.net/issues/2796?journal_id=109362017-03-11T20:46:51Zhorgh
<ul></ul><p>Thank you for the patch! And thanks for the explanation about how the different <code>server-response-body</code> options work and their purpose! That is very clear.</p>
<p>I added your explanation to the wiki page about the option. I hope that is okay! I figure it might help others understand in the future too which might be the best for them (<a class="external" href="https://redmine.lighttpd.net/projects/lighttpd/wiki/Server_stream-response-bodyDetails">https://redmine.lighttpd.net/projects/lighttpd/wiki/Server_stream-response-bodyDetails</a>).</p>
<p>I also tested your patch against 1.4.45. Unfortunately the issue still occurs. My client hangs receiving varying amounts from lighttpd.</p>
<p>One other thing I noticed: On the FastCGI server, the amount it sends to lighttpd varies too (as well as what the client sees). In some cases it reports sending the entire response to lighttpd. In others, it blocks part way through sending to lighttpd.</p>
<p>Regarding the case on line 2423 (<code>ioctl()</code> returning 0): I believe if this happens then lighttpd closes the connection to the FastCGI server and reports unexpected EOF. I've seen this error, but it's not happening in my test case currently.</p>
<p>I've been thinking about this after what you said about blocking the backend. Maybe wherever the buffer gets drained (<code>con->write_queue</code>?) we should add back interest in <code>FDEVENT_IN</code> (as in your patch). I'm not sure where that would be though!</p> Lighttpd - Bug #2796: mod_fastcgi can fail to read entire response from serverhttps://redmine.lighttpd.net/issues/2796?journal_id=109372017-03-12T01:56:06Zhorgh
<ul><li><strong>File</strong> <a href="/attachments/1765">patch-v2.patch</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/1765/patch-v2.patch">patch-v2.patch</a> added</li></ul><p>I've looked at this some more.</p>
<p>I discovered that the module's subrequest function gets called after writing from the write queue (<code>CON_STATE_WRITE</code> in <code>connection_state_machine()</code>). This is the function that has the optimistic read.</p>
<p>I found that if I start listening for <code>FDEVENT_IN</code> again here instead of trying the optimistic read, then the problem goes away. The entire response from the FastCGI server is read by lighttpd.</p>
<p>As well, from what you said, it seems like the optimistic read could lead to that spurious EOF error which I originally encountered.</p>
<p>I attached a patch showing this change. What do you think?</p>
<p>Thanks again!</p> Lighttpd - Bug #2796: mod_fastcgi can fail to read entire response from serverhttps://redmine.lighttpd.net/issues/2796?journal_id=109392017-03-13T17:57:03Zgstrauss
<ul></ul><blockquote>
<p>I attached a patch showing this change. What do you think?</p>
</blockquote>
<p>Since the behavior occurs only when <code>server.stream-response-body = 2</code>, adding back FDEVENT_IN interest was missing in another location. While I found that location, the place where you add it in your patch is a better place. In the patch I will push shortly, I kept the optimistic read, but if the request is not finished (and not an error) after the optimistic read, FDEVENT_IN interest is added (as in your patch). Thanks!</p> Lighttpd - Bug #2796: mod_fastcgi can fail to read entire response from serverhttps://redmine.lighttpd.net/issues/2796?journal_id=109402017-03-13T18:10:05Zgstrauss
<ul><li><strong>Status</strong> changed from <i>Patch Pending</i> to <i>Fixed</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>Applied in changeset <a class="changeset" title="[mod_cgi,fastcgi,scgi,proxy] fix streaming response (fixes #2796) fix streaming response when se..." href="https://redmine.lighttpd.net/projects/lighttpd/repository/14/revisions/e4bb56222fa7c1e315ea7d05cb4ea2f0de781534">e4bb56222fa7c1e315ea7d05cb4ea2f0de781534</a>.</p> Lighttpd - Bug #2796: mod_fastcgi can fail to read entire response from serverhttps://redmine.lighttpd.net/issues/2796?journal_id=109442017-03-13T22:42:05Zhorgh
<ul></ul><p>Awesome!</p>
<p>I've tested with the latest git changes, including this patch, and everything is working well.</p>
<p>Thank you for fixing it so quickly!</p> Lighttpd - Bug #2796: mod_fastcgi can fail to read entire response from serverhttps://redmine.lighttpd.net/issues/2796?journal_id=109732017-03-25T15:59:14Zgstrauss
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-5 priority-3 priority-lowest closed" href="/issues/2802">Feature #2802</a>: Could the error-log be improved a tiny bit in regards to "Permission denied" errors propagated from the filesystem (on Linux at the least)?</i> added</li></ul>