Lighttpd

Thanks for the details on how to reproduce. I won't have a chance to look into this further for a few days, but will update when I do.

FYI: your suggested patch makes server.stream-response-body = 2 equivalent to server.stream-response-body = 1, so that is not the correct fix for server.stream-response-body = 2. Curious: why are you using server.stream-response-body = 2 ?

Thanks for the reply!

That is interesting that my patch makes the behaviour the same as stream-response-body = 1. I did not realize that.

I started using stream-response-body = 2 as I found the streaming behaviour changed when I updated from 1.4.39. The default of 0 broke an app I have that streams continuously. I actually did not try stream-response-body = 1, primarily because 2 sounded more like what I wanted ("minimal buffering"). Given what you said, it sounds like server-response-body = 1 would work just fine for me.

I suppose I was/am not clear about the intended difference of the two settings (1 vs. 2). Is it supposed to be that setting it to 2 avoids writing temporary files, and that is all?

Thanks again. I'd be pleased to help more if I can!

Thanks to your detailed description of the issue, I was able to quickly narrow in on what I think is the problem.
This is what I see in the code around line 3678:

                        /* optimistic read from backend, which might re-enable FDEVENT_IN */
                        handler_t rc = fcgi_recv_response(srv, hctx); /*(might invalidate hctx)*/

which can theoretically result in the subsequent ioctl() returning 0 (success), while reporting bytes-ready of 0 bytes if none are ready to be read. Normally, if lighttpd receives FDEVENT_IN from the backend, ioctl() reporting 0 bytes ready would likely indicate EOF. In the optimistic read case, we have not been checking for FDEVENT_IN, so we do not know if any bytes are ready to be read. Here is an untested patch against lighttpd 1.4.45 which might handle the optimistic read case for you:

--- a/src/mod_fastcgi.c
+++ b/src/mod_fastcgi.c
@@ -2421,6 +2421,10 @@ static int fcgi_demux_response(server *srv, handler_ctx *hctx) {
                }
        }
        if (0 == r) {
+               if (!(fdevent_event_get_interest(srv->ev, hctx->fd) & FDEVENT_IN)) {
+                       fdevent_event_add(srv->ev, &(hctx->fde_ndx), hctx->fd, FDEVENT_IN);
+                       return 0;
+               }
                log_error_write(srv, __FILE__, __LINE__, "ssdsb",
                                "unexpected end-of-file (perhaps the fastcgi process died):",
                                "pid:", proc->pid,

(If this works for you, then a similar patch also needs to be applied in mod_proxy)

To answer your question about the difference between server.stream-response-body = 1 and server.stream-response-body = 2:

lighttpd is typically faster than the client and typically faster than the backend. Users who stream a large response will most likely want server.stream-response-body = 1 so that lighttpd buffers (to tempfiles) the response from the backend as quickly as the backend produces it, even if lighttpd was unable to send the response as quickly to a (far away) client. Buffering the response from the backend allows the backend to finish sending its response quickly, and then to go on serving other requests. This is quite advantageous when the backend is a fat PHP client compared to the slim lighttpd.

server.stream-response-body = 2 is more appropriate when the backend is producing what might be an extremely large stream and it is desired that the backend be blocked from sending more response data when lighttpd is unable to send response data just as quickly to the client. While this avoids the tempfiles, it also blocks the backend producer from sending more data, which might be desirable for some use cases, though probably not for the most common use cases. This might be appropriate for large generated responses from a backend to lighttpd on a small embedded system which does not have a large amount of disk space for temporary files. server.stream-response-body = 0 just means that lighttpd should buffer the entire response from the backend before starting to send the response to the client. This currently allows mod_deflate (if configured) to compress the response before sending it to the client. In most cases, the response from the backend is small enough, the backend is located on the same machine as the lighttpd server, and the backend produces the response quickly enough that there is not a big perceived difference between server.stream-response-body = 0 and server.stream-response-body = 1. However, the tunable exists for those who do have use cases which suggest a different tuning from the default.

In your case, it would seem that server.stream-response-body = 1 will be best for you, though I would very much appreciate some feedback if you get a chance to test the above patch with server.stream-response-body = 2.

I'll be off the grid for the next few days, so won't be able to respond to further posts until next week.

Thank you for the patch! And thanks for the explanation about how the different server-response-body options work and their purpose! That is very clear.

I added your explanation to the wiki page about the option. I hope that is okay! I figure it might help others understand in the future too which might be the best for them (https://redmine.lighttpd.net/projects/lighttpd/wiki/Server_stream-response-bodyDetails).

I also tested your patch against 1.4.45. Unfortunately the issue still occurs. My client hangs receiving varying amounts from lighttpd.

One other thing I noticed: On the FastCGI server, the amount it sends to lighttpd varies too (as well as what the client sees). In some cases it reports sending the entire response to lighttpd. In others, it blocks part way through sending to lighttpd.

Regarding the case on line 2423 (ioctl() returning 0): I believe if this happens then lighttpd closes the connection to the FastCGI server and reports unexpected EOF. I've seen this error, but it's not happening in my test case currently.

I've been thinking about this after what you said about blocking the backend. Maybe wherever the buffer gets drained (con->write_queue?) we should add back interest in FDEVENT_IN (as in your patch). I'm not sure where that would be though!

I attached a patch showing this change. What do you think?

Since the behavior occurs only when server.stream-response-body = 2, adding back FDEVENT_IN interest was missing in another location. While I found that location, the place where you add it in your patch is a better place. In the patch I will push shortly, I kept the optimistic read, but if the request is not finished (and not an error) after the optimistic read, FDEVENT_IN interest is added (as in your patch). Thanks!

Awesome!

I've tested with the latest git changes, including this patch, and everything is working well.

Thank you for fixing it so quickly!