https://redmine.lighttpd.net/https://redmine.lighttpd.net/favicon.ico?13667327412016-08-02T09:00:16Zlighty labsLighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103732016-08-02T09:00:16Zgstrauss
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Need Feedback</i></li></ul><p>Who is sending the 500, the buildbot or lighttpd?</p>
<p>lighttpd mod_proxy currently sends HTTP/1.0 request to backend, and an explicit 'Connection: close' is included. lighttpd calling shutdown(fd, SHUT_WR) to the backend is an additional indication to the backend that there are no further requests to be sent, and that TCP socket shutdown can proceed as soon as the backend sends the response.</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103742016-08-02T09:06:10Zflynn
<ul><li><strong>File</strong> <a href="/attachments/1709">mod_proxy-problem.pcap</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/1709/mod_proxy-problem.pcap">mod_proxy-problem.pcap</a> added</li></ul><p>The lighttpd sends the 500.</p>
<p>The connection is closed just after sending the request, lighttpd does not wait for the response.</p>
<p>In the tcpdump trace I see no response packets from the buildbot, only the request isself and the close of the connection.</p>
<p>In the attached example the port 8010 ist the remote port of the buildbot instance.</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103752016-08-02T09:12:12Zgstrauss
<ul></ul><p>Is lighttpd not waiting for a response or does the buildbot close the connection to lighttpd without sending a response?</p>
<p>What is the buildbot? (is it open source?)</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103762016-08-02T09:17:08Zgstrauss
<ul></ul><p>See <a class="changeset" title="drain backend socket/pipe bufs upon FDEVENT_HUP (mod_cgi, mod_fastcgi, mod_scgi, mod_proxy)" href="https://redmine.lighttpd.net/projects/lighttpd/repository/14/revisions/923688d2da036f3cefc4fb494dcd770acaab1691">923688d2</a> in lighttpd which adds more robust POLLHUP handling to lighttpd, and also adds the call to shutdown(). My hunch is that the buildbot you are using gets a POLLHUP and fails to read remaining data from the socket buffers.</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103772016-08-02T09:27:53Zflynn
<ul></ul><p>Buildbot is open source: <a class="external" href="http://buildbot.net/">http://buildbot.net/</a><br />It is a continous integration application based on the python twisted framework.</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103792016-08-02T10:12:26Zgstrauss
<ul></ul><p>What operating system are you on? From a quick glance, it would appear that Twisted properly handles POLLHUP when it also receives POLLIN. However, many OSes are inconsistent with one another about how BSD-style socket events are reported. Linux introduced POLLRDHUP (which must be requested) to signify that there is no more data to be read. Many applications consider POLLHUP to be an error indicating client disconnect, when it might not be an error if the client did a half-close on the socket with shutdown(fd, SHUT_WR), and the OS returned POLLHUP as an event on the polled fd, even if the connection is still writable.</p>
<p>Would you please confirm whether or not lighttpd is seeing that the backend (buildbot) closed the connection before lighttpd returns 500 because the backend did not send a response? In an strace, you should see that lighttpd reads (read() or recv()) 0 bytes from the backend, indicating end-of-stream.</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103812016-08-02T11:10:01Zflynn
<ul></ul><p>OS is Linux. A longer strace:</p>
<p>writev(63, [{"GET /default.css HTTP/1.0\r\nUser-"..., 185}], 1) = 185<br />stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2945, ...}) = 0<br />write(27, "2016-08-02 12:38:55: (mod_proxy."..., 71) = 71<br />epoll_ctl(28, EPOLL_CTL_MOD, 63, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=63, u64=63}}) = 0<br />shutdown(63, SHUT_WR) = 0<br />epoll_wait(28, {{EPOLLIN|EPOLLHUP, {u32=63, u64=63}}}, 1025, 1000) = 1<br />ioctl(63, FIONREAD, [0]) = 0<br />epoll_ctl(28, EPOLL_CTL_DEL, 63, {0, {u32=0, u64=0}}) = 0<br />close(63) = 0</p>
<p>There is no read/recv from remote site.<br />epoll reports an IN event without any data available or error??</p>
<p>The problem is, that ioctl FIONREAD reports 0 bytes to read, <br />this leads lighttpd to close the connection.</p>
<p>In the tcpdump you can see:<br />- the packet with payload containing the http request<br />- an ACK packet from remote<br />- lighttpd begins closing the connection (FIN, ACK)</p>
<p>Without the call to shutdown, ioctl FIONREAD reports the correct size and everything is fine.</p>
<p>The call to the shutdown function modifies the result ioctl FIONREAD ...</p>
<p>But why?</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103822016-08-02T11:36:02Zgstrauss
<ul></ul><p>Would you strace the buildbot side of the connection?<br />(lighttpd does not need to read() or recv() 0 bytes if ioctl() tells lighttpd that it will read 0 bytes)</p>
<p>Let me be clear so that I don't keep asking the same question:<br />please demonstrate that buildbot has sent a response before buildbot closes its side of the connection.</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103832016-08-02T12:09:46Zflynn
<ul></ul><p>Buildbot does not send any response, because itself receives zero bytes and closes the connection.</p>
<p>strace Buildbot with timestamps:<br /><pre>
13:41:32.816716 accept(17, {sa_family=AF_INET, sin_port=htons(40978), sin_addr=inet_addr("127.0.0.1")}, [16]) = 12
13:41:32.816852 fcntl(12, F_GETFD) = 0
13:41:32.816906 fcntl(12, F_SETFD, FD_CLOEXEC) = 0
13:41:32.817038 fcntl(12, F_GETFL) = 0x2 (flags O_RDWR)
13:41:32.817083 fcntl(12, F_SETFL, O_RDWR|O_NONBLOCK) = 0
13:41:32.817191 epoll_ctl(6, EPOLL_CTL_ADD, 12, {EPOLLIN, {u32=12, u64=40046962262671372}}) = 0
13:41:32.817287 accept(17, 0x7ffc0da91700, [16]) = -1 EAGAIN (Resource temporarily unavailable)
13:41:32.817411 epoll_wait(6, {{EPOLLIN, {u32=12, u64=40046962262671372}}}, 6, 660) = 1
13:41:32.817484 recvfrom(12, "GET /default.css HTTP/1.0\r\nUser-"..., 65536, 0, NULL, NULL) = 185
13:41:32.817731 getsockname(12, {sa_family=AF_INET, sin_port=htons(8010), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
13:41:32.817876 stat("/var/lib/buildbot/masters/MapOneLibs/public_html", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0
13:41:32.817998 stat("/var/lib/buildbot/masters/MapOneLibs/public_html/default.css", {st_mode=S_IFREG|0640, st_size=10137, ...}) = 0
13:41:32.818126 stat("/var/lib/buildbot/masters/MapOneLibs/public_html/default.css", {st_mode=S_IFREG|0640, st_size=10137, ...}) = 0
13:41:32.818211 open("/var/lib/buildbot/masters/MapOneLibs/public_html/default.css", O_RDONLY) = 13
13:41:32.818256 fstat(13, {st_mode=S_IFREG|0640, st_size=10137, ...}) = 0
13:41:32.818361 fstat(13, {st_mode=S_IFREG|0640, st_size=10137, ...}) = 0
13:41:32.818398 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0478506000
13:41:32.818443 read(13, "body.interface {\n\tmargin-left: 3"..., 65536) = 10137
13:41:32.818490 read(13, "", 53248) = 0
13:41:32.818655 epoll_ctl(6, EPOLL_CTL_MOD, 12, {EPOLLIN|EPOLLOUT, {u32=12, u64=40046962262671372}}) = 0
13:41:32.818774 epoll_wait(6, {{EPOLLIN|EPOLLOUT, {u32=12, u64=40046962262671372}}}, 6, 659) = 1
13:41:32.818843 recvfrom(12, "", 65536, 0, NULL, NULL) = 0
13:41:32.818895 epoll_ctl(6, EPOLL_CTL_MOD, 12, {EPOLLOUT, {u32=12, u64=40046962262671372}}) = 0
13:41:32.818989 close(13) = 0
13:41:32.819025 munmap(0x7f0478506000, 4096) = 0
13:41:32.819084 epoll_ctl(6, EPOLL_CTL_DEL, 12, {EPOLLWRNORM|EPOLLWRBAND|EPOLLRDHUP|EPOLLONESHOT|0x3540c800, {u32=32516, u64=24336577484324612}}) = 0
13:41:32.819160 shutdown(12, SHUT_RDWR) = 0
13:41:32.819246 close(12) = 0
</pre></p>
<p>Comparing with a corresponding tcpdump shows that</p>
<pre>13:41:32.818843 recvfrom(12, "", 65536, 0, NULL, NULL) = 0</pre>
<p>is the reaction to <FIN,ACK> TCP packet from the lighttpd port.</p>
<p>strace lighttpd:<br /><pre>
13:41:32.816353 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 63
13:41:32.816398 fcntl(63, F_SETFD, FD_CLOEXEC) = 0
13:41:32.816433 fcntl(63, F_SETFL, O_RDWR|O_NONBLOCK) = 0
13:41:32.816470 connect(63, {sa_family=AF_INET, sin_port=htons(8010), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
13:41:32.816603 epoll_ctl(28, EPOLL_CTL_ADD, 63, {EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=63, u64=63}}) = 0
13:41:32.816668 accept(10, 0x7ffc69d03020, [112]) = -1 EAGAIN (Resource temporarily unavailable)
13:41:32.816737 epoll_wait(28, {{EPOLLOUT, {u32=63, u64=63}}}, 1025, 1000) = 1
13:41:32.816802 getsockopt(63, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
13:41:32.816868 epoll_ctl(28, EPOLL_CTL_MOD, 63, {EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=63, u64=63}}) = 0
13:41:32.816924 writev(63, [{"GET /default.css HTTP/1.0\r\nUser-"..., 185}], 1) = 185
13:41:32.817035 epoll_ctl(28, EPOLL_CTL_MOD, 63, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=63, u64=63}}) = 0
13:41:32.817092 shutdown(63, SHUT_WR) = 0
13:41:32.817148 epoll_wait(28, {{EPOLLIN|EPOLLHUP, {u32=63, u64=63}}}, 1025, 1000) = 1
13:41:32.819253 ioctl(63, FIONREAD, [0]) = 0
13:41:32.819319 epoll_ctl(28, EPOLL_CTL_DEL, 63, {0, {u32=0, u64=0}}) = 0
13:41:32.819374 close(63) = 0
</pre></p>
<p>I hope this is, what you requested.</p>
<p>I forgot to mention, that I have this "double" proxy configuration, that is needed to rewrite a request, that is proxied,<br />see the last answer to <a class="external" href="https://redmine.lighttpd.net/issues/164">https://redmine.lighttpd.net/issues/164</a> (This is maybe another issue to fix better in lighttpd).</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103842016-08-02T12:37:04Zgstrauss
<ul></ul><p>13:41:32.817484 recvfrom(12, "GET /default.css HTTP/1.0\r\nUser-"..., 65536, 0, NULL, NULL) = 185<br />13:41:32.818843 recvfrom(12, "", 65536, 0, NULL, NULL) = 0</p>
<p>buildbot receives the request, but is not sending a response. Please file an issue with buildbot/twisted and post the link here. I'd like to follow it, but might not have time myself to dig into those many thousands of lines of python.</p>
<p>.</p>
<p>Separately, regarding <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: mod_proxy and mod_rewrite inside the same URL conditional? (Fixed)" href="https://redmine.lighttpd.net/issues/164">#164</a>, if you'd like to submit a patch, I'll review it. However, this is lower priority than tickets such as TLS and mod_auth. Also, I can predict that there will be a follow-up request by someone who wants to parse the HTML or XML or ??? and rewrite all the paths in the response from the backend.</p>
<p>Is buildbot really written in such a way as to require the root of the website? It doesn't have a /$prefix/ on the URL? Do all the links generated by buildbot work, or are they missing the "/$prefix/"?</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103862016-08-03T12:58:05Zflynn
<ul></ul><p>Buildbot really does not support a prefix option or feature, but I posted an alternative approach rewriting proxy requests with lua/mod_magnet in ticket <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: mod_proxy and mod_rewrite inside the same URL conditional? (Fixed)" href="https://redmine.lighttpd.net/issues/164">#164</a>.</p>
<p>Back to the problem with shutdown(..., SHUT_WR): yesterday I have had the same problem with self written scgi-backends and on this backends I could debug and understand the problem.</p>
<p>shutdown(..., SHUT_WR) needs special handling on the remote side: it is reported on the remote side with a read of 0 bytes.<br />Many applications treat a read of zero bytes as socket close and start to close their socket. This is what happens here.</p>
<p>This is wrong, as long as no error is reported.<br />The application must distinguish between read 0 bytes with and without reported error.<br />With this I could fix my scgi-backends, they are running now with an unmodified mod_scgi.<br />I'll take a look in twisted, whether I can suggest a fix there.<br />The perl/moose framework may have the same problem ...</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103872016-08-03T13:28:18Zstbuehler
<ul></ul><p>I'm pretty sure HTTP defines a "client side close" as connection abort (whether explicitly in the RFC, or implicitly because everyone did it), as the client has no other way to indicate it isn't interested in the response anymore; if the client closes the connection completely, the server would get a TCP RST, but only <strong>after</strong> it tries to send additional response data, not while simply polling for some events for example.</p>
<p>In other words, an HTTP client (including <code>mod_proxy</code>) must not call <code>shutdown(SHUT_WR)</code> before it received the response, as long as it is interested in getting the response.</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103882016-08-03T17:12:48Zgstrauss
<ul></ul><p><a class="user active user-mention" href="https://redmine.lighttpd.net/users/2609">@flynn</a>: thanks for your continued troubleshooting. It remains to be seen how many others run into this, and I recognize that there is a potential for others to trip over this. As you pointed out above, your code (and maybe buildbot/twisted) made the assumption that reading end-of-stream meant that the socket was closed, and this was a mistaken assumption since it did not differentiate between a half-close of a socket and an error on the socket. Handling TCP sockets has numerous edge cases, and there are many frameworks that attempt to hide this complexity from application-level programmers. If you're writing an asynchronous program, then there is an additional burden of understanding some of these nuances.</p>
<p>On the other hand, I posit that most (not all) code will have been coded simply. Most code will simply read a request in a blocking fashion and will process the request once the request is complete. Such simple code which handles a single request (and not HTTP/1.1 keep-alive) may not ever read end-of-stream since the code will read a single request, send a single response, and close the socket connection.</p>
<p>.</p>
<p><a class="user active user-mention" href="https://redmine.lighttpd.net/users/7">@stbuehler</a>: can you point me to any references to back up the claims you make in your post above?</p>
<p>Some excerpts from the following appear to indicate the opposite of what you are saying<br /><a class="external" href="https://www.safaribooksonline.com/library/view/http-the-definitive/1565925092/ch04s07.html">https://www.safaribooksonline.com/library/view/http-the-definitive/1565925092/ch04s07.html</a></p>
<blockquote>
<p>TCP close and reset errors</p>
<p>Simple HTTP applications can use only full closes. But when applications start talking to many other types of HTTP clients, servers, and proxies, and when they start using pipelined persistent connections, it becomes important for them to use half closes to prevent peers from getting unexpected write errors.</p>
<p>In general, closing the output channel of your connection is always safe. The peer on the other side of the connection will be notified that you closed the connection by getting an end-of-stream notification once all the data has been read from its buffer.</p>
<p>[...]</p>
<p>Graceful close</p>
<p>The HTTP specification counsels that when clients or servers want to close a connection unexpectedly, they should "issue a graceful close on the transport connection," but it doesn't describe how to do that.</p>
<p>In general, applications implementing graceful closes will first close their output channels and then wait for the peer on the other side of the connection to close its output channels. When both sides are done telling each other they won't be sending any more data (i.e., closing output channels), the connection can be closed fully, with no risk of reset.</p>
<p>Unfortunately, there is no guarantee that the peer implements or checks for half closes. For this reason, applications wanting to close gracefully should half close their output channels and periodically check the status of their input channels (looking for data or for the end of the stream). If the input channel isn't closed by the peer within some timeout period, the application may force connection close to save resources.</p>
</blockquote>
<p>All that said, here's a link to a discussion from 2001, which is much older than the book above:<br /><a class="external" href="https://lists.w3.org/Archives/Public/ietf-http-wg-old/2001JanApr/0037.html">https://lists.w3.org/Archives/Public/ietf-http-wg-old/2001JanApr/0037.html</a><br /><a class="external" href="https://lists.w3.org/Archives/Public/ietf-http-wg-old/2001JanApr/0039.html">https://lists.w3.org/Archives/Public/ietf-http-wg-old/2001JanApr/0039.html</a></p>
<p>That last link above is from one of the authors of the HTTP spec, Roy Fielding.</p>
<p><a class="external" href="https://tools.ietf.org/html/rfc7230#section-6.6">https://tools.ietf.org/html/rfc7230#section-6.6</a> Tear-down<br />recommends that the server tear down connections in multiple steps, starting with a half-close, but the spec does not say anything about whether the client can send half-close. See Roy Fielding's note in the ietf-http-wg message above for his suggestion why the spec should not specify such details, since HTTP protocol can be sent over many different transports, of which TCP is one possible transport.</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103892016-08-06T05:16:37Zgstrauss
<ul><li><strong>Subject</strong> changed from <i>mod_proxy fails on some requests with 1.4.40/41</i> to <i>1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwin</i></li><li><strong>Status</strong> changed from <i>Need Feedback</i> to <i>Patch Pending</i></li></ul><p>It appears that a similar situation was reported in the past: <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: [PATCH] LightyTest.pm: do not shutdown on darwin (Fixed)" href="https://redmine.lighttpd.net/issues/2688">#2688</a> "[PATCH] LightyTest.pm: do not shutdown on darwin"</p>
<p>I tested on Darwin and was able to reproduce what <a class="user active user-mention" href="https://redmine.lighttpd.net/users/2609">@flynn</a> is seeing.</p>
<p>Following this post will be two commits. The first restricts the shutdown() calls in mod_proxy and mod_scgi to non-*BSD and non-Darwin platforms, and then only when the "remote" is localhost, since a remote machine could be one of the affected OSes even if the machine running lighttpd is not. The second commit will update lighttpd to better handle POLLHUP received when the event is due to a half-close (shutdown(fd, SHUT_WR) by the client) as opposed to a full-close or TCP RST, as lighttpd is also treating POLLHUP on the client socket as an unconditional error. The fix applies to most *BSD/Darwin platforms, except on OpenBSD, which does not support getsockopt() TCP_INFO.</p>
<p>The reason this fix is special-casing *BSD and Darwin is that the Single Unix Specification and POSIX.1-2013 clearly specify that POLLHUP event should be returned by poll only when the stream is no longer writable. A half-closed socket that is still writable clearly does not match that condition, yet that is what I am seeing on Darwin (El Capitan), and presumably what others are seeing on *BSD, from which Apple originally inherited the Darwin TCP stack.</p>
<p>Single Unix Specification (SUSv2) from 1997 (yes, that is nearly 20 years ago):<br /><a class="external" href="http://pubs.opengroup.org/onlinepubs/007908799/xsh/poll.html">http://pubs.opengroup.org/onlinepubs/007908799/xsh/poll.html</a></p>
<blockquote>
<p>POLLHUP<br />The device has been disconnected. This event and POLLOUT are mutually exclusive; a stream can never be writable if a hangup has occurred. However, this event and POLLIN, POLLRDNORM, POLLRDBAND or POLLPRI are not mutually exclusive. This flag is only valid in the revents bitmask; it is ignored in the events member.</p>
</blockquote>
<p>Updated version of The Open Group Base Specifications Issue 7, published in 2013:<br /><a class="external" href="http://pubs.opengroup.org/onlinepubs/9699919799/">http://pubs.opengroup.org/onlinepubs/9699919799/</a></p>
<blockquote>
<p>POLLHUP<br />A device has been disconnected, or a pipe or FIFO has been closed by the last process that had it open for writing. Once set, the hangup state of a FIFO shall persist until some process opens the FIFO for writing or until all read-only file descriptors for the FIFO are closed. This event and POLLOUT are mutually-exclusive; a stream can never be writable if a hangup has occurred. However, this event and POLLIN, POLLRDNORM, POLLRDBAND, or POLLPRI are not mutually-exclusive. This flag is only valid in the revents bitmask; it shall be ignored in the events member.</p>
</blockquote>
<p>I would be very interested if someone from the *BSD community would explain the history behind what appears to me to be a direct violation of these core Unix specifications, which have not changed how POLLHUP should be handled over the last (at least) ~20 years.</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103902016-08-06T05:16:48Zgstrauss
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-5 priority-4 priority-default closed" href="/issues/2688">Bug #2688</a>: [PATCH] LightyTest.pm: do not shutdown on darwin</i> added</li></ul> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103932016-08-06T05:26:40Zgstrauss
<ul></ul><p>Having posted all that specific to *BSD/Darwin, <a class="user active user-mention" href="https://redmine.lighttpd.net/users/2609">@flynn</a> reports above that his system is Linux. I haven't been able to reproduce this on Linux (using Linux kernel 4.6.4 on Fedora 24 and a small test-case I wrote). A further patch to lighttpd may be advisable, depending on whether the problem is tracked down to Python's Twisted framework or if others see similar issues in other frameworks. In @flynn's case, I am under the impression that Linux is not sending POLLHUP, but Twisted is reading end-of-stream (0 bytes for recv()/recvfrom()) and is treating that as a fully-closed socket instead of checking to see if the socket is half-closed.</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103942016-08-06T08:20:46Zgstrauss
<ul></ul><p>I pushed the second patch to personal/gstrauss/master branch at<br /> <code>git clone https://git.lighttpd.net/lighttpd/lighttpd1.4.git</code></p>
<p>I'd like some feedback from some folks on *BSD platforms before this patch gets pushed to master. Thanks.</p>
<p>(Note: my dev branch might be rewritten without warning)</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103962016-08-07T05:50:44Zgstrauss
<ul></ul><p><a class="changeset" title="[core] check if client half-closed TCP if POLLHUP (#2743) Check if client half-closed TCP connec..." href="https://redmine.lighttpd.net/projects/lighttpd/repository/14/revisions/4bc06bfc0b72b92c67f21ac0dc7de7d67be73728">4bc06bfc</a> demonstrates how to check for half-closed socket on some popular OSes, after recv() of 0 bytes on AF_INET or AF_INET6 socket.</p>
<p><a class="user active user-mention" href="https://redmine.lighttpd.net/users/2609">@flynn</a>: have you had a chance to dig deeper into buildbot or twisted to track down where the one or the other detects end-of-stream and treats it as a socket error?</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103972016-08-07T08:59:55Zstbuehler
<ul></ul><p>gstrauss wrote:</p>
<blockquote>
<p>Some excerpts from the following appear to indicate the opposite of what you are saying<br /><a class="external" href="https://www.safaribooksonline.com/library/view/http-the-definitive/1565925092/ch04s07.html">https://www.safaribooksonline.com/library/view/http-the-definitive/1565925092/ch04s07.html</a></p>
<blockquote>
<p>TCP close and reset errors</p>
<p>Simple HTTP applications can use only full closes. But when<br />applications start talking to many other types of HTTP clients,<br />servers, and proxies, and when they start using pipelined persistent<br />connections, it becomes important for them to use half closes to<br />prevent peers from getting unexpected write errors.</p>
<p>In general, closing the output channel of your connection is always<br />safe. The peer on the other side of the connection will be notified<br />that you closed the connection by getting an end-of-stream<br />notification once all the data has been read from its buffer.</p>
</blockquote></blockquote>
<p>Don't see how that section is related to anything apart from<br />recommmending a graceful close so the other end doesn't see hard errors<br />(aka RST, which it still needs to be able to handle.)</p>
<blockquote><blockquote>
<p>[...]</p>
<p>Graceful close</p>
<p>The HTTP specification counsels that when clients or servers want to<br />close a connection unexpectedly, they should "issue a graceful close<br />on the transport connection," but it doesn't describe how to do<br />that.</p>
</blockquote></blockquote>
<p>This clearly supports my point; if you want to abort ("close<br />unexpectedly") close the connection (and again the recommendation to do<br />it gracefully). While it is not written that way, imho this clearly<br />indicates an unexpected close (graceful or not) is to be considered a<br />connection abort.</p>
<blockquote><blockquote>
<p>In general, applications implementing graceful closes will first<br />close their output channels and then wait for the peer on the other<br />side of the connection to close its output channels. When both sides<br />are done telling each other they won't be sending any more data<br />(i.e., closing output channels), the connection can be closed fully,<br />with no risk of reset.</p>
</blockquote></blockquote>
<p>This just describes how to avoid TCP RST packets.</p>
<blockquote><blockquote>
<p>Unfortunately, there is no guarantee that the peer implements or<br />checks for half closes. For this reason, applications wanting to<br />close gracefully should half close their output channels and<br />periodically check the status of their input channels (looking for<br />data or for the end of the stream). If the input channel isn't<br />closed by the peer within some timeout period, the application may<br />force connection close to save resources.</p>
</blockquote></blockquote>
<p>Don't wait too long for the other end. Nothing interesting here.</p>
<blockquote>
<p>All that said, here's a link to a discussion from 2001, which is much<br />older than the book above:</p>
<p><a class="external" href="https://lists.w3.org/Archives/Public/ietf-http-wg-old/2001JanApr/0037.html">https://lists.w3.org/Archives/Public/ietf-http-wg-old/2001JanApr/0037.html</a></p>
</blockquote>
<p>This guy clearly states that he assumes most implementations are of type<br />"2" (send FIN after receiving respoe) and "B" (treat FIN as abort),<br />which is exactly what I described.</p>
<p>It would be somewhat safe to impement "A" (ignore FIN on server), but as<br />I alraedy said this makes it impossible to implement long polls - all<br />your requests need to be handled quickly, otherwise you'll end up with a<br />lot of hanging connections. (Long poll requests must not be combined<br />with pipelining ofc, but afaik no client does pipelining by default).</p>
<p>I think lighttpd 1.4.x is probably of type "A" right now, but lighttpd 2 <br />is of type "B" <br />(<a class="external" href="https://git.lighttpd.net/lighttpd/lighttpd2.git/tree/src/main/connection.c#n205">https://git.lighttpd.net/lighttpd/lighttpd2.git/tree/src/main/connection.c#n205</a>).</p>
<blockquote>
<p><a class="external" href="https://lists.w3.org/Archives/Public/ietf-http-wg-old/2001JanApr/0039.html">https://lists.w3.org/Archives/Public/ietf-http-wg-old/2001JanApr/0039.html</a></p>
</blockquote>
<p>At least that guy supports your theory. Doesn't have any arguments<br />though. As I said before, this interpretation takes away the possibility<br />for a client to abort a request / "really close the connection".</p>
<blockquote>
<p>That last link above is from one of the authors of the HTTP spec, Roy<br />Fielding.</p>
</blockquote>
<p>"proof by authority" or "ignore this crazy idiot who designed this<br />broken shit"...</p>
<blockquote>
<p><a class="external" href="https://tools.ietf.org/html/rfc7230#section-6.6">https://tools.ietf.org/html/rfc7230#section-6.6</a> Tear-down recommends<br />that the server tear down connections in multiple steps, starting with<br />a half-close, but the spec does not say anything about whether the<br />client can send half-close. See Roy Fielding's note in the ietf-http-<br />wg message above for his suggestion why the spec should not specify<br />such details, since HTTP protocol can be sent over many different<br />transports, of which TCP is one possible transport.</p>
</blockquote>
<p>It's a lame excuse not to want to care about important details, and<br />makes his opinion even less important.</p>
You'd actually need to clearly specify what you <strong>want</strong> from the<br />transport. E.g. you could say:
<ul>
<li>Transport needs semantics for connection close (<strong>not</strong> half-close)</li>
<li>Transport <strong>optionally</strong> can <strong>additionally</strong> provide semantics for half-<br /> close</li>
</ul>
<p>Then you'd define how you want to handle these in context of HTTP; and<br />for each transport you want to run HTTP over you'd need to define how to<br />map them.</p>
<p>As TCP is the <strong>official</strong> way to run HTTP on the standard should have<br />specified how to map semantics; but as TCP can only provide one of the<br />requirements the mapping would be obvious.</p>
<p>The recommendation to do graceful TCP closes belongs only in the TCP<br />part, because it has nothing to do with HTTP.</p>
<p>Btw: HTTP/2 does it exactly that way:</p>
<p><a href="https://tools.ietf.org/html/rfc7540#section-5.4.1" class="external">RFC 7540: 5.4.1 Connection Error Handling</a> (<a href="http://httpwg.org/specs/rfc7540.html#rfc.section.5.4.1" class="external">alternative link</a>, ietf seems slow today)</p>
<blockquote>
<p>An endpoint that encounters a connection error SHOULD first send a<br />GOAWAY frame (Section 6.8) [...]. After sending the GOAWAY frame for<br />an error condition, the endpoint MUST close the TCP connection.</p>
</blockquote>
<p>Nothing about "half-close" anymore, a TCP connection is only open or<br />closed as seen by HTTP.</p>
<p><a href="https://tools.ietf.org/html/rfc7230#section-6.5" class="external">RFC 7230: 6.5. Failures and Timeouts</a></p>
<blockquote>
<p>[...]<br />A client or server that wishes to time out SHOULD issue a graceful<br />close on the connection. Implementations SHOULD constantly monitor<br />open connections for a received closure signal and respond to it as<br />appropriate, since prompt closure of both sides of a connection<br />enables allocated system resources to be reclaimed.<br />[...]</p>
</blockquote>
<p>"appropriate" could mean a lot of things of course; but a peer that<br />"timed out" a connection is no longer expecting a useful response, and<br />the part about the system resources clearly indicates you should simply<br />cancel the request and close the connection.</p>
<p>If you received the full request body it is imho a good idea (but not a<br />hard requirement) to forward that request to the backend, and then<br />indicate to the backend in a similar way (for HTTP: graceful close) that<br />you are not interested in a response.</p>
<p><a href="https://tools.ietf.org/html/rfc7230#section-6.6" class="external">RFC 7230: 6.6. Tear-down</a></p>
<blockquote>
<p>[...]<br />A client that sends a "close" connection option MUST NOT send further<br />requests on that connection (after the one containing "close") and<br />MUST close the connection after reading the final response message<br />corresponding to this request.<br />[...]</p>
</blockquote>
<p>Stated very clearly: close <strong>after</strong> reading the response.</p>
Summary:
<ul>
<li>`mod_proxy` has always been of type "2", you broke it.</li>
<li>TCP half-close is the only way a client has to reliably indicate<br /> connection abort, and all major clients use it exactly that way. RFC<br /> 7230 clearly supports this interpretation, whatever the original HTTP<br /> author thought in 2001.</li>
</ul>
<p>Please revert your change now.</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103982016-08-07T11:50:54Zflynn
<ul></ul><p>I took a look at haproxy, how socket closing is handled.</p>
<p>haproxy uses shutdown(..., SHUT_WR) only on pure TCP-connections, never on http-connections.</p>
<p>I verified this in the code and live with strace (i have a bigger installation of haproxy running ...)</p>
<p>There must be a reason, why haproxy does not use shutdown, and either lighttpd takes this as a hint<br />or the author of haproxy should be asked. I think he is a person with very deep knowledge and experience <br />in http proxying.</p>
I also want to suggest a compromise:
<ul>
<li>keep shutdown(..., SHUT_WR) as default</li>
<li>add an option in mod_proxy, .e.g. proxy.broken_backend = 0/1 to omit the call to shutdown</li>
</ul>
<p>P.S.: I did not have time to look deeper at python/twisted, but I will do it ...</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=103992016-08-07T17:18:26Zgstrauss
<ul></ul><p><a class="user active user-mention" href="https://redmine.lighttpd.net/users/7">@stbuehler</a> wrote:</p>
<blockquote>
<ul>
<li>TCP half-close is the only way a client has to reliably indicate<br />connection abort</li>
</ul>
</blockquote>
<p>Please keep in mind that the HTTP spec does not say this.<br />Also, you previously criticized the HTTP spec, suggesting that<br />the HTTP spec should be more explicit about TCP behavior.<br />(I am interpreting your comments in the context of HTTP.)</p>
<p>The HTTP spec recommends a graceful close when a mechanism is available.<br />When HTTP uses TCP transport, a TCP socket half-close is available to<br />indicate a graceful close.</p>
<p>A graceful close is not necessarily the same things as connection abort.<br />I think you and I have different ideas about what encompasses a<br />"connection abort". I consider "normal connection close" different from<br />"connection abort". Also, I do not consider an "unexpected close" <br />necessarily always equivalent to "connection abort".</p>
<blockquote>
<ul>
<li>TCP half-close is the only way a client has to reliably indicate<br />connection abort</li>
</ul>
</blockquote>
<p>Connection abort is very reliably ("portably") implemented in TCP by<br />sending TCP RST, although TCP RST is not graceful.<br /><pre>
struct linger li = { 1, 0 }; /* { .l_onoff = 1, .l_linger = 0 } */
setsockopt(fd, SOL_SOCKET, SO_LINGER, &li, sizeof(li));
close(fd);
</pre><br />Similarly, shutdown(fd, SHUT_RD) is not graceful, but indicates abort<br />the next time (or maybe the time after that) the remote tries to write.<br />On pipes, closing the read side (so write by the remote gets EPIPE)<br />indicates abort, but is not graceful.</p>
<p>.</p>
<p><a class="user active user-mention" href="https://redmine.lighttpd.net/users/7">@stbuehler</a>: I disagree with your use of the word "clearly" in your<br />arguments, conclusions, and tl;dr, but it is clearly not worth my effort<br />to try to convince you otherwise.</p>
<p>.</p>
<p>I am planning to remove the shutdown() calls in mod_proxy, mod_scgi<br />for practical reasons. Due to the POLLHUP behavior triggered on<br />*BSD/Darwin, the shutdown() has been limited to local connections.<br />If interested in squeezing the last bits of performance out of a<br />machine, an admin should configure local connections to be AF_UNIX<br />instead of AF_INET or AF_INET6 to localhost. The reason I originally<br />added the shutdown() in mod_proxy and mod_scgi was to aggressively<br />reduce the number of potential sockets in TIME_WAIT held by lighttpd.<br />(See <a class="changeset" title="drain backend socket/pipe bufs upon FDEVENT_HUP (mod_cgi, mod_fastcgi, mod_scgi, mod_proxy)" href="https://redmine.lighttpd.net/projects/lighttpd/repository/14/revisions/923688d2da036f3cefc4fb494dcd770acaab1691">923688d2</a> "drain backend socket/pipe bufs upon FDEVENT_HUP",<br /> done for reliability given the aforementioned *BSD/Darwin behavior.)<br />When using AF_UNIX, the TIME_WAIT issue does not exist, ergo, the<br />recommendation is to use AF_UNIX for local sockets, when available.<br />Using AF_UNIX sockets is a better solution to eliminate TIME_WAIT<br />than is TCP shutdown() half-close which, as we have seen, might not<br />be handled well by frameworks which are more complex than basic read<br />request, send response, and close.</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=104002016-08-07T17:20:04Zgstrauss
<ul><li><strong>Status</strong> changed from <i>Patch Pending</i> to <i>Fixed</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>Applied in changeset <a class="changeset" title="[core] proxy,scgi omit shutdown() to backend (fixes #2743) Due to the POLLHUP behavior triggered..." href="https://redmine.lighttpd.net/projects/lighttpd/repository/14/revisions/27f85dbdf43e462c380bb5a738c3bdf90275eeea">27f85dbdf43e462c380bb5a738c3bdf90275eeea</a>.</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=104492016-09-02T12:08:15Zstbuehler
<ul><li><strong>Has duplicate</strong> <i><a class="issue tracker-1 status-10 priority-4 priority-default closed" href="/issues/2751">Bug #2751</a>: mod_proxy closes connection</i> added</li></ul> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=104742016-09-15T01:42:08Zbrad@comstyle.com
<ul></ul><p>The commit 4bc06bfc0b72b92c67f21ac0dc7de7d67be73728 breaks building on OpenBSD.</p>
<p>fdevent.c: In function 'fdevent_is_tcp_half_closed':<br />fdevent.c:305: error: storage size of 'tcpi' isn't known<br />fdevent.c:307: error: 'TCP_INFO' undeclared (first use in this function)<br />fdevent.c:307: error: (Each undeclared identifier is reported only once<br />fdevent.c:307: error: for each function it appears in.)<br />fdevent.c:305: warning: unused variable 'tcpi'</p>
<p>OpenBSD has TCPS_CLOSE_WAIT but not TCP_INFO.</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=104752016-09-15T02:27:25Zgstrauss
<ul></ul><p>Brad: please see subsequent <a class="changeset" title="[core] better DragonFlyBSD support (fixes #2746) (thx xenu) x-ref: "[PATCH] better DragonFlyB..." href="https://redmine.lighttpd.net/projects/lighttpd/repository/14/revisions/6ec66c4dce677b7f52f393bea3d609115476048f">6ec66c4d</a> from 20 Aug which fixed that issue on OpenBSD and DragonflyBSD</p> Lighttpd - Bug #2743: 1.4.40/41 mod_proxy, mod_scgi may trigger POLLHUP on *BSD,Darwinhttps://redmine.lighttpd.net/issues/2743?journal_id=104762016-09-15T18:59:36Zbrad@comstyle.com
<ul></ul><p>Oh, thanks. I had been looking through all of the commits to HEAD but skipped that one as I thought it was only DragonFly related changes. The diff with the relevant bit is similar to what I had come up with locally to make the code build.</p>