Bug #2554


weighttp closing connection too early(?)

Added by ezyang over 6 years ago. Updated over 5 years ago.



Steps to reproduce:

1. Install Happstack <> and a simple application, such as pong:
2. Run weighttp with -n 1000 (or something similar)

Expected behavior: Test proceeds with no socket timeouts

Actual behavior: Approximately half of the requests timeout

"But wait!" you might say, "Clearly this is a bug in Happstack, not in weighttp!" And to that I would say, you're probably right. But of course, happstack works fine when used in ordinary workloads (e.g. httperf, or even just loading it from a browser), so I was wondering if I could get some guidance on what it is that weighttp is doing differently that is triggering the bad behavior here. E.g. is weighttp using a really short timeout, or something else...? (Or, maybe it is a weighttp bug...)

Actions #1

Updated by stbuehler over 6 years ago

  • Description updated (diff)
Actions #2

Updated by stbuehler over 6 years ago

(this bug was restored from backup)

Actions #3

Updated by gstrauss over 5 years ago

There are shortcomings in how weighttp handles chunked responses, specifically when the chunked parts are received across multiple reads.

Patch submitted in

A snipped of syscalls from Happstack (running the pong program referenced above) receiving request and sending chunked response:

recvfrom(11, "GET / HTTP/1.1\r\nHost: any\r\nConne"..., 65536, 0, NULL, NULL) = 48
sendto(11, "HTTP/1.1 200 OK\r\nTransfer-Encodi"..., 169, 0, NULL, 0) = 169
sendto(11, "4\r\n", 3, 0, NULL, 0) = 3
sendto(11, "PONG", 4, 0, NULL, 0) = 4
sendto(11, "\r\n", 2, 0, NULL, 0) = 2
sendto(11, "0\r\n\r\n", 5, 0, NULL, 0) = 5

Regarding Happstack:
Now then, Happstack could also use some improvement for better performance, using writev() instead of multiple syscalls to write the chunked parts. Additionally, before calling close(fd) on a client connection, Happstack ought to shutdown(fd, SHUT_WR) and then read() from the connection until EOF (or until a configured timeout), so that the HTTP client has a better chance of receiving the final packets of the response, instead of a TCP RST should the client attempt to write more to the socket (e.g. for HTTP pipelining or should the server send the HTTP response while the client is still sending request body). Another thing: Happstack running on Linux calls epoll_ctl with EPOLL_CTL_MOD (which fails on a new connection) before falling back to epoll_ctl with EPOLL_CTL_ADD. Happstack should know the connection is new because it just called accept(). One more thing, directly related to weighttp, is that the performance of Happstack serving weighttp requests is terrible when weighttp keep-alive is enabled -- Happstack is 100x slower than weighttp without keep-alive! Repro with weighttp -n 1000 with and without -k. This is likely a problem with Happstack since weighttp -k against lighttpd performs much better -- the expected behavior -- than weighttp without -k.

Actions #4

Updated by gstrauss over 5 years ago

Happstack is also very busy with syscalls for thread IPC and short timers. Happstack should wait for OS events to wake it when there is nothing to do, instead of waking itself so frequently with timers.

In any case, use weighttp, once weighttp more robustly handles chunked encoding, makes it easier to spot these Happstack performance deficiencies.

Actions #5

Updated by gstrauss over 5 years ago

The issue in this ticket has been fixed.

stbuehler: please review additional patches pending in

Actions #6

Updated by Anonymous over 5 years ago

  • Status changed from New to Fixed
  • % Done changed from 0 to 100

Also available in: Atom