Lighttpd

I've been trying to track down a case of Ruby processes getting stuck for a long time now. Ruby processes incrementally get stuck, until all backends are frozen and I have to kill them manually to kickstart things again. This happens about one process a day or more, depending on site load.

I've recently narrowed it down:

- It happens when uploading files.  The files are almost always large enough to require multiple Lighttpd temp files.
 - Lighttpd is always in the middle of sending the uploaded files to the backend; Rails has a partially-written /tmp/CGI* file and Lighttpd has some temporary files left to send.
 - Ruby is in read(), in FCGI code.
 - The Lighttpd connection is in FCGI_STATE_WRITE, as expected.
 - fcgi's hctx->fde_ndx != 1, so the fastcgi context thinks that there's an fdevent installed, as there should be.  (FreeBSD, using kevent.)
 - srv>ev->kq_bevents for hctx->fd is 0, so fdevent_freebsd_kqueue thinks it's set to wake on writable, as it should be.

It's not a content size mismatch--Rails isn't trying to read more data than we're sending. Lighttpd has more data to send: mod_fastcgi is still in WRITE and hctx->wb->bytes_out < hctx->wb->bytes_in.

When I break in during this state and tracked down the backend's fd, I could do this:

(gdb) p write(hctx->fd, hctx, 1)
1

(This is just write(fd, 'x', 1), but that doesn't work in gdb in FreeBSD, so I had to use a real pointer in the app to give it something to write.)

It's writing the data, so the FD is writable. fdevent/kevent should be waking up the connection. I stopped on a fdevent_poll call from server.c and examined the results; none of them matched hctx->fde_ndx.

It seems like the kevent is getting desynced from the fdevent, but I can't find any way that could happen. I can't find an API to examine the actual contents of a kqueue, either (this is basic stuff; I hate FreeBSD). Nothing unusual is logged around when this happens. I can't imagine that kqueue is losing events--this seems like a very core API.

I'll probably switch to poll or select and see what happens, since at least their complete state can be examined from a debugger, but I'm leery of just hiding the problem.

Suggestions welcome; I'm tired of banging my head on this.