Lighttpd

I will concede that max-connections and listen() backlog values are not 'synonymous', but they are absolutely related.

server.max-connections is exactly that, the maximum number of connections that will be actively 'handled' by the service. It makes no inference whatsoever as to whether they are short-lived, long-running, or the rate at which they come in.

The listen() backlog value, on the other hand, if set incorrectly can directly impact service whether the hardware or application can handle it or not. If a server is capable of processing 4096 concurrent requests, then a static listen backlog of 1024 is tantamount to an artificially imposed bottleneck in use cases such as that mentioned in the description.

As far as whether 1024 should be good enough or not is entirely dependent on the workload in question. Remember, 640k 'should have been' enough memory for anybody too :]

I agree on the Low priority, and would have re-prioritised the ticket given the permissions. I incorrectly assumed that a hard-coded value in an application with an otherwise great track record in high-performance web serving might have been a forgotten TODO item and just never got picked up.

Additionally, apache and nginx both provide user-configurable options for setting the listen() backlog value:

http://httpd.apache.org/docs/2.2/mod/mpm_common.html#listenbacklog

nginx goes one further and defaults it to -1 (SOMAXCONN)

http://wiki.nginx.org/NginxHttpCoreModule#listen

I'd patch it, but it's not that 'I' need it, I think lighttpd and its community would be better for it.

This report is merely to demonstrate the fact that a hard-coded value that makes assumptions about its environment and workload could be improved upon. Whether this results in a better default value (max-connection or -1 (SOMAXCONN)), a configuration option, or a value that scales with something else is entirely up to you and your team.

If it were me, I'd like to see a server.listen-backlog directive.

Edit: If nothing else and at the absolute least, it's a case of the Magic Number Anti-Pattern.

Awesome stuff :]

From what I read, if listen()'s backlog is set to > SOMAXCONN, the kernel will silently limit it to SOMAXCONN. (FreeBSD currently defaults to 128 as well)

For the sake of argument (and possibly avoiding 'yet another configuration option'), can you think of any cases where backlog would need to be < max-connections? I haven't yet.

Additionally, should it be global? What about when multiple bind()ing using $SERVER["socket"] ?

Yes, i think it should be possible to have backlog < max-connections (that is what most people have now).

I'm not convinced that a high backlog has no drawbacks, so it should be possible to set low values.

I guess we could do it right and support it per $SERVER["socket"] (and take the global value as default)

I think it's important to make a distinction between allowing customisation (which is good), and what the default value should be.

If most people have max-connections > backlog now, it's because the value it statically set. Further, these are users who have spent up to a considerable amount of effort to remove system imposed default limitations for resource consumption.

If 'not set', I still recommend listen() backlog be set equal to max-connections (whether set, and if not, the default), as there is no other more appropriate value to refer to without looking into platform specific settings such as sysctl() for somaxconn), which would be the same as setting an arbitrarily high value for backlog and having the kernel limit it silently.

Consider these cases:

1. listen(fd, 128*8) = what we have now (doesn't consider workload configured for OS and lighttpd)
2. listen(fd, sysctl(SOMAXCONN) = retrieve global OS somaxconn (but requires an additional system call, and functionally equivalent to (3))
3. listen(fd, 999999) = kernel limits to SOMAXCONN silently (but doesn't allow user to configure per application or socket listen queues)
4. listen(fd, config(server.max-connections)) = based on a setting the user has explicitly configured for lighttpd.

eyecue wrote:

It can result in listen queue overflows in cases where burst of new connections > 1024 but < server.max-connections are observed, causing connection resets.

< server.max-connections isn't required for this to happen, if for example keep alive isn't used for whatever reason.

1. listen(fd, 999999) = kernel limits to SOMAXCONN silently (but doesn't allow user to configure per application or socket listen queues)

This looks like a reasonable and simple improvement.

Thanks for the feedback Olaf,

Olaf-van-der-Spek wrote:

< server.max-connections isn't required for this to happen, if for example keep alive isn't used for whatever reason.

Absolutely, my idea was to define the 'most' defined use-case where the symptoms could be exhibited. Clearly there are other instances where queue overflows could occur.

1. listen(fd, 999999) = kernel limits to SOMAXCONN silently (but doesn't allow user to configure per application or socket listen queues)

This looks like a reasonable and simple improvement.

Both options (3) and (4) are relatively trivial to implement with the following distinction:

I agree that option (3) is better than what we have now, but option (4) is more appropriate for the following reasons:

a) It keeps the application-level listen queue within the scope of the application, and not the scope of global system resource limits

b) Its value can be easily based on an pre-existing default value (max-connections=1024)

c) Its value scales automatically with users manually increasing max-connections

The proposed complete solution (for this bug, at present) is as follows:

- Add server.listen-backlog configuration variable
- If not set, server.listen-backlog defaults to server.max-connections

This configuration also paves the way for future work on per-listen() socket queue lengths, for instance:

server.bind = 1.1.1.1
server.max-connections = 2048
server.listen-backlog = 1024
server['socket'] == *:443 {
  server.listen-backlog = 1024
}

If a new connection comes in once a millisecond and lighttpd immediately accept()s it, then at the end of 1 second, lighttpd can have 1000 active connections, and yet the listen backlog queue never had more than one (1) connection waiting in the listen backlog queue.

The choices for size of listen backlog queue depend on how you desire to handle a bursts of traffic, and whether or not you want to leverage the listen backlog queue to tell upstream clients trying to connect that you are busy.

Let's say I have a slow CGI program which can handle 4 requests per second (and let's use small numbers). If the listen backlog queue size is 4, then I can be confident that my server will handle 4 requests per second, and there might be up to 4 more requests waiting, who will see 1 second of latency before I accept the connection, and then 1 second of processing (assuming I process each connection in parallel and processing takes 1 second). This effectively means that if I load test my capacity, I can reliably serve 4 requests per second, and from the perspective of the client, can do so in 2 seconds or less. If more than 8 requests (4 being processed and 4 in the listen backlog queue) come in quickly, then then kernel will reject the TCP connection requests. This can be useful to tell a load balancer that you're busy.

On the other hand, if you can serve 1000 requests per second and you want to be able to handle a sudden surge of 4000 connection requests in the one second, you might want to configure a larger listen backlog queue since you expect to be able to catch up quickly. You might have server.max-connections=1000, but want your listen backlog queue to be 4000. However, if your listen backlog queue is too large or you take too long to process requests, then by the time you accept() after a few seconds, the client may have given up, but you do not know this yet. You go ahead and waste resources processing the request, leaving new connections at the end of the listen backlog queue to get old waiting for their turn. The condition is sometime known as livelock. (http://www.hpl.hp.com/techreports/98/HPL-98-164R1.pdf) In short, there are definitely bad scenarios that can arise from using too large a value for listen backlog queue size.

I agree that the listen backlog value should be configurable, but I think the default value should remain at 1024 (where, if larger than SOMAXCONN, is silently reduced by the kernel to SOMAXCONN, which is typically 128).