Project

General

Profile

Feature #2116

add server.listen-backlog option instead of hard-coded value (128 * 8) for listen()

Added by eyecue over 7 years ago. Updated about 1 year ago.

Status:
Fixed
Priority:
Low
Assignee:
-
Category:
core
Target version:
Start date:
2009-12-10
Due date:
% Done:

0%

Missing in 1.5.x:
No

Description

Lighttpd currently sets it's queue length to a hard coded value of 1024 (128 * 8) when calling listen(). This is non-optimal in circumstances where both the operating system 'and' lighttpd.conf have been configured for maxconn values > 1024.

It can result in listen queue overflows in cases where burst of new connections > 1024 but < server.max-connections are observed, causing connection resets.

On FreeBSD, netstat -Lan can display per-listening-socket queue lengths, but I don't think there's an equivalent command on Linux.

Current Behaviour:

lighttpd.conf: server.max-connections: NOT SET (defaults to 1024)

Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen         Local Address
tcp4  0/0/1024       *.80

lighttpd.conf: server.max-connections: 2048

Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen         Local Address
tcp4  0/0/1024       *.80

lighttpd.conf: server.max-connections: 512

Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen         Local Address
tcp4  0/0/1024       *.80

Relevant code section in question:

http://redmine.lighttpd.net/projects/lighttpd/repository/entry/branches/lighttpd-1.4.x/src/network.c

if (-1 == listen(srv_socket->fd, 128 * 8)) {

Expected Behaviour:

server.max-connections value should be used for listen()

Slightly off-topic, HAProxy exhibits the correct behaviour of reflecting it's per frontend maxconn values in the listen queue lengths. I mention this purely for a representative and example case of expected behaviour.

Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen         Local Address
tcp4  0/0/2048       *.443
tcp4  0/0/4096       *.80
tcp4  0/0/2000       *.1357

Related issues

Related to Bug #1825: Don't disable backend when overloaded Invalid 2008-11-18

Associated revisions

Revision 71ed1912 (diff)
Added by gstrauss about 1 year ago

[config] server.listen-backlog option (fixes #1825, #2116)

See doc/config/lighttpd.conf for explanation of listen() backlog queue

Additionally, mod_fastcgi and mod_scgi backend servers can now also be
configured with separate listen-backlog settings per server

x-ref:
"add server.listen-backlog option instead of hard-coded value (128 * 8) for listen()"
https://redmine.lighttpd.net/issues/2116
"Don't disable backend when overloaded"
https://redmine.lighttpd.net/issues/1825

github:
Closes #50

History

#1 Updated by stbuehler over 7 years ago

  • Priority changed from High to Low
  • Target version changed from 1.4.26 to 1.4.x

I really see no direct relation between the two values. High max-connections is for long-running connections (slow clients, large files), somaxconn for handling bursts. And i really think 1024 should be enough for bursts.

And i see no high priority - patch it yourself if you really think you need it, although i guess there is something wrong with your setup if you do need that (sounds like something is blocking instead of async).

#2 Updated by eyecue over 7 years ago

I will concede that max-connections and listen() backlog values are not 'synonymous', but they are absolutely related.

server.max-connections is exactly that, the maximum number of connections that will be actively 'handled' by the service. It makes no inference whatsoever as to whether they are short-lived, long-running, or the rate at which they come in.

The listen() backlog value, on the other hand, if set incorrectly can directly impact service whether the hardware or application can handle it or not. If a server is capable of processing 4096 concurrent requests, then a static listen backlog of 1024 is tantamount to an artificially imposed bottleneck in use cases such as that mentioned in the description.

As far as whether 1024 should be good enough or not is entirely dependent on the workload in question. Remember, 640k 'should have been' enough memory for anybody too :]

I agree on the Low priority, and would have re-prioritised the ticket given the permissions. I incorrectly assumed that a hard-coded value in an application with an otherwise great track record in high-performance web serving might have been a forgotten TODO item and just never got picked up.

Additionally, apache and nginx both provide user-configurable options for setting the listen() backlog value:

http://httpd.apache.org/docs/2.2/mod/mpm_common.html#listenbacklog

nginx goes one further and defaults it to -1 (SOMAXCONN)

http://wiki.nginx.org/NginxHttpCoreModule#listen

I'd patch it, but it's not that 'I' need it, I think lighttpd and its community would be better for it.

This report is merely to demonstrate the fact that a hard-coded value that makes assumptions about its environment and workload could be improved upon. Whether this results in a better default value (max-connection or -1 (SOMAXCONN)), a configuration option, or a value that scales with something else is entirely up to you and your team.

If it were me, I'd like to see a server.listen-backlog directive.

Edit: If nothing else and at the absolute least, it's a case of the Magic Number Anti-Pattern.

#3 Updated by stbuehler over 7 years ago

  • Tracker changed from Bug to Feature

#4 Updated by stbuehler over 7 years ago

  • Subject changed from listen() call should use server.max-connections instead of hard-coded value (128 * 8) to add server.listen-backlog option instead of hard-coded value (128 * 8) for listen()

I think a config option is a good idea; i even could live with a higher default (as at least on linux the default system limit is 128 iirc)

#5 Updated by eyecue over 7 years ago

Awesome stuff :]

From what I read, if listen()'s backlog is set to > SOMAXCONN, the kernel will silently limit it to SOMAXCONN. (FreeBSD currently defaults to 128 as well)

For the sake of argument (and possibly avoiding 'yet another configuration option'), can you think of any cases where backlog would need to be < max-connections? I haven't yet.

Additionally, should it be global? What about when multiple bind()ing using $SERVER["socket"] ?

#6 Updated by stbuehler over 7 years ago

Yes, i think it should be possible to have backlog < max-connections (that is what most people have now).

I'm not convinced that a high backlog has no drawbacks, so it should be possible to set low values.

I guess we could do it right and support it per $SERVER["socket"] (and take the global value as default)

#7 Updated by eyecue over 7 years ago

I think it's important to make a distinction between allowing customisation (which is good), and what the default value should be.

If most people have max-connections > backlog now, it's because the value it statically set. Further, these are users who have spent up to a considerable amount of effort to remove system imposed default limitations for resource consumption.

If 'not set', I still recommend listen() backlog be set equal to max-connections (whether set, and if not, the default), as there is no other more appropriate value to refer to without looking into platform specific settings such as sysctl() for somaxconn), which would be the same as setting an arbitrarily high value for backlog and having the kernel limit it silently.

Consider these cases:

  1. listen(fd, 128*8) = what we have now (doesn't consider workload configured for OS and lighttpd)
  2. listen(fd, sysctl(SOMAXCONN) = retrieve global OS somaxconn (but requires an additional system call, and functionally equivalent to (3))
  3. listen(fd, 999999) = kernel limits to SOMAXCONN silently (but doesn't allow user to configure per application or socket listen queues)
  4. listen(fd, config(server.max-connections)) = based on a setting the user has explicitly configured for lighttpd.

#8 Updated by Olaf-van-der-Spek over 7 years ago

eyecue wrote:

It can result in listen queue overflows in cases where burst of new connections > 1024 but < server.max-connections are observed, causing connection resets.

< server.max-connections isn't required for this to happen, if for example keep alive isn't used for whatever reason.

  1. listen(fd, 999999) = kernel limits to SOMAXCONN silently (but doesn't allow user to configure per application or socket listen queues)

This looks like a reasonable and simple improvement.

#9 Updated by eyecue over 7 years ago

Thanks for the feedback Olaf,

Olaf-van-der-Spek wrote:

< server.max-connections isn't required for this to happen, if for example keep alive isn't used for whatever reason.

Absolutely, my idea was to define the 'most' defined use-case where the symptoms could be exhibited. Clearly there are other instances where queue overflows could occur.

  1. listen(fd, 999999) = kernel limits to SOMAXCONN silently (but doesn't allow user to configure per application or socket listen queues)

This looks like a reasonable and simple improvement.

Both options (3) and (4) are relatively trivial to implement with the following distinction:

I agree that option (3) is better than what we have now, but option (4) is more appropriate for the following reasons:

a) It keeps the application-level listen queue within the scope of the application, and not the scope of global system resource limits

b) Its value can be easily based on an pre-existing default value (max-connections=1024)

c) Its value scales automatically with users manually increasing max-connections

The proposed complete solution (for this bug, at present) is as follows:

- Add server.listen-backlog configuration variable
- If not set, server.listen-backlog defaults to server.max-connections

This configuration also paves the way for future work on per-listen() socket queue lengths, for instance:

server.bind = 1.1.1.1
server.max-connections = 2048
server.listen-backlog = 1024
server['socket'] == *:443 {
  server.listen-backlog = 1024
}

#10 Updated by gstrauss about 1 year ago

  • Related to Bug #1825: Don't disable backend when overloaded added

#11 Updated by gstrauss about 1 year ago

  • Related to Bug #1825: Don't disable backend when overloaded added

#12 Updated by gstrauss about 1 year ago

  • Related to deleted (Bug #1825: Don't disable backend when overloaded)

#13 Updated by gstrauss about 1 year ago

If a new connection comes in once a millisecond and lighttpd immediately accept()s it, then at the end of 1 second, lighttpd can have 1000 active connections, and yet the listen backlog queue never had more than one (1) connection waiting in the listen backlog queue.

The choices for size of listen backlog queue depend on how you desire to handle a bursts of traffic, and whether or not you want to leverage the listen backlog queue to tell upstream clients trying to connect that you are busy.

Let's say I have a slow CGI program which can handle 4 requests per second (and let's use small numbers). If the listen backlog queue size is 4, then I can be confident that my server will handle 4 requests per second, and there might be up to 4 more requests waiting, who will see 1 second of latency before I accept the connection, and then 1 second of processing (assuming I process each connection in parallel and processing takes 1 second). This effectively means that if I load test my capacity, I can reliably serve 4 requests per second, and from the perspective of the client, can do so in 2 seconds or less. If more than 8 requests (4 being processed and 4 in the listen backlog queue) come in quickly, then then kernel will reject the TCP connection requests. This can be useful to tell a load balancer that you're busy.

On the other hand, if you can serve 1000 requests per second and you want to be able to handle a sudden surge of 4000 connection requests in the one second, you might want to configure a larger listen backlog queue since you expect to be able to catch up quickly. You might have server.max-connections=1000, but want your listen backlog queue to be 4000. However, if your listen backlog queue is too large or you take too long to process requests, then by the time you accept() after a few seconds, the client may have given up, but you do not know this yet. You go ahead and waste resources processing the request, leaving new connections at the end of the listen backlog queue to get old waiting for their turn. The condition is sometime known as livelock. (http://www.hpl.hp.com/techreports/98/HPL-98-164R1.pdf) In short, there are definitely bad scenarios that can arise from using too large a value for listen backlog queue size.

I agree that the listen backlog value should be configurable, but I think the default value should remain at 1024 (where, if larger than SOMAXCONN, is silently reduced by the kernel to SOMAXCONN, which is typically 128).

#14 Updated by gstrauss about 1 year ago

  • Target version changed from 1.4.x to 1.4.40

#15 Updated by gstrauss about 1 year ago

  • Status changed from New to Patch Pending

#16 Updated by gstrauss about 1 year ago

  • Status changed from Patch Pending to Fixed

Also available in: Atom