Project

General

Profile

"sockets disabled, connection limit reached" - server.max-fds and open files is already high, but doesn't help

Added by mangoo about 14 years ago

I have a problem with recurring "sockets disabled, connection limit reached".

2011-04-21 10:52:58: (server.c.1391) [note] sockets disabled, connection limit reached 
2011-04-21 10:53:08: (server.c.1345) [note] sockets enabled again 
2011-04-21 10:53:08: (server.c.1391) [note] sockets disabled, connection limit reached 
2011-04-21 10:53:15: (server.c.1345) [note] sockets enabled again 
2011-04-21 10:53:15: (server.c.1391) [note] sockets disabled, connection limit reached 
2011-04-21 10:53:19: (server.c.1345) [note] sockets enabled again 

I tried following some advices on lighttpd forum, and increased the limits from 1024 to 2048, but it didn't help.

I have:

server.max-fds = 2048
#  cat /proc/`ps ax | grep lighttpd | grep -v grep | awk -F " " '{print $1}'`/limits |grep "Max open files" 
Max open files            2048                 2048                 files     

I've been also running netstat and lsof to see if I might be hitting the 2048 limit, but it's not even close:

lsof - number of files open by lighttpd (lsof -n | grep -c lighttpd):

2011-04-21-10:48:01 1156
2011-04-21-10:49:01 1189
2011-04-21-10:50:01 1222
2011-04-21-10:51:01 1225
2011-04-21-10:52:01 1296
2011-04-21-10:53:01 1200

netstat - open connections, system wide (netstat -tpna|wc -l):

     1420 2011-04-21-10:48:01
     1336 2011-04-21-10:49:01
     1221 2011-04-21-10:50:01
     1302 2011-04-21-10:51:01
     1126 2011-04-21-10:52:01
     1221 2011-04-21-10:53:01

Why is lighttpd complaining? What else do I have to change, apart from server.max-fds and the limit of open files? How can I debug it further?


Replies (10)

RE: "sockets disabled, connection limit reached" - server.max-fds and open files is already high, but doesn't help - Added by icy about 14 years ago

You also have to raise the maximum for connections.

I suggest setting max fds to 10k and max connections to 3k to see if it helps.

Also keep an eye on the mod_status page to see why you have so many connections and maybe consider lowering the keep alive timeout.

RE: "sockets disabled, connection limit reached" - server.max-fds and open files is already high, but doesn't help - Added by mangoo about 14 years ago

Unfortunately, it doesn't seem to help.

I have:

server.max-fds = 8192
server.max-connections = 4096
# cat /proc/`ps ax | grep lighttpd | grep -v grep | awk -F " " '{print $1}'`/limits |grep "Max open files" 
Max open files            2048                 2048                 files   

lsof and netstat started every minute show that generally I hit up to ~1100 opened files when lighttpd shows "sockets disabled, connection limit reached".

Is there any way to debug this any further?

RE: "sockets disabled, connection limit reached" - server.max-fds and open files is already high, but doesn't help - Added by mangoo almost 14 years ago

I have it set to:

server.max-keep-alive-requests = 8
server.max-keep-alive-idle = 3

But it doesn't matter what I have it set to.

What matters, is that as soon as lighttpd reaches 1024 connections open, it refuses any communication - despite having these limits sets way beyond 1024.

RE: "sockets disabled, connection limit reached" - server.max-fds and open files is already high, but doesn't help - Added by mangoo almost 14 years ago

What kind of limits? The one below? I've set it to 2048 from default 1024; according to lsof, I'm hitting the "sockets disabled, connection limit reached" when lighttpd has around 1000-1200 files open - so is still about 1000 files away from the limit.

# cat /proc/`ps ax | grep lighttpd | grep -v grep | awk -F " " '{print $1}'`/limits |grep "Max open files" 
Max open files            2048                 2048                 files  

I'm using Linux - should I be looking at something else?

RE: "sockets disabled, connection limit reached" - server.max-fds and open files is already high, but doesn't help - Added by KuJoe almost 14 years ago

Which OS are you using? I had a similiar issue with CentOS but when I setup Lighttpd on Arch Linux with the exact same config the problem never returned. I believe the issue is how the OS has the network setup and not with Lighttpd specifically.

RE: "sockets disabled, connection limit reached" - server.max-fds and open files is already high, but doesn't help - Added by mangoo almost 14 years ago

I'm using Debian Lenny, which has 2.6.26 kernel. It doesn't have any SELinux etc. security policies enabled (which could be the case with CentOS).

RE: "sockets disabled, connection limit reached" - server.max-fds and open files is already high, but doesn't help - Added by stbuehler almost 14 years ago

lighty requires at least two fds per connection (one socket, one for backend, static files, ...)

RE: "sockets disabled, connection limit reached" - server.max-fds and open files is already high, but doesn't help - Added by Leath almost 14 years ago

Hi,

I have the same problem with my lighttpd installation on an up to date Debian Squeeze. This is my lighttpd.conf:


server.modules = (
"mod_access",
"mod_alias",
"mod_compress",
"mod_redirect",
"mod_rewrite",
)

#server.document-root = "/var/www"
server.document-root = "/drbd/www"
server.upload-dirs = ( "/var/cache/lighttpd/uploads" )
server.errorlog = "/var/log/lighttpd/error.log"
server.pid-file = "/var/run/lighttpd.pid"
server.username = "www-data"
server.groupname = "www-data"
server.max-fds = 8192
server.max-connections = 4096

index-file.names = ( "index.php", "index.html",
"index.htm", "default.htm",
" index.lighttpd.html" )

url.access-deny = ( "~", ".inc" )

static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )

include_shell "/usr/share/lighttpd/use-ipv6.pl"

dir-listing.encoding = "utf-8"
#server.dir-listing = "enable"
server.dir-listing = "disable"

compress.cache-dir = "/var/cache/lighttpd/compress/"
compress.filetype = ( "application/x-javascript", "text/css", "text/html", "text/plain" )

include_shell "/usr/share/lighttpd/create-mime.assign.pl"
include_shell "/usr/share/lighttpd/include-conf-enabled.pl"
fastcgi.server = (".php" =>
( "localhost" =>
(
"socket" => "/tmp/php-fastcgi.socket",
"bin-path" => "/usr/bin/php-cgi",
"bin-environment" => (
"PHP_FCGI_MAX_REQUESTS" => "10000"
)
)
)
)


Enabled modules are: auth fastcgi simple-vhost status

lsof -n | grep -c lighttpd never goes above 130 and /server-status(generated from status plugin) counts about 100 connections max. Never the less, I got "sockets disabled, connection limit reached", which never resets.

Any ideas for deeper debugging or solutions?

    (1-10/10)