[Solved] Request timeout without any response
Added by dinkan over 7 years ago
lighttpd-1.4.35 on
mipsel Linux
For some requests with timeout 20 seconds, lighttpd did not respond & request timed out.
Client application is also in the same box using port 8080 to communicate with server.
Among 100 linux embedded boxes with same software this happens only 5-6 times daily.
The timing is random.
Almost all request from the client is served, except one or two in a week or so.
I checked the access log, there is log corresponding to this failed request.
I enabled the debug log & still no clue of those failed requests received.
Config spec in the bottom of this post
Also would like to know the purpose of this 100 retries.
What happens if accept fails for 100 tries?
for (loops = 0; loops < 100 && NULL != (con = connection_accept(srv, srv_socket)); loops++) {
handler_t r;
connection_state_machine(srv, con);
switch(r = plugins_call_handle_joblist(srv, con)) {
case HANDLER_FINISHED:
case HANDLER_GO_ON:
break;
default:
log_error_write(srv, __FILE__, __LINE__, "d", r);
break;
}
}
#### log directory
var.log_root = "/mnt/log/webdmn/log/"
#### modules to load
server.modules = ( "mod_auth",
"mod_access",
"mod_evasive",
"mod_fastcgi",
"mod_status",
"mod_rewrite",
"mod_redirect",
"mod_accesslog" )
#### performance options (aggressive timeouts)
server.max-keep-alive-requests = 6
server.max-keep-alive-idle = 15
server.max-read-idle = 15
server.max-write-idle = 15
## number of child worker processes to spawn (0 for lightly loaded sites)
# server.max-worker = 0
## number of file descriptors (leave off for lighty loaded sites)
server.max-fds = 640
## maximum concurrent connections the server will accept (1/2 of server.max-fds)
server.max-connections = 320
## single client connection bandwidth limit in kilobytes (0=unlimited)
connection.kbytes-per-second = 0
## global server bandwidth limit in kilobytes (0=unlimited)
server.kbytes-per-second = 0
#### bind to interface (default: all interfaces)
server.bind = "0.0.0.0"
#### bind to port (default: 80)
server.port = "9999"
#### run daemon as uid (default: don't care)
server.username = "nfsnobody"
#### run daemon as gid (default: don't care)
server.groupname = "nfsnobody"
#### set the pid file (newsyslog)
server.pid-file = "/tmp/lighttpd.pid"
#### name the server daemon publicly displays
server.tag = "echostar"
#### static document-root
server.document-root = "/var/mnt/sft/server"
#### chroot() to directory (default: no chroot() )
# server.chroot = "/"
#### files to check for if .../ is requested
index-file.names = ( "index.html" )
#### disable auto index directory listings
dir-listing.activate = "disable"
#### expire module
expire.url = ( "" => "access plus 6 hours")
#### accesslog format (enable for using a proxy, like Pound, in front of Lighttpd)
# accesslog.format = "%{X-Forwarded-For}i %V %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""
accesslog.format = "%h %V:%p %X in%I out%O %u %t \"%r\" %s %b"
#### accesslog module
accesslog.filename = log_root + "lighttpd_access.log"
#### error log
server.errorlog = log_root + "lighttpd_error.log"
#### enable debugging (un-comment to debug server problems)
auth.debug = 2
fastcgi.debug = 1
server.breakagelog = log_root + "breakage.log"
#### mod_evasive
evasive.max-conns-per-ip = 64
#### limit request method "POST" size in kilobytes (KB)
#server.max-request-size = 1
#### disable multi range requests
server.range-requests = "disable"
#### disable symlinks
server.follow-symlink = "disable"
#### ONLY serve files with all lowercase file names
server.force-lowercase-filenames = "disable"
#### server status
status.status-url = "/hidden_dir/server-status"
#### mimetype mapping
mimetype.assign = (
".pdf" => "application/pdf",
".sig" => "application/pgp-signature",
".spl" => "application/futuresplash",
".class" => "application/octet-stream",
".ps" => "application/postscript",
".torrent" => "application/x-bittorrent",
".dvi" => "application/x-dvi",
".gz" => "application/x-gzip",
".pac" => "application/x-ns-proxy-autoconfig",
".swf" => "application/x-shockwave-flash",
".tar.gz" => "application/x-tgz",
".tgz" => "application/x-tgz",
".tar" => "application/x-tar",
".zip" => "application/zip",
".mp3" => "audio/mpeg",
".m3u" => "audio/x-mpegurl",
".wma" => "audio/x-ms-wma",
".wax" => "audio/x-ms-wax",
".ogg" => "application/ogg",
".wav" => "audio/x-wav",
".gif" => "image/gif",
".jpg" => "image/jpeg",
".jpeg" => "image/jpeg",
".png" => "image/png",
".xbm" => "image/x-xbitmap",
".xpm" => "image/x-xpixmap",
".xwd" => "image/x-xwindowdump",
".css" => "text/css",
".html" => "text/html",
".htm" => "text/html",
".js" => "text/javascript",
".asc" => "text/plain",
".c" => "text/plain",
".cpp" => "text/plain",
".log" => "text/plain",
".conf" => "text/plain",
".text" => "text/plain",
".txt" => "text/plain",
".dtd" => "text/xml",
".xml" => "text/xml",
".mpeg" => "video/mpeg",
".mpg" => "video/mpeg",
".mov" => "video/quicktime",
".qt" => "video/quicktime",
".avi" => "video/x-msvideo",
".asf" => "video/x-ms-asf",
".asx" => "video/x-ms-asf",
".wmv" => "video/x-ms-wmv",
".bz2" => "application/x-bzip",
".tbz" => "application/x-bzip-compressed-tar",
".tar.bz2" => "application/x-bzip-compressed-tar"
)
#### denied extensions
static-file.exclude-extensions = ( ".fcgi", ".php", ".rb", "~", ".inc" )
$SERVER["socket"] == ":8080" {
accesslog.filename = log_root + "lighttpd_access_local.log"
#### rewrite and redirect rules
url.rewrite-once = (
"^/www/myurl" => "app.test",
"^/www/myurl_ssl" => "app.test",
"^/myurl_noauth" => "noauth.test",
)
# connects to port 2005
".test" =>
(("host" => "127.0.0.1",
"port" => 2005,
"check-local" => "disable"
)),
Replies (3)
RE: Request timeout without any response - Added by gstrauss over 7 years ago
You're using a very old version of lighttpd. Please upgrade.
If your response timeout is not long enough for your code to respond, then you might try increasing the timeout.
The following accepts up to 100 new connections before continuing to process other connections and periodic jobs. If more connections are pending, they will begin to be accepted the next time through the event loop.
for (loops = 0; loops < 100 && NULL != (con = connection_accept(srv, srv_socket)); loops++) {
RE: Request timeout without any response - Added by dinkan over 7 years ago
Thanks for the reply gstrauss.
This is happening for random requests.
That too very rare. There is no log on the request received on error log.
BTW forgot to mentioned these are post requests to FCGI applicaiton
Do you know whether there is any possible fix for this issue in newer versions?
Curious to know this issue id already handled.
I edited the code we discussed to the following
for (loops = 0; loops < 30; loops++) {
if(NULL != (con = connection_accept(srv, srv_socket)))
{
handler_t r;
connection_state_machine(srv, con);
switch(r = plugins_call_handle_joblist(srv, con)) {
case HANDLER_FINISHED:
case HANDLER_GO_ON:
break;
default:
log_error_write(srv, __FILE__, __LINE__, "d", r);
break;
}
}
else
{
log_error_write(srv, __FILE__, __LINE__, "s",
"connection_accept() returned NULL");
}
}
I see the following error messages sometimes happening for some requests.
Does this say anything about the issue?
I do not see any time-outs or failures on client, may be the retries did work
2016-12-01 12:06:54: (connections.c.1294) accept failed: Resource temporarily unavailable 11 2016-12-01 12:06:54: (network.c.86) connection_accept() returned NULL 2016-12-01 12:06:54: (connections.c.1294) accept failed: Resource temporarily unavailable 11 2016-12-01 12:06:54: (network.c.86) connection_accept() returned NULL 2016-12-01 12:06:54: (connections.c.1294) accept failed: Resource temporarily unavailable 11 2016-12-01 12:06:54: (network.c.86) connection_accept() returned NULL 2016-12-01 12:06:54: (connections.c.1294) accept failed: Resource temporarily unavailable 11 2016-12-01 12:06:54: (network.c.86) connection_accept() returned NULL 2016-12-01 12:06:54: (connections.c.1294) accept failed: Resource temporarily unavailable 11 2016-12-01 12:06:54: (network.c.86) connection_accept() returned NULL 2016-12-01 12:06:54: (connections.c.1294) accept failed: Resource temporarily unavailable 11 2016-12-01 12:06:54: (network.c.86) connection_accept() returned NULL
RE: Request timeout without any response - Added by gstrauss over 7 years ago
Do you know whether there is any possible fix for this issue in newer versions?
Curious to know this issue id already handled.
Very likely already handled, including better resource handling of FastCGI with POST, but without more specific error messages and information surrouding what you are observing, I can't be sure. 1.4.35 was released over 2 1/2 years ago and there have been lots of fixes and improvements committed in the 497 commits since then (in 1.4.36, 1.4.37, 1.4.38, 1.4.39, 1.4.40, 1.4.41, 1.4.42, 1.4.43). Do this give you a hint of how far out of date your version is and why you are being told to upgrade instead of attempting to troubleshoot such an old version?
I edited the code we discussed to the following
...
I see the following error messages sometimes happening for some requests.
Does this say anything about the issue?
No. That's how that loop is supposed to work.
Please try lighttpd git master
$ git clone https://git.lighttpd.net/lighttpd/lighttpd1.4.git
or lighttpd 1.4.43 plus subsequent 40e05e9b