Project

General

Profile

[Solved] Connection break during file transfer. ('server.stream-response-body = 2')

Added by laszlo.gyimothy about 1 month ago

Hi All,

I have implemented a server in C. My server consists of two EXE files, the Lighttpd.exe and the BusinessLogic.exe. They communicate using the FastCGI.

My use case is the following:

1. the client sends a GET request to the server
2. the server starts to generate the response file and sends it back to the client in chunked mode

The problem is the following:

After a given time the connection is suddenly closed by the Lighttpd server during the active file transfer. (if the generated data blocks are smaller, this connection break happens quicker)

Wireshark trace: there is an '\r\n' at the end of the last TCP chunk (this chunk is shorter than the others) and then the connection is finalized by the server.

Business logic: fastCGI returns a negative number (FCGX_PutStr) and the connection is closed.

My solution is the following:

1. the business logic generates the data and send them to the Lighttpd in a while loop: "n_write = FCGX_PutStr((const char *)p_send_buf, (int)len_send_buf, p_response->p_fcgi->out);"
2. if the fastCGI cannot write all of the data into the stream (n_write < len_send_buf) then I wait for 100 ms and try to write the rest of the data block later
3. the loop runs still all of the data sent

OS:

Linux, Windows

The Lighttpd server configuration:

lighttpd main configuration ##
general variables
var.log_root = "/var/log" 
var.server_root = "/var/www" 
var.state_dir = "/var/run" 
var.home_dir = "/var/lib/lighttpd" 
load server modules
server.modules = (
"mod_access",
"mod_accesslog",
"mod_alias",
"mod_cgi",
"mod_compress",
"mod_status",
"mod_redirect",
"mod_rewrite",
"mod_fastcgi",
)
general server definitions
server.port = 80
server.bind = "0.0.0.0" 
server.username = "www-data" 
server.groupname = "www-data" 
server.tag = "Test" 
server.document-root = "til/apps/test" 
server.max-fds = 2048
server.max-connections = 128
#server.event-handler = "libev" 
server.network-backend = "writev" 
server.upload-dirs = ( "c:\tmp" )
server.max-keep-alive-requests = 200000
server.stat-cache-engine = "simple" 
server.follow-symlink = "disable" 
server.stream-response-body = 2
supported index files
index-file.names += (
"index.xhtml", "index.html", "index.htm", "default.htm" 
)
disable dirlisting
dir-listing.activate = "disable" 
additional configuration
include "debug.conf" 
include "rewrite.conf" 
include "fastcgi.conf" 
include "redirect.conf" 
include "alias.conf" 
include "mime.conf" 

The FastCGI configuration:

fastcgi configuration ##
fastcgi.debug = 0
fastcgi.server = (

begin entry
"/test/" => ((
"host" => "127.0.0.1",
"port" => "41280",
"check-local" => "disable" 
)),
end entry
begin entry
"/test/" => ((
"host" => "127.0.0.1",
"port" => "41280",
"check-local" => "disable" 
)),
end entry
begin entry
"/test_internal/" => ((
"host" => "127.0.0.1",
"port" => "39312",
"check-local" => "disable" 
),(
"host" => "127.0.0.1",
"port" => "39313",
"check-local" => "disable" 
)),
end entry
Add new entries before this line
)

Do you have any idea?

Thank you for your help!

Laszlo


Replies (11)

RE: Connection break during file transfer. ('server.stream-response-body = 2') - Added by gstrauss about 1 month ago

As noted in https://redmine.lighttpd.net/boards/2/topics/7503

It is recommended that your backend FastCGI not send Transfer-Encoding: chunked. Just send the response from your backend, and do not include HTTP/1.1 hop-by-hop headers (such as Transfer-Encoding), and do not send HTTP/1.1 "chunked"-encoded bits of data, since your backend is speaking FastCGI protocol, not HTTP/1.1 protocol.

Are you still sending Transfer-Encoding: chunked from your FastCGI backend? If you are, then stop. Transfer-Encoding: chunked is not "business logic".

RE: Connection break during file transfer. ('server.stream-response-body = 2') - Added by laszlo.gyimothy about 1 month ago

Thank you again for your quick response!

No, I dont send any HTTP relevant information anymore from the "business logic". And in the Wireshark trace I can see that the HTTP response header includes "Transfer-Encoding: chunked", added by the Lighttpd server.

And previously I tested only using big (>8k) and very small data blocks (<4bytes) and it worked fined. But sending e.g.: 10 bytes data blocks, you can reproduce the problem.

RE: Connection break during file transfer. ('server.stream-response-body = 2') - Added by laszlo.gyimothy about 1 month ago

I did some screenshots about the Wireshark trace, see attachement! (the trace itself is too big to attacht it)

RE: Connection break during file transfer. ('server.stream-response-body = 2') - Added by laszlo.gyimothy about 1 month ago

Some additional information:

1. when the error happens: the FCGX_PutStr(...) returns with -1 and the fastCGI error code is 104 (Connection reset by peer) /the "business layer" doesnt close the connection/

2. and this is what I can see in the Lighttpd trace:

Aug 11 12:00:07 sicam lighttpd[1057]: (response.c.547) Doc-Root     : /ies/apps/sicweb 
Aug 11 12:00:07 sicam lighttpd[1057]: (response.c.548) Basedir      : /ies/apps/sicweb 
Aug 11 12:00:07 sicam lighttpd[1057]: (response.c.549) Rel-Path     : /rtum85_internal/system 
Aug 11 12:00:07 sicam lighttpd[1057]: (response.c.550) Path         : /ies/apps/sicweb/rtum85_internal/system 
Aug 11 12:00:07 sicam lighttpd[1057]: (mod_fastcgi.c.3041) got proc: pid: 0 socket: tcp:127.0.0.1:33717 load: 1 
Aug 11 12:00:07 sicam lighttpd[1057]: (response.c.122) Response-Header: \nHTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\nDate: Fri, 11 Aug 2017 10:00:07 GMT\r\nServer: A8000\r\n\r\n 
Aug 11 12:00:10 sicam lighttpd[1057]: (mod_fastcgi.c.2575) unexpected end-of-file (perhaps the fastcgi process died): pid: 0 socket: tcp:127.0.0.1:33717 
Aug 11 12:00:10 sicam lighttpd[1057]: (mod_fastcgi.c.3371) response already sent out, but backend returned error on socket: tcp:127.0.0.1:33717 for /rtum85_internal/system?SICWEB-SID=f3+ZO+Yy8+o34x2rKgcjH2d501C6WoyEClGxWI4U9dWEj2vf, terminating connection 
Aug 11 12:00:10 sicam lighttpd[1057]: (mod_fastcgi.c.1590) released proc: pid: 0 socket: tcp:127.0.0.1:33717 load: 0 
Aug 11 12:00:10 sicam lighttpd[1057]: (configfile-glue.c.692) === start of condition block === 
Aug 11 12:00:10 sicam lighttpd[1057]: (configfile-glue.c.620) 1 (cached) result: true 
Aug 11 12:00:10 sicam lighttpd[1057]: (configfile-glue.c.692) === start of condition block === 
Aug 11 12:00:10 sicam lighttpd[1057]: (configfile-glue.c.620) 2 (cached) result: false 
Aug 11 12:00:10 sicam lighttpd[1057]: (configfile-glue.c.692) === start of condition block === 
Aug 11 12:00:10 sicam lighttpd[1057]: (configfile-glue.c.620) 3 (cached) result: skipped

RE: Connection break during file transfer. ('server.stream-response-body = 2') - Added by gstrauss about 1 month ago

Does this occur when using server.stream-response-body = 1 ?

What version of lighttpd are you using? lighttpd 1.4.41 or 1.4.42? (based on the line numbers reported above)

Before wasting everyone's time, please test with the latest release, currently lighttpd 1.4.45.

When requesting help, always include version numbers where available.

RE: Connection break during file transfer. ('server.stream-response-body = 2') - Added by laszlo.gyimothy 25 days ago

I am sorry for the late answer and for the missing information!

1. Using server.stream-response-body = 1 -> it works fine! (I have tested it with 1.4.41 and 1.4.45.)
2. Previously I tested only with 1.4.41, this week I updated to 1.4.45 but I have the same problem, server.stream-response-body = 2 -> connection break.

Aug 25 13:42:01 sicam lighttpd[1058]: (response.c.563) Doc-Root     : /ies/apps/sicweb 
Aug 25 13:42:01 sicam lighttpd[1058]: (response.c.564) Basedir      : /ies/apps/sicweb 
Aug 25 13:42:01 sicam lighttpd[1058]: (response.c.565) Rel-Path     : /rtum85_internal/system 
Aug 25 13:42:01 sicam lighttpd[1058]: (response.c.566) Path         : /ies/apps/sicweb/rtum85_internal/system 
Aug 25 13:42:01 sicam lighttpd[1058]: (mod_fastcgi.c.2875) got proc: pid: 0 socket: tcp:127.0.0.1:33718 load: 1 
Aug 25 13:42:01 sicam lighttpd[1058]: (response.c.122) Response-Header: \nHTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\nDate: Fri, 25 Aug 2017 11:42:01 GMT\r\nServer: A8000\r\n\r\n 
Aug 25 13:42:05 sicam lighttpd[1058]: (mod_fastcgi.c.2424) unexpected end-of-file (perhaps the fastcgi process died): pid: 0 socket: tcp:127.0.0.1:33718 
Aug 25 13:42:05 sicam lighttpd[1058]: (mod_fastcgi.c.3180) response already sent out, but backend returned error on socket: tcp:127.0.0.1:33718 for /rtum85_internal/system?SICWEB-SID=URjbJYi+oUXSjX7rhko2pDtCKiQrI4GPcT7bGrTa42txf29V, terminating connection 
Aug 25 13:42:05 sicam lighttpd[1058]: (mod_fastcgi.c.1642) released proc: pid: 0 socket: tcp:127.0.0.1:33718 load: 0 
Aug 25 13:42:05 sicam lighttpd[1058]: (configfile-glue.c.694) === start of condition block === 
Aug 25 13:42:05 sicam lighttpd[1058]: (configfile-glue.c.622) 1 (cached) result: true

Do you have any idea?

RE: Connection break during file transfer. ('server.stream-response-body = 2') - Added by gstrauss 14 days ago

Thanks for the additional information. I haven't had a chance yet to try to reproduce this, but would be interested if this is still happening with lighttpd git master (git.lighttpd.net) which is intended to be lighttpd 1.4.46 soon. See DevelGit

RE: Connection break during file transfer. ('server.stream-response-body = 2') - Added by gstrauss 14 days ago

The issue above sounds similar to https://redmine.lighttpd.net/issues/2796, fixed in e4bb5622 and 0a635fc8 back in March, and so I hope that you can confirm that the issue you are seeing is no longer present in lighttpd git master.

RE: Connection break during file transfer. ('server.stream-response-body = 2') - Added by laszlo.gyimothy 14 days ago

Thank you! I will test it using the latest version and give you a feedback!

RE: Connection break during file transfer. ('server.stream-response-body = 2') - Added by laszlo.gyimothy 12 days ago

Using the git master version I cannot reproduce the problem anymore! Thank you for your help!
(In connection with 1.4.46: what does "intended to be lighttpd 1.4.46 soon" mean? Is is planned to release it in one month? Or does soon mean a longer period?)

    (1-11/11)