Project

General

Profile

Bug #2787

cgi chunked encoding breaks with 1.4.40

Added by af123 8 months ago. Updated 8 months ago.

Status:
Invalid
Priority:
Normal
Assignee:
-
Category:
mod_cgi
Target version:
Start date:
2017-02-02
Due date:
% Done:

0%

Estimated time:
Missing in 1.5.x:

Description

I am in the process of upgrading from 1.4.39; some of my CGI scripts send data using chunked encoding and this stops working with 1.4.40 and later.

Output from CGI script:

@Content-Type: text/plain; charset="UTF-8"; no-cache
Expires: -1
Connection: close
Pragma: no-cache
Cache-Control: no-cache
Transfer-Encoding: chunked

opkg update

Downloading http://hpkg.tv/hdrfoxt2/base/Packages.gz.
@

and packet trace with 1.4.39 shows:

@HTTP/1.1 200 OK
Content-Type: text/plain; charset="UTF-8"; no-cache
Expires: -1
Connection: close
Pragma: no-cache
Cache-Control: no-cache
Transfer-Encoding: chunked
Date: Thu, 02 Feb 2017 22:43:15 GMT
Server: unknown

11

opkg update

36
Downloading http://hpkg.tv/hdrfoxt2/base/Packages.gz.

@

With 1.4.40, there is a Content-Length: header as well as Transfer-Encoding: and the data is not chunked:

@HTTP/1.1 200 OK
Content-Type: text/plain; charset="UTF-8"; no-cache
Expires: -1
Connection: close
Pragma: no-cache
Cache-Control: no-cache
Transfer-Encoding: chunked
Content-Length: 594
Date: Thu, 02 Feb 2017 23:23:44 GMT
Server: Humax

opkg update

Downloading http://hpkg.tv/hdrfoxt2/base/Packages.gz.
@

resulting in an error page in the browser.

History

#1 Updated by af123 8 months ago

Reverting https://redmine.lighttpd.net/projects/lighttpd/repository/revisions/5863cb57522e46cde5a35784f4ce83459a8561f4/diff/src/mod_cgi.c fixes this for me.
Is it a bug or am I doing something wrong?

Thanks in advance!

#2 Updated by gstrauss 8 months ago

  • Status changed from New to Invalid

Please don't file bugs to ask questions. Yes, you're doing it wrong and inefficiently. You're not sending an NPH response and so you should be sending a CGI response, following the CGI/1.1 gateway protocol, and not an HTTP response (CGI gateway protocol is slightly different from an HTTP proxy).

some of my CGI scripts send data using chunked encoding

Your CGI script should not send Transfer-Encoding: chunked. Just send the response without encoding and let lighttpd choose whether or not to send Transfer-Encoding: chunked back to the client. By default, lighttpd 1.4.40 and later will collect the response before sending to client, and therefore will be able to set Content-Length, unless you set server.stream-response-body. In general, your CGI will respond faster than the remote client can receive the file over the network, so lighttpd might as well fully buffer the response so that your CGI script can exit and free up resources as soon as possible.

See also Docs_ModCGI. If you are merely returning files from disk, it sounds like you should be using cgi.x-sendfile = "enable" and having lighttpd send the file directly from disk (e.g. your script returns response header "X-Sendfile: /path/to/base/Packages.gz") rather than your CGI script reading and copying the file contents through a pipe to lighttpd to then send.

#3 Updated by gstrauss 8 months ago

BTW, latest version of lighttpd is lighttpd 1.4.45. If you're upgrading, please use lighttpd 1.4.45. It is stable and on target to be in the next Debian major release (Debian Stretch), which, I have been told, is (tentatively) going into hard freeze around Feb 5.

#4 Updated by gstrauss 8 months ago

BTW, questions can be asked on the forums: https://redmine.lighttpd.net/projects/lighttpd/boards or on IRC freenode #lighttpd

#5 Updated by af123 8 months ago

Thanks for the replies, I will fix my scripts and look at the new server.stream-response-body option to do what I need.

#6 Updated by gstrauss 8 months ago

Is your script sending Transfer-Encoding: chunked ? If so, I am curious: why? I ask this since I took another look at the trace you provided, and it would seem as if your script is not sending HTTP chunked data, but is sending Transfer-Encoding: chunked. To say one thing and send another is inconsistent and incorrect (in every protocol).

#7 Updated by af123 8 months ago

It seems to be a hold-over from when we used Mongoose as the web server and sent NPH for this one page which generates output slowly. When we switched to lighttpd the chunk lengths were removed from the output but for whatever reason the header was left. It worked up until 1.4.40 so wasn't noticed.

Also available in: Atom