Feature #2156
closedrequest: support Chunked Transfer Coding for HTTP PUT
Added by morgan about 15 years ago. Updated about 7 years ago.
Description
The HTTP spec section 3.6.1 describes Chunked Transfer Coding :
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
which neither lighttpd 1.4 nor 1.5 supports. It would be very useful for cases where clients wish to upload a file without first specifying the Content-length header, perhaps because they are uploading it while it is being generated.
Curl supports it on the client side (handy for testing):
http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTUPLOAD
"If you use PUT to a HTTP 1.1 server, you can upload data without knowing
the size before starting the transfer if you use chunked encoding. You
enable this by adding a header like "Transfer-Encoding: chunked" with
CURLOPT_HTTPHEADER
With HTTP 1.0 or without chunked transfer, you must specify the size."
Would others find this useful ?
Morgan
Files
| lighttpd_1.4.26_chunked_decoding2.patch (18.3 KB) lighttpd_1.4.26_chunked_decoding2.patch | andreas_stoe, 2010-04-29 16:00 | ||
| lighttpd_1.4.26_chunked_decoding3b.patch (19.1 KB) lighttpd_1.4.26_chunked_decoding3b.patch | andreas_stoe, 2010-05-10 09:29 | ||
| lighttpd_1.4.29_chunked_decoding.patch (19.4 KB) lighttpd_1.4.29_chunked_decoding.patch | lvs, 2012-08-17 12:43 | ||
| lighttpd_1.4.43_chunked_prep.patch (12.1 KB) lighttpd_1.4.43_chunked_prep.patch | gstrauss, 2016-12-12 00:29 |
Updated by morgan about 15 years ago
- Target version changed from 1.4.x to 1.4.26
Looks like the 1.4.26 is the place to be, so moving request to 1.4.26 queue.
Updated by stbuehler about 15 years ago
- Target version changed from 1.4.26 to 1.4.x
Sorry, no. Of course this would be useful, but i don't see it coming in 1.x; the code assumes in too many places to know the content-length of the upload after parsing the request headers.
Updated by andreas_stoe almost 15 years ago
Hi,
the attached "hack" adds support for basic "transfer-encoding: chunked" request handling. Chunked data which is sent to the server will be decoded using a small library (chunk_decoder.c, chunk_decoder.h) and is finally stored in tempfiles. When all chunks have been read, con->request.content_length is set accordingly. So all further processing steps in the webserver will believe data has been sent with a "content-length" header.
I hope this patch might be useful,
Andreas
Updated by andreas_stoe almost 15 years ago
My previous patch contained a serious bug which might have caught the web-server to stop responding if a connection got closed abruptly. I also added support for hex numbers written with small letters.
Updated by pprkut almost 15 years ago
I tried your patch, but I couldn't get lighttpd to compile with it. Here's the error I get:
libtool: link: gcc -g -O2 -Wall -W -Wshadow -pedantic -std=gnu99 -o lighttpd server.o response.o connections.o network.o configfile.o configparser.o request.o proc_open.o buffer.o log.o keyvalue.o chunk.o http_chunk.o stream.o fdevent.o stat_cache.o plugin.o joblist.o etag.o array.o data_string.o data_count.o data_array.o data_integer.o md5.o data_fastcgi.o fdevent_select.o fdevent_linux_rtsig.o fdevent_poll.o fdevent_linux_sysepoll.o fdevent_solaris_devpoll.o fdevent_freebsd_kqueue.o data_config.o bitset.o inet_ntop_cache.o crc32.o connections-glue.o configfile-glue.o http-header-glue.o network_write.o network_linux_sendfile.o network_freebsd_sendfile.o network_writev.o network_solaris_sendfilev.o network_openssl.o splaytree.o status_counter.o -Wl,--export-dynamic -L/usr/lib64 /usr/lib64/libpcre.so -ldl /usr/lib64/libfam.so -Wl,-rpath -Wl,/usr/lib64 -Wl,-rpath -Wl,/usr/lib64
connections.o: In function `connection_handle_read_state':
/mnt/progs/slack/test/lighttpd-1.4.26/src/connections.c:1200: undefined reference to `chunk_decoder_work'
/mnt/progs/slack/test/lighttpd-1.4.26/src/connections.c:1336: undefined reference to `chunk_decoder_free'
/mnt/progs/slack/test/lighttpd-1.4.26/src/connections.c:1283: undefined reference to `chunk_decoder_free'
/mnt/progs/slack/test/lighttpd-1.4.26/src/connections.c:1186: undefined reference to `chunk_decoder_create'
connections.o: In function `connection_reset':
/mnt/progs/slack/test/lighttpd-1.4.26/src/connections.c:806: undefined reference to `chunk_decoder_free'
connections.o: In function `connection_close':
/mnt/progs/slack/test/lighttpd-1.4.26/src/connections.c:127: undefined reference to `chunk_decoder_free'
collect2: ld returned 1 exit status
make3: *** [lighttpd] Error 1
Updated by lvs over 12 years ago
I attach the current version of the patch. It applies to 1.4.31, please don't mind the name.
Updated by gstrauss about 9 years ago
FYI: recent patch to lighttpd will communicate to client 411 Length Required, so while Transfer-Encoding: chunked is still not supported, the server and client should behave better when the server sends 411 response.
commit 06d3c754403b2df99775641e05999ea9b2081618
Author: Glenn Strauss <gstrauss@gluelogic.com>
Date: Sat Mar 26 12:58:33 2016 +0000
[core] respond 411 Length Required if request has Transfer-Encoding: chunked (fixes #631)
lighttpd does not currently support request body transfer-codings
From: Glenn Strauss <gstrauss@gluelogic.com>
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@3128 152afb58-edef-0310-8abb-c4023f1b3aa9
Updated by gstrauss almost 9 years ago
- Status changed from New to Need Feedback
https://github.com/lighttpd/lighttpd1.4/pull/53 moves request body reading to connection_handle_read_post_state(). This location might be enhanced to support chunking (and then request.c modified to not reject chunked POST requests with 411 Length Required)
However, this is low priority and might not be worth the effort if clients are responding properly to 411 Length Required.
Is anyone having issues with clients that do not properly handle 411 Length Required?
Updated by gstrauss almost 9 years ago
- Status changed from Need Feedback to Missing Feedback
- Priority changed from Normal to Low
While supporting Transfer-Encoding: chunked for request body would be a nice feature to have, to increase the priority of this feature request, someone would need to present a (non-contrived) use case where this was really needed.
Well-behaved clients should handle PUTs properly now that lighttpd responds with 411 Length Required (which is compliant with the RFCs) if client sends Transfer-Encoding: chunked.
Updated by flynn over 8 years ago
Any progress on this issue?
The relevant usecase is:- webdav with lighttpd/php/sabredav
- OSX finder as client
This combinations results in PUT request with chunked encoding: http://sabre.io/dav/clients/finder/
I would offer extended testing, if a patch is provided.
Updated by gstrauss over 8 years ago
- Status changed from Missing Feedback to Reopened
Thanks for noting your interest and the offer to help test.
Modifying connection_handle_read_post_state() to handle chunked encoding is the central place to implement it. However, con->request.content_length is used in more than a few places to flag the presence of content or not. Those locations would need to be updated to handle delayed content, including sometimes handling delayed content of 0 octets.
While none of this is very difficult, it will take some effort. I'll re-open this feature request, but the priority is still "low" until someone finds some time to do it.
Updated by gstrauss over 8 years ago
Uploading patch to be applied against lighttpd git master HEAD.
This patch changes existing modules to be aware that Transfer-Encoding: chunked may be used, resulting in an unknown Content-Length of request body, where previously the length of request body was known up-front. For most modules, this means returning 411 Length Requiredt to preserve existing behavior.
For modules handling requests that are not configured to stream the request body to the backend, which collect all content prior to starting the backend, it is now possible (but not yet coded) for request body to be collected from client and de-chunked.
Before this patch is committed, it needs some wider testing to ensure that existing behavior has not been broken. Specific behavior that needs testing is POST of request body to various dynamic handlers (mod_cgi, mod_fastcgi, mod_scgi, mod_proxy, mod_webdav).
Future work is largely isolated in connections-glue.c:connection_handle_read_post_state(), which needs to keep chunked state as it reads and decodes input from client from con->read_queue to con->request_content_queue.
Updated by gstrauss over 8 years ago
- Status changed from Reopened to Patch Pending
- Target version set to 1.4.44
flynn wrote:
I would offer extended testing, if a patch is provided.
flynn, please test Transfer-Encoding: chunked support in using personal/gstrauss/master branch in lighttpd git repo:
(DevelGit)
$ git clone https://git.lighttpd.net/lighttpd/lighttpd1.4.git
Updated by gstrauss over 8 years ago
- Status changed from Patch Pending to Fixed
- % Done changed from 0 to 100
Applied in changeset 4d7f5737f150c4bab8d3832bab9da62eddfea7d2.
Updated by gstrauss about 8 years ago
FYI: a patch was provided for this feature a month ago, but no feedback was received. I added tests to the test suite and released the feature (Transfer-Encoding: chunked for HTTP request body, e.g. HTTP PUT)
Separately, a regression occurred in mod_webdav in lighttpd 1.4.43 which led to error trace in the log, and a different change turned this into a crashing bug in lighttpd 1.4.44. Now then, the bug is my fault, not yours, and I'll be adding tests to the test suite to attempt to avoid such mod_webdav issues in the future. Still, I am disappointed that I received no such feedback from you for either lighttpd 1.4.43 or lighttpd 1.4.44 after you seemed so eager for the new feature.
x-ref:
Lighttpd 1.4.43/44 dies - missing cleanup in webdav
https://redmine.lighttpd.net/boards/2/topics/7034
Updated by flynn over 7 years ago
Sorry for not testing your patch, but OSX in newer versions does not use chunked PUT-request anymore.
I just tested some uploads, all PUT-requests are now with Content-Length. Tested with
server.stream-response-body = 1 server.stream-request-body = 0 server.reject-expect-100-with-417 = "disable"
I do not see any 411 Length Required response.
Updated by flynn about 7 years ago
server.stream-request-body = 0
is necessary for chunked PUT requests!
Verified now with OSX finder and
debug.log-request-header = "enable".
Works perfect now.
Updated by gstrauss about 7 years ago
That should not be necessary after the patch in #2854 is released in lighttpd 1.4.49.
[Edit]: Correction: server.stream-request-body = 0 is needed with requests that send "Transfer-Encoding: chunked" since lighttpd will return "411 Length Required" before passing to backends, as the CGI spec requires setting CONTENT_LENGTH. mod_proxy is not a full HTTP/1.1 reverse proxy, and also returns 411 Length Required.
Applications like curl correctly handle 411 Length Required, and resend the request with Content-Length, e.g.curl -T sample.txt -H "Transfer-Encoding: chunked" http://localhost:8080/cgi-bin/upload.pl
Updated by gstrauss about 7 years ago
If you want server.stream-request-body enabled when "Content-Length" is set, but want to workaround when "Transfer-Encoding: chunked" is set, then use this in lighttpd.conf:
$REQUEST_HEADER["Content-Length"] == "" { server.stream-request-body = 0 }
Added by gstrauss over 8 years ago
Also available in: Atom
[core] support Transfer-Encoding: chunked req body (fixes #2156)
support Transfer-Encoding: chunked request body in conjunction with
server.stream-request-body = 0
dynamic handlers will still return 411 Length Required if
server.stream-request-body = 1 or 2 (!= 0)
since CGI-like env requires CONTENT_LENGTH be set
(and mod_proxy currently sends HTTP/1.0 requests to backends,
and Content-Length recommended for robust interaction with backend)
x-ref:
"request: support Chunked Transfer Coding for HTTP PUT"
https://redmine.lighttpd.net/issues/2156