request: support Chunked Transfer Coding for HTTP PUT
The HTTP spec section 3.6.1 describes Chunked Transfer Coding :
which neither lighttpd 1.4 nor 1.5 supports. It would be very useful for cases where clients wish to upload a file without first specifying the Content-length header, perhaps because they are uploading it while it is being generated.
Curl supports it on the client side (handy for testing):
"If you use PUT to a HTTP 1.1 server, you can upload data without knowing
the size before starting the transfer if you use chunked encoding. You
enable this by adding a header like "Transfer-Encoding: chunked" with
With HTTP 1.0 or without chunked transfer, you must specify the size."
Would others find this useful ?
[core] support Transfer-Encoding: chunked req body (fixes #2156)
support Transfer-Encoding: chunked request body in conjunction with
server.stream-request-body = 0
dynamic handlers will still return 411 Length Required if
server.stream-request-body = 1 or 2 (!= 0)
since CGI-like env requires CONTENT_LENGTH be set
(and mod_proxy currently sends HTTP/1.0 requests to backends,
and Content-Length recommended for robust interaction with backend)
"request: support Chunked Transfer Coding for HTTP PUT"
#3 Updated by andreas_stoe over 7 years ago
the attached "hack" adds support for basic "transfer-encoding: chunked" request handling. Chunked data which is sent to the server will be decoded using a small library (chunk_decoder.c, chunk_decoder.h) and is finally stored in tempfiles. When all chunks have been read, con->request.content_length is set accordingly. So all further processing steps in the webserver will believe data has been sent with a "content-length" header.
I hope this patch might be useful,
#4 Updated by andreas_stoe over 7 years ago
My previous patch contained a serious bug which might have caught the web-server to stop responding if a connection got closed abruptly. I also added support for hex numbers written with small letters.
#5 Updated by pprkut over 7 years ago
I tried your patch, but I couldn't get lighttpd to compile with it. Here's the error I get:
libtool: link: gcc -g -O2 -Wall -W -Wshadow -pedantic -std=gnu99 -o lighttpd server.o response.o connections.o network.o configfile.o configparser.o request.o proc_open.o buffer.o log.o keyvalue.o chunk.o http_chunk.o stream.o fdevent.o stat_cache.o plugin.o joblist.o etag.o array.o data_string.o data_count.o data_array.o data_integer.o md5.o data_fastcgi.o fdevent_select.o fdevent_linux_rtsig.o fdevent_poll.o fdevent_linux_sysepoll.o fdevent_solaris_devpoll.o fdevent_freebsd_kqueue.o data_config.o bitset.o inet_ntop_cache.o crc32.o connections-glue.o configfile-glue.o http-header-glue.o network_write.o network_linux_sendfile.o network_freebsd_sendfile.o network_writev.o network_solaris_sendfilev.o network_openssl.o splaytree.o status_counter.o -Wl,--export-dynamic -L/usr/lib64 /usr/lib64/libpcre.so -ldl /usr/lib64/libfam.so -Wl,-rpath -Wl,/usr/lib64 -Wl,-rpath -Wl,/usr/lib64
connections.o: In function `connection_handle_read_state':
/mnt/progs/slack/test/lighttpd-1.4.26/src/connections.c:1200: undefined reference to `chunk_decoder_work'
/mnt/progs/slack/test/lighttpd-1.4.26/src/connections.c:1336: undefined reference to `chunk_decoder_free'
/mnt/progs/slack/test/lighttpd-1.4.26/src/connections.c:1283: undefined reference to `chunk_decoder_free'
/mnt/progs/slack/test/lighttpd-1.4.26/src/connections.c:1186: undefined reference to `chunk_decoder_create'
connections.o: In function `connection_reset':
/mnt/progs/slack/test/lighttpd-1.4.26/src/connections.c:806: undefined reference to `chunk_decoder_free'
connections.o: In function `connection_close':
/mnt/progs/slack/test/lighttpd-1.4.26/src/connections.c:127: undefined reference to `chunk_decoder_free'
collect2: ld returned 1 exit status
make3: *** [lighttpd] Error 1
#7 Updated by gstrauss over 1 year ago
FYI: recent patch to lighttpd will communicate to client 411 Length Required, so while Transfer-Encoding: chunked is still not supported, the server and client should behave better when the server sends 411 response.
commit 06d3c754403b2df99775641e05999ea9b2081618 Author: Glenn Strauss <firstname.lastname@example.org> Date: Sat Mar 26 12:58:33 2016 +0000 [core] respond 411 Length Required if request has Transfer-Encoding: chunked (fixes #631) lighttpd does not currently support request body transfer-codings From: Glenn Strauss <email@example.com> git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@3128 152afb58-edef-0310-8abb-c4023f1b3aa9
#8 Updated by gstrauss over 1 year ago
- Status changed from New to Need Feedback
https://github.com/lighttpd/lighttpd1.4/pull/53 moves request body reading to connection_handle_read_post_state(). This location might be enhanced to support chunking (and then request.c modified to not reject chunked POST requests with 411 Length Required)
However, this is low priority and might not be worth the effort if clients are responding properly to 411 Length Required.
Is anyone having issues with clients that do not properly handle 411 Length Required?
#9 Updated by gstrauss over 1 year ago
- Status changed from Need Feedback to Missing Feedback
- Priority changed from Normal to Low
While supporting Transfer-Encoding: chunked for request body would be a nice feature to have, to increase the priority of this feature request, someone would need to present a (non-contrived) use case where this was really needed.
Well-behaved clients should handle PUTs properly now that lighttpd responds with 411 Length Required (which is compliant with the RFCs) if client sends Transfer-Encoding: chunked.
- Status changed from Missing Feedback to Reopened
Thanks for noting your interest and the offer to help test.
Modifying connection_handle_read_post_state() to handle chunked encoding is the central place to implement it. However, con->request.content_length is used in more than a few places to flag the presence of content or not. Those locations would need to be updated to handle delayed content, including sometimes handling delayed content of 0 octets.
While none of this is very difficult, it will take some effort. I'll re-open this feature request, but the priority is still "low" until someone finds some time to do it.
Uploading patch to be applied against lighttpd git master HEAD.
This patch changes existing modules to be aware that Transfer-Encoding: chunked may be used, resulting in an unknown Content-Length of request body, where previously the length of request body was known up-front. For most modules, this means returning 411 Length Requiredt to preserve existing behavior.
For modules handling requests that are not configured to stream the request body to the backend, which collect all content prior to starting the backend, it is now possible (but not yet coded) for request body to be collected from client and de-chunked.
Before this patch is committed, it needs some wider testing to ensure that existing behavior has not been broken. Specific behavior that needs testing is POST of request body to various dynamic handlers (mod_cgi, mod_fastcgi, mod_scgi, mod_proxy, mod_webdav).
Future work is largely isolated in connections-glue.c:connection_handle_read_post_state(), which needs to keep chunked state as it reads and decodes input from client from con->read_queue to con->request_content_queue.
- Status changed from Reopened to Patch Pending
- Target version set to 1.4.44
FYI: a patch was provided for this feature a month ago, but no feedback was received. I added tests to the test suite and released the feature (Transfer-Encoding: chunked for HTTP request body, e.g. HTTP PUT)
Separately, a regression occurred in mod_webdav in lighttpd 1.4.43 which led to error trace in the log, and a different change turned this into a crashing bug in lighttpd 1.4.44. Now then, the bug is my fault, not yours, and I'll be adding tests to the test suite to attempt to avoid such mod_webdav issues in the future. Still, I am disappointed that I received no such feedback from you for either lighttpd 1.4.43 or lighttpd 1.4.44 after you seemed so eager for the new feature.
Lighttpd 1.4.43/44 dies - missing cleanup in webdav
Sorry for not testing your patch, but OSX in newer versions does not use chunked PUT-request anymore.
I just tested some uploads, all PUT-requests are now with Content-Length. Tested with
server.stream-response-body = 1 server.stream-request-body = 0 server.reject-expect-100-with-417 = "disable"
I do not see any 411 Length Required response.
Also available in: Atom