Project

General

Profile

Feature #2156

request: support Chunked Transfer Coding for HTTP PUT

Added by morgan almost 8 years ago. Updated 3 months ago.

Status:
Fixed
Priority:
Low
Assignee:
-
Category:
mod_webdav
Target version:
Start date:
2010-01-26
Due date:
% Done:

100%

Estimated time:
Missing in 1.5.x:
No

Description

The HTTP spec section 3.6.1 describes Chunked Transfer Coding :

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

which neither lighttpd 1.4 nor 1.5 supports. It would be very useful for cases where clients wish to upload a file without first specifying the Content-length header, perhaps because they are uploading it while it is being generated.

Curl supports it on the client side (handy for testing):

http://curl.haxx.se/libcurl/c/curl_easy_setopt.html#CURLOPTUPLOAD

"If you use PUT to a HTTP 1.1 server, you can upload data without knowing
the size before starting the transfer if you use chunked encoding. You
enable this by adding a header like "Transfer-Encoding: chunked" with
CURLOPT_HTTPHEADER
With HTTP 1.0 or without chunked transfer, you must specify the size."

Would others find this useful ?

Morgan

Associated revisions

Revision 4d7f5737 (diff)
Added by gstrauss 12 months ago

[core] support Transfer-Encoding: chunked req body (fixes #2156)

support Transfer-Encoding: chunked request body in conjunction with
server.stream-request-body = 0

dynamic handlers will still return 411 Length Required if
server.stream-request-body = 1 or 2 (!= 0)
since CGI-like env requires CONTENT_LENGTH be set
(and mod_proxy currently sends HTTP/1.0 requests to backends,
and Content-Length recommended for robust interaction with backend)

x-ref:
"request: support Chunked Transfer Coding for HTTP PUT"
https://redmine.lighttpd.net/issues/2156

History

#1

Updated by morgan almost 8 years ago

  • Target version changed from 1.4.x to 1.4.26

Looks like the 1.4.26 is the place to be, so moving request to 1.4.26 queue.

#2

Updated by stbuehler almost 8 years ago

  • Target version changed from 1.4.26 to 1.4.x

Sorry, no. Of course this would be useful, but i don't see it coming in 1.x; the code assumes in too many places to know the content-length of the upload after parsing the request headers.

#3

Updated by andreas_stoe over 7 years ago

Hi,

the attached "hack" adds support for basic "transfer-encoding: chunked" request handling. Chunked data which is sent to the server will be decoded using a small library (chunk_decoder.c, chunk_decoder.h) and is finally stored in tempfiles. When all chunks have been read, con->request.content_length is set accordingly. So all further processing steps in the webserver will believe data has been sent with a "content-length" header.

I hope this patch might be useful,
Andreas

#4

Updated by andreas_stoe over 7 years ago

My previous patch contained a serious bug which might have caught the web-server to stop responding if a connection got closed abruptly. I also added support for hex numbers written with small letters.

#5

Updated by pprkut over 7 years ago

I tried your patch, but I couldn't get lighttpd to compile with it. Here's the error I get:

libtool: link: gcc -g -O2 -Wall -W -Wshadow -pedantic -std=gnu99 -o lighttpd server.o response.o connections.o network.o configfile.o configparser.o request.o proc_open.o buffer.o log.o keyvalue.o chunk.o http_chunk.o stream.o fdevent.o stat_cache.o plugin.o joblist.o etag.o array.o data_string.o data_count.o data_array.o data_integer.o md5.o data_fastcgi.o fdevent_select.o fdevent_linux_rtsig.o fdevent_poll.o fdevent_linux_sysepoll.o fdevent_solaris_devpoll.o fdevent_freebsd_kqueue.o data_config.o bitset.o inet_ntop_cache.o crc32.o connections-glue.o configfile-glue.o http-header-glue.o network_write.o network_linux_sendfile.o network_freebsd_sendfile.o network_writev.o network_solaris_sendfilev.o network_openssl.o splaytree.o status_counter.o -Wl,--export-dynamic -L/usr/lib64 /usr/lib64/libpcre.so -ldl /usr/lib64/libfam.so -Wl,-rpath -Wl,/usr/lib64 -Wl,-rpath -Wl,/usr/lib64
connections.o: In function `connection_handle_read_state':
/mnt/progs/slack/test/lighttpd-1.4.26/src/connections.c:1200: undefined reference to `chunk_decoder_work'
/mnt/progs/slack/test/lighttpd-1.4.26/src/connections.c:1336: undefined reference to `chunk_decoder_free'
/mnt/progs/slack/test/lighttpd-1.4.26/src/connections.c:1283: undefined reference to `chunk_decoder_free'
/mnt/progs/slack/test/lighttpd-1.4.26/src/connections.c:1186: undefined reference to `chunk_decoder_create'
connections.o: In function `connection_reset':
/mnt/progs/slack/test/lighttpd-1.4.26/src/connections.c:806: undefined reference to `chunk_decoder_free'
connections.o: In function `connection_close':
/mnt/progs/slack/test/lighttpd-1.4.26/src/connections.c:127: undefined reference to `chunk_decoder_free'
collect2: ld returned 1 exit status
make3: *** [lighttpd] Error 1

#6

Updated by lvs over 5 years ago

I attach the current version of the patch. It applies to 1.4.31, please don't mind the name.

#7

Updated by gstrauss over 1 year ago

FYI: recent patch to lighttpd will communicate to client 411 Length Required, so while Transfer-Encoding: chunked is still not supported, the server and client should behave better when the server sends 411 response.

commit 06d3c754403b2df99775641e05999ea9b2081618
Author: Glenn Strauss <gstrauss@gluelogic.com>
Date:   Sat Mar 26 12:58:33 2016 +0000

    [core] respond 411 Length Required if request has Transfer-Encoding: chunked (fixes #631)

    lighttpd does not currently support request body transfer-codings

    From: Glenn Strauss <gstrauss@gluelogic.com>

    git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@3128 152afb58-edef-0310-8abb-c4023f1b3aa9

#8

Updated by gstrauss over 1 year ago

  • Status changed from New to Need Feedback

https://github.com/lighttpd/lighttpd1.4/pull/53 moves request body reading to connection_handle_read_post_state(). This location might be enhanced to support chunking (and then request.c modified to not reject chunked POST requests with 411 Length Required)

However, this is low priority and might not be worth the effort if clients are responding properly to 411 Length Required.

Is anyone having issues with clients that do not properly handle 411 Length Required?

#9

Updated by gstrauss over 1 year ago

  • Status changed from Need Feedback to Missing Feedback
  • Priority changed from Normal to Low

While supporting Transfer-Encoding: chunked for request body would be a nice feature to have, to increase the priority of this feature request, someone would need to present a (non-contrived) use case where this was really needed.

Well-behaved clients should handle PUTs properly now that lighttpd responds with 411 Length Required (which is compliant with the RFCs) if client sends Transfer-Encoding: chunked.

#10

Updated by stbuehler over 1 year ago

  • Target version deleted (1.4.x)
#11

Updated by flynn about 1 year ago

Any progress on this issue?

The relevant usecase is:
  • webdav with lighttpd/php/sabredav
  • OSX finder as client

This combinations results in PUT request with chunked encoding: http://sabre.io/dav/clients/finder/

I would offer extended testing, if a patch is provided.

#12

Updated by gstrauss about 1 year ago

  • Status changed from Missing Feedback to Reopened

Thanks for noting your interest and the offer to help test.

Modifying connection_handle_read_post_state() to handle chunked encoding is the central place to implement it. However, con->request.content_length is used in more than a few places to flag the presence of content or not. Those locations would need to be updated to handle delayed content, including sometimes handling delayed content of 0 octets.

While none of this is very difficult, it will take some effort. I'll re-open this feature request, but the priority is still "low" until someone finds some time to do it.

#13

Updated by gstrauss 12 months ago

Uploading patch to be applied against lighttpd git master HEAD.

This patch changes existing modules to be aware that Transfer-Encoding: chunked may be used, resulting in an unknown Content-Length of request body, where previously the length of request body was known up-front. For most modules, this means returning 411 Length Requiredt to preserve existing behavior.

For modules handling requests that are not configured to stream the request body to the backend, which collect all content prior to starting the backend, it is now possible (but not yet coded) for request body to be collected from client and de-chunked.

Before this patch is committed, it needs some wider testing to ensure that existing behavior has not been broken. Specific behavior that needs testing is POST of request body to various dynamic handlers (mod_cgi, mod_fastcgi, mod_scgi, mod_proxy, mod_webdav).

Future work is largely isolated in connections-glue.c:connection_handle_read_post_state(), which needs to keep chunked state as it reads and decodes input from client from con->read_queue to con->request_content_queue.

#14

Updated by gstrauss 12 months ago

  • Status changed from Reopened to Patch Pending
  • Target version set to 1.4.44

flynn wrote:

I would offer extended testing, if a patch is provided.

flynn, please test Transfer-Encoding: chunked support in using personal/gstrauss/master branch in lighttpd git repo:
(DevelGit)
$ git clone https://git.lighttpd.net/lighttpd/lighttpd1.4.git

#15

Updated by gstrauss 12 months ago

  • Status changed from Patch Pending to Fixed
  • % Done changed from 0 to 100
#16

Updated by gstrauss 11 months ago

FYI: a patch was provided for this feature a month ago, but no feedback was received. I added tests to the test suite and released the feature (Transfer-Encoding: chunked for HTTP request body, e.g. HTTP PUT)

Separately, a regression occurred in mod_webdav in lighttpd 1.4.43 which led to error trace in the log, and a different change turned this into a crashing bug in lighttpd 1.4.44. Now then, the bug is my fault, not yours, and I'll be adding tests to the test suite to attempt to avoid such mod_webdav issues in the future. Still, I am disappointed that I received no such feedback from you for either lighttpd 1.4.43 or lighttpd 1.4.44 after you seemed so eager for the new feature.

x-ref:
Lighttpd 1.4.43/44 dies - missing cleanup in webdav
https://redmine.lighttpd.net/boards/2/topics/7034

#17

Updated by flynn 3 months ago

Sorry for not testing your patch, but OSX in newer versions does not use chunked PUT-request anymore.

I just tested some uploads, all PUT-requests are now with Content-Length. Tested with

server.stream-response-body = 1
server.stream-request-body = 0
server.reject-expect-100-with-417 = "disable" 

I do not see any 411 Length Required response.

Also available in: Atom