Lighttpd

ideally we would implement range request support for x-sendfile at the same time.

and how about:
x-sendfile: path [range]

so range becomes an optional argument for x-sendfile

darix: it's impossible to add a range parameter to X-sendfile without breaking backwardscompatibility (quoting issues).

shellsage wrote:

I've implemented the X-Sendfile-Extended header for mod_fastcgi.

Header field definition:
[...]
Where x is the zero-based first byte to serve of the file, y is the last byte to serve of the file + 1, and /path/to/your/file is obviously the path of the file to serve. x and y may have a value of -1, although when x -1 an error will be noted in the log. When y -1, y is automatically set to the size of the file.

Hi,

why this special syntax?

I mean there is already a "Range" syntax in http:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35

ralf wrote:

why this special syntax?

I mean there is already a "Range" syntax in http:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35

Yeah, I agree with ralf here. Shouldn't X-Sendfile "just" be patched to take Range into account (if set at all).

No, we cannot use X-Sendfile without breaking old scripts.

The difference, I guess, is that this is used to send slices of multiple files. I do think it's cleaner to extend X-Sendfile with X-Sendfile-Range rather than redefining it completely. Define X-Sendfile-Range as affecting the following X-Sendfile only, and permit multiple X-Sendfile headers.

In any case, this should probably mimic the byte-range-resp-spec syntax (defined for Content-Range). For example,

X-Sendfile-Extended: 500-599 /tmp/data.txt # [500,599] inclusive; 500 bytes
X-Sendfile-Extended: 57- /tmp/data2.txt
X-Sendfile-Extended: 0-998 /tmp/data3.txt

This isn't the same as byte-range-resp-spec, which doesn't allow omitting last-byte-pos, but it's closer.

I'd also recommend mimicing 14.16 Content-Range's error handling behavior: if the range is invalid (end > start), ignore the header entirely; don't try to guess what was intended. Don't be lenient when parsing a new header; you don't have existing code to accomodate, so being lenient will just encourage new code to be sloppy.

• I think the new header should be named "X-Sendfile-Ranged" (not "-range", as it includes range + filename)
• the rfc says that you should be able to combine multiple header values with "," - so i guess we should encode the filename somehow to allow this
  urlencode would be nice, but i doesn't encode , or " by default iirc
• i just had a look again at the http range syntax: i think it is too complicated and not really needed as we will support multiple headers
• i think we only have to support the following types of ranges - these should be pretty obvious to read; i don't think we should support "-end" or "-lastbytes"
  • complete file (i guess "*" should be ok)
  • a normal range: "start-end"
  • a start offset: "start-"

peto wrote:

In any case, this should probably mimic the byte-range-resp-spec syntax (defined for Content-Range). For example,

X-Sendfile-Extended: 500-599 /tmp/data.txt # [500,599] inclusive; 500 bytes
X-Sendfile-Extended: 57- /tmp/data2.txt
X-Sendfile-Extended: 0-998 /tmp/data3.txt

I agree, this seems like a good way to do the range.

@stbuehler, I think we should maybe just name the header X-Sendfile-Range, or X-Send-File-Range, because the header will be used to send a "range" from a "file."

To send the complete file, we could just mandate the use of "0-"

Is it not enough to just escape commas in filenames provided to the header? e.g. X-Sendfile-Range: 100-200 /tmp/myfile\,withcomma.txt, 44-555 /tmp/otherfile\,test.txt

Ah yes, "*" is not needed. escaping commas is not enough, as filenames may even contain "\r\n".
But we could use urlencode if we use "X-Sendfile-Ranged: filename+with%20spaces 0-,another-file 500-"

You've arrived back at exactly the range syntax I suggested...

I'd recommend sticking with URL escapes with the added note that commas must be escaped. (Some URL encoders do escape commas; Python's urllib does.) It fits cleanly with generic header parsing, uses standard urldecode methods with no changes, and the only extra work that might needed for clients is to say urlencode(s).replace(",", "%2C") instead of just urlencode(s).

Is ignoring the header entirely in case of a bad range really a good idea?
What if multiple headers are specified and one has a bad range. Should we ignore just that one or all X-Sendfile-Range headers?
Probably all because headers could be merged...
But then there wont be any content in the response and still a 200 OK status.
What if we would throw a generic 500 error? Or a 416 "Requested range not satisfiable" or maybe even 502 "Bad Gateway"?

Also I'd like everyone to give a thought on how to handle normal response content in conjunktion with this header. Throw away, prepend or append?

I think will be very nice to include support for X-LIGHTTPD-send-tempfile

X-LIGHTTPD-send-tempfile-extednded /dev/shm/pre-data.txt
    X-Sendfile-Extended: 5 5000 /tmp/data.txt 
    X-LIGHTTPD-send-tempfile-extednded /dev/shm/post-data.txt

normal response content in conjunktion with this header

Do whatever X-Sendfile does, so the headers can be used interchangably without causing obscure differences.

Let's call this X-Sendfile2. Don't call it X-Sendfile-Range; assume that this header will do more things than specifying ranges. Don't call it X-Sendfile-Extended, so if someone really needs to start from scratch with X-Sendfile again, they can call it X-Sendfile3.

Then, let's define the header as having space-separated parameters, like so:

X-Sendfile2: url-escaped-filename range [param1 [param2...]]

and saying that any unknown parameters will cause an error. This lets us extend the header cleanly later on; for example, by saying "if any parameter is the string delete-after-sending, the file will be deleted when the request terminates".

And let's not actually define more features for this header here. I can think of some trickiness related to the notion of "deleting a file after it's finished sending" that should probably be discussed, if that's what tx meant--let's do that another time, after the basic range stuff is done.

I don't think an "uber" powerful specification for the header is a good idea, if we need to combine more than ranges+tmp-file we should do it with JSON and a new header (I don't see this coming in 1.x)

1.5 already supports send-tempfile, so it should be no problem to support x-send-tempfile-range; I propose using two different header names. The downside with 2 header names is, that the http rfc doesn't care about the order of different headers, so you have to rely on implementation details to be able to mix normal sendfile-range with send-tempfile-range.