Bug #2384
closedX-sendfile2 ignores range parameter
Description
When using X-Sendfile in fcgi with php, lighttpd ignores the range parameter and instead sends the whole file.
The client runs: wget -c http://server/get/i7b35952163f845f4
Which in turn sends the request:
GET /get/i7b35952163f845f4 HTTP/1.1
Range: bytes=4428831-
User-Agent: Wget/1.13.4 (linux-gnu)
Accept: */*
Host: server.com
Connection: Keep-Alive
The php script outputs the header among others:
X-Sendfile2: %2Fvar%2Fwww%2Fuploads%2Fi7b35952163f845f4%2Fasterisk.pdf 4428831-
lighttpd replies with a 200 status and the full file instead of a 206 status and a part of the file:
HTTP/1.1 200 OK
Content-Description: File Transfer
Content-Type: application/octet-stream
Content-Disposition: attachment; filename="asterisk.pdf"
Content-Transfer-Encoding: binary
Expires: 0
Cache-Control: must-revalidate, post-check=0, pre-check=0
Pragma: public
Content-Length: 11195330
Date: Tue, 24 Jan 2012 05:42:23 GMT
Relevant strace is attached.
Tested with lighttpd 1.4.29 and 1.4.30.
Files
Updated by jpc almost 12 years ago
I came across the same issue today. I have not yet got a chance to investigate the issue. I will probably do so in the next few weeks. lameventanas did you look into the problem?
Updated by lameventanas almost 12 years ago
jpc wrote:
I came across the same issue today. I have not yet got a chance to investigate the issue. I will probably do so in the next few weeks. lameventanas did you look into the problem?
I didn't look into this, instead I implemented it in PHP.
But please feel free to investigate, fix and post a patch here :)
Updated by stbuehler over 11 years ago
- Status changed from New to Invalid
It is your job to generate the appropriate headers; this ensures the solution can be used in other places too (for example pseudo seeking which uses the query string instead of the range http header)
Updated by lameventanas over 11 years ago
stbuehler wrote:
It is your job to generate the appropriate headers; this ensures the solution can be used in other places too (for example pseudo seeking which uses the query string instead of the range http header)
This is the header I am generating:
X-Sendfile2: %2Fvar%2Fwww%2Fuploads%2Fi7b35952163f845f4%2Fasterisk.pdf 4428831-
Why is this incorrect?
Updated by jpc over 11 years ago
- Status changed from Invalid to Reopened
stbuehler wrote:
It is your job to generate the appropriate headers; this ensures the solution can be used in other places too (for example pseudo seeking which uses the query string instead of the range http header)
Yes if you could clarify what headers are expected, it'll be great. The only documentation that I have found so far is on the wiki. I also followed the changeset r2651 (from ticket #2008). The changeset includes a test case which shows how to use the header (format is the same as on the wiki). Basically, the header shown is: header('X-Sendfile2: /path/to/file 50000-');
Now, after looking at bit about this today, I get the same results as lameventanas mentioned in the ticket description:
- I run wget with -c parameter to resume a download
- Observed behavior: lighttpd respond with a 200 OK status code and returns the entire file back
- Expected behavior: lighttpd should response with a 206 status code and include a "Content-Range" header in the response (which should only include part of the file)
- RFC2616 section 14.35.2 which specifies how an HTTP client can specify the range to be downloaded
- RFC2616 section 10.2.7 which specifies that 206 status code should be returned when a range query is received from the client. It also indicates that the response should include a "Content-Range" header.
- RFC2616 section 14.16 which specifies the "Content-Range" header definition.
The last link give an example of the expected response
HTTP/1.1 206 Partial content
Date: Wed, 15 Nov 1995 06:25:24 GMT
Last-Modified: Wed, 15 Nov 1995 04:58:08 GMT
Content-Range: bytes 21010-47021/47022
Content-Length: 26012
Content-Type: image/gif
I started to look to the source code of mod_fastcgi.c
but I am not too familiar with it :(
Updated by stbuehler over 11 years ago
- Status changed from Reopened to Invalid
Where did we ever claim that X-Sendfile2 implements the HTTP range spec? It is named "X-Sendfile2", not "Range" or "Content-Range".
If you need the Content-Range header, just send it. If you want another HTTP status code, just set the status from the backend.
X-Sendfile does just what the name says: it sends the file.
Updated by jpc over 11 years ago
stbuehler wrote:
Where did we ever claim that X-Sendfile2 implements the HTTP range spec? It is named "X-Sendfile2", not "Range" or "Content-Range".
The lighttpd documentation is not extensive on the topic and while it never claims to implement the HTTP spec for resumable download, it never claims the opposite! Hence, my assumption that I could leverage X-Sendfile2 to support the wget -c
command.
Now, thanks to your last comment, I was able to make this work (sort of). In case someone comes to this issue: a possible workaround is to ignore the HTTP spec and use a GET parameter instead.
Take the following PHP snippet:
$file = '/tmp/big-file.bin';
if (isset($_GET['restart-at']) && (int)$_GET['restart-at']>0) {
header('X-Sendfile2: '.$file.' '.$_GET['restart-at'].'-');
} else {
header('X-Sendfile: '.$file);
}
If you run the command wget http://localhost/test1.php?restart-at=500000
it will start the download at byte 500000.
Now, the problem with not following the HTTP spec, means that doing a wget -c
will not work properly. I used the following PHP script:
$file = '/tmp/big-file.bin';
if (isset($_SERVER['HTTP_RANGE']) && preg_match('/\Abytes=[0-9]+-[0-9]*\z/', $_SERVER['HTTP_RANGE'])) {
header('X-Sendfile2: '.$file.' '.substr($_SERVER['HTTP_RANGE'],6));
} else {
header('X-Sendfile: '.$file);
}
wget -c http://localhost/test2.php
, stop it and restart it:
- I can see that only the second part of the file is downloaded the 2nd time I run wget (which is what I wanted)
- But at the end, it's not properly attached to the first part of the file (which was retrieved by the first wget)
Updated by icy over 11 years ago
Try sending a 206 status code and the Content-Range header from your backend.
Updated by lameventanas over 11 years ago
Ok thanks to the clarification I was able to make my code work.
I assumed that lighttpd took care of the status and content-range headers because so many things could go wrong with the sendfile() call (ie: file doesn't exist, permission denied, range not satisfiable, etc).
What happens in these cases? Does lighttpd overwrite the headers in these cases? Or is there any way for the backend to know about these error conditions?
Lighttpd definitely needs better documentation.
Updated by jpc over 11 years ago
icy wrote:
Try sending a 206 status code and the Content-Range header from your backend.
Thank you for the pointer!
lameventanas wrote:
Ok thanks to the clarification I was able to make my code work.
Me too. I generate the following headers and it works when I run wget -c
:
header('Content-Range: bytes 2375680-12103815/12103816');
header('HTTP/1.1 206 Partial content');
header('X-Sendfile2: /tmp/big-file.bin 2375680-');
Lighttpd definitely needs better documentation.
I added an example on the wiki. Hopefully, it'll help the next person having the problem :)
Also available in: Atom