Project

General

Profile

Bug #2384

X-sendfile2 ignores range parameter

Added by lameventanas over 5 years ago. Updated about 5 years ago.

Status:
Invalid
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Start date:
2012-01-24
Due date:
% Done:

0%

Missing in 1.5.x:
No

Description

When using X-Sendfile in fcgi with php, lighttpd ignores the range parameter and instead sends the whole file.

The client runs: wget -c http://server/get/i7b35952163f845f4

Which in turn sends the request:
GET /get/i7b35952163f845f4 HTTP/1.1
Range: bytes=4428831-
User-Agent: Wget/1.13.4 (linux-gnu)
Accept: */*
Host: server.com
Connection: Keep-Alive

The php script outputs the header among others:
X-Sendfile2: %2Fvar%2Fwww%2Fuploads%2Fi7b35952163f845f4%2Fasterisk.pdf 4428831-

lighttpd replies with a 200 status and the full file instead of a 206 status and a part of the file:

HTTP/1.1 200 OK
Content-Description: File Transfer
Content-Type: application/octet-stream
Content-Disposition: attachment; filename="asterisk.pdf"
Content-Transfer-Encoding: binary
Expires: 0
Cache-Control: must-revalidate, post-check=0, pre-check=0
Pragma: public
Content-Length: 11195330
Date: Tue, 24 Jan 2012 05:42:23 GMT

Relevant strace is attached.

Tested with lighttpd 1.4.29 and 1.4.30.

sendfile-trace.txt View - strace of x-sendfile2 (6.96 KB) lameventanas, 2012-01-24 06:57

History

#1 Updated by jpc about 5 years ago

I came across the same issue today. I have not yet got a chance to investigate the issue. I will probably do so in the next few weeks. lameventanas did you look into the problem?

#2 Updated by lameventanas about 5 years ago

jpc wrote:

I came across the same issue today. I have not yet got a chance to investigate the issue. I will probably do so in the next few weeks. lameventanas did you look into the problem?

I didn't look into this, instead I implemented it in PHP.
But please feel free to investigate, fix and post a patch here :)

#3 Updated by stbuehler about 5 years ago

  • Status changed from New to Invalid

It is your job to generate the appropriate headers; this ensures the solution can be used in other places too (for example pseudo seeking which uses the query string instead of the range http header)

#4 Updated by lameventanas about 5 years ago

stbuehler wrote:

It is your job to generate the appropriate headers; this ensures the solution can be used in other places too (for example pseudo seeking which uses the query string instead of the range http header)

This is the header I am generating:
X-Sendfile2: %2Fvar%2Fwww%2Fuploads%2Fi7b35952163f845f4%2Fasterisk.pdf 4428831-

Why is this incorrect?

#5 Updated by jpc about 5 years ago

  • Status changed from Invalid to Reopened

stbuehler wrote:

It is your job to generate the appropriate headers; this ensures the solution can be used in other places too (for example pseudo seeking which uses the query string instead of the range http header)

Yes if you could clarify what headers are expected, it'll be great. The only documentation that I have found so far is on the wiki. I also followed the changeset r2651 (from ticket #2008). The changeset includes a test case which shows how to use the header (format is the same as on the wiki). Basically, the header shown is: header('X-Sendfile2: /path/to/file 50000-');

Now, after looking at bit about this today, I get the same results as lameventanas mentioned in the ticket description:

  • I run wget with -c parameter to resume a download
  • Observed behavior: lighttpd respond with a 200 OK status code and returns the entire file back
  • Expected behavior: lighttpd should response with a 206 status code and include a "Content-Range" header in the response (which should only include part of the file)
The expected behavior mentioned above is my understanding of the HTTP spec. See the following links:
  • RFC2616 section 14.35.2 which specifies how an HTTP client can specify the range to be downloaded
  • RFC2616 section 10.2.7 which specifies that 206 status code should be returned when a range query is received from the client. It also indicates that the response should include a "Content-Range" header.
  • RFC2616 section 14.16 which specifies the "Content-Range" header definition.

The last link give an example of the expected response

HTTP/1.1 206 Partial content
Date: Wed, 15 Nov 1995 06:25:24 GMT
Last-Modified: Wed, 15 Nov 1995 04:58:08 GMT
Content-Range: bytes 21010-47021/47022
Content-Length: 26012
Content-Type: image/gif

I started to look to the source code of mod_fastcgi.c but I am not too familiar with it :(

#6 Updated by stbuehler about 5 years ago

  • Status changed from Reopened to Invalid

Where did we ever claim that X-Sendfile2 implements the HTTP range spec? It is named "X-Sendfile2", not "Range" or "Content-Range".

If you need the Content-Range header, just send it. If you want another HTTP status code, just set the status from the backend.

X-Sendfile does just what the name says: it sends the file.

#7 Updated by jpc about 5 years ago

stbuehler wrote:

Where did we ever claim that X-Sendfile2 implements the HTTP range spec? It is named "X-Sendfile2", not "Range" or "Content-Range".

The lighttpd documentation is not extensive on the topic and while it never claims to implement the HTTP spec for resumable download, it never claims the opposite! Hence, my assumption that I could leverage X-Sendfile2 to support the wget -c command.

Now, thanks to your last comment, I was able to make this work (sort of). In case someone comes to this issue: a possible workaround is to ignore the HTTP spec and use a GET parameter instead.

Take the following PHP snippet:

$file = '/tmp/big-file.bin';
if (isset($_GET['restart-at']) && (int)$_GET['restart-at']>0) {
header('X-Sendfile2: '.$file.' '.$_GET['restart-at'].'-');
} else {
header('X-Sendfile: '.$file);
}

If you run the command wget http://localhost/test1.php?restart-at=500000 it will start the download at byte 500000.

Now, the problem with not following the HTTP spec, means that doing a wget -c will not work properly. I used the following PHP script:

$file = '/tmp/big-file.bin';
if (isset($_SERVER['HTTP_RANGE']) && preg_match('/\Abytes=[0-9]+-[0-9]*\z/', $_SERVER['HTTP_RANGE'])) {
header('X-Sendfile2: '.$file.' '.substr($_SERVER['HTTP_RANGE'],6));
} else {
header('X-Sendfile: '.$file);
}

This script takes the HTTP_RANGE header (which is sent by wget when used with -c option) and passed the value to X-Sendfile2. When I run wget -c http://localhost/test2.php, stop it and restart it:
  • I can see that only the second part of the file is downloaded the 2nd time I run wget (which is what I wanted)
  • But at the end, it's not properly attached to the first part of the file (which was retrieved by the first wget)

#8 Updated by icy about 5 years ago

Try sending a 206 status code and the Content-Range header from your backend.

#9 Updated by lameventanas about 5 years ago

Ok thanks to the clarification I was able to make my code work.

I assumed that lighttpd took care of the status and content-range headers because so many things could go wrong with the sendfile() call (ie: file doesn't exist, permission denied, range not satisfiable, etc).
What happens in these cases? Does lighttpd overwrite the headers in these cases? Or is there any way for the backend to know about these error conditions?

Lighttpd definitely needs better documentation.

#10 Updated by jpc about 5 years ago

icy wrote:

Try sending a 206 status code and the Content-Range header from your backend.

Thank you for the pointer!

lameventanas wrote:

Ok thanks to the clarification I was able to make my code work.

Me too. I generate the following headers and it works when I run wget -c:

header('Content-Range: bytes 2375680-12103815/12103816');
header('HTTP/1.1 206 Partial content');
header('X-Sendfile2: /tmp/big-file.bin 2375680-');

Lighttpd definitely needs better documentation.

I added an example on the wiki. Hopefully, it'll help the next person having the problem :)

Also available in: Atom