Project

General

Profile

Bug #1548

range requests not working

Added by Anonymous almost 12 years ago. Updated over 3 years ago.

Status:
Obsolete
Priority:
High
Assignee:
-
Category:
core
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Missing in 1.5.x:
Yes

Description

Test case I use:
wget -t1 --debug http://site.com/content/movies/pretty_big.wmv --header="Range:bytes=100-200" -O wget.out

Result: wget.out remains empty.


---request begin---
GET /content/movies/pretty_big.wmv HTTP/1.0
User-Agent: Mozilla/5.0 (compatible; Panther)
Accept: */*
Host: site.com
Connection: Keep-Alive
Range: bytes=100-200
X-Panther: Validate

---request end---
HTTP request sent, awaiting response... 
---response begin---
HTTP/1.0 206 Partial Content
Cache-Control: max-age=31536000
Cache-Control: max-age=31536000
Cache-Control: private
X-Powered-By: PHP/5.2.0-8+etch10
Pragma: public
Expires: 0
Content-Type: application/force-download
Content-Disposition: attachment; filename="pretty_big.wmv";
Content-Transfer-Encoding: binary
ETag: "1615866504" 
Accept-Ranges: bytes
Last-Modified: Fri, 14 Dec 2007 12:53:52 GMT
Content-Range: bytes 100-200/2553951618
Content-Length: 101
Connection: keep-alive
Date: Fri, 01 Feb 2008 08:57:00 GMT
Server: lighttpd/1.5.0

---response end---
206 Partial Content
Registered socket 5 for persistent reuse.
Disabling further reuse of socket 5.
Closed fd 5
Giving up.

strace: attached

Nothing I can see in the logs.
The requests are rewritten to go through a small php script to facilitate authentication.
I tried the test with large files, small files, with authorizer and without,
network-backends linux-sendfile and gthread-aio. Same result.

The code was checked out of trunk on 2008_01_22
I added a patch from this site that enables setting the basis of etag generation. I also use this build in clusters. I'll attach that patch too.

I enabled request debugging and will attach the output of that as well.

Logs have been sanitized to change paths and camouflage host names and such. I did not intentionally remove debugging info. If the logs look consistent I am happy to provide the originals. (in a less open channel)

If you need more information don't hesitate to contact me.

Thanks in advance for looking at the ticket.

-- oscar

light.trace (9.59 KB) light.trace strace from a byte-ranged get request Anonymous, 2008-02-01 09:28
actualized_etags.patch (5.53 KB) actualized_etags.patch etags patch I applied Anonymous, 2008-02-01 09:29
lighty.log (3.73 KB) lighty.log debug log output Anonymous, 2008-02-01 09:30

History

#1

Updated by Anonymous almost 12 years ago

huh... I just tested with a 1.4.18 server. I don't ever get anything in the wget.out file.

I wonder if my test is broken...

-- oscar

#2

Updated by Anonymous almost 12 years ago

ughh. I did the test using curl and I am getting the partial transfers ok.
Let's put this on hold while I figure out what's going on.

#3

Updated by Anonymous almost 12 years ago

Found one possible culprit:

doing the test the following way:
--header "Range:bytes=2147483650-2553951618"
fails. Any address range starting above 2G fails!

This time however I found some noise in the log:


network_gthread_aio.c.280: (error) thread returned: 2
connections.c.1381: (trace) (network-subsys sent us a fatal-error)
network_gthread_aio.c.186: (error) reading file failed: 22 (Invalid argument)
network_gthread_aio.c.280: (error) thread returned: 2
connections.c.1381: (trace) (network-subsys sent us a fatal-error)
network_gthread_aio.c.186: (error) reading file failed: 22 (Invalid argument)
network_gthread_aio.c.280: (error) thread returned: 2
connections.c.1381: (trace) (network-subsys sent us a fatal-error)

Good news is: If I switch to linux-sendfile backend the problem disappears.
gthread-aio didn't appreciate this particular task.

So I have a workaround, how can I help creating a fix?

-- oscar

#4

Updated by Anonymous almost 12 years ago

confirmed this is a > 2gb issue on my debian etch box w/ libaio.so.1.0.1, kernel 2.6.18-4-486, lighty 1.5.0 r1992:


network_gthread_aio.c.186: (error) reading file failed: 22 (Invalid argument)
network_gthread_aio.c.280: (error) thread returned: 2
connections.c.1370: (trace) (network-subsys sent us a fatal-error)

when requesting range beyond 2 gig.

this is high priority for me as i need aio to avoid constant i/o bottleneck ...

-- njahnke

#5

Updated by Anonymous almost 12 years ago

edit network_gthread_aio.c

change occurrence of pread() to pread64()

recompile

-- grenola

#6

Updated by Anonymous almost 12 years ago

thanks grenola. i've confirmed this works on my debian box. please integrate into svn (we're too lazy) :P

-- njahnke

#7

Updated by stbuehler over 10 years ago

If you compile with large-file-support (enabled by default), pread should be the right function. I don't think we should use pread64 directly (pread with lfs will probably use a "pread64" syscall).

#8

Updated by darix over 10 years ago

  • Status changed from New to Need Feedback

can any of the commenters please mention if they had their glib2 and lighttpd compiled with largefile support (LFS) and if the bug also triggered with LFS enabled?

#9

Updated by gstrauss over 3 years ago

  • Assignee deleted (jan)
  • Missing in 1.5.x set to Yes
#10

Updated by gstrauss over 3 years ago

  • Status changed from Need Feedback to Obsolete

Also available in: Atom