Bug #2529
closedX-Sendfile not working with multipart streaming traffic (fastcgi)
Description
Hi, I have a fastcgi backend that is sending an MJPEG stream over HTTP. It works fine when the backend sends the image data through the fastcgi interface, but is quite slow when sending larger volumes of data (say, 1080p). I was hoping to use X-Sendfile to serve up a static file of each image (stored to /dev/shm, for instance), and that works if I send the image statically (i.e., with Content-type: video/jpeg or image/jpeg). However, when I use multipart headers for MJPEG streaming, the X-Sendfile (or X-LIGHTTPD-send-file) directive never gets pulled out of my header (and hence is not replaced with the file data).
At a guess, I'm thinking that X-Sendfile is not parsed if there is no HTTP response code at the top of the header, but I can't prove this. In any case, why is X-Sendfile not being parsed? Am I doing something wrong? This seems like a bug, but maybe I missed something?
I'm running lighttpd on an embedded ARM, configured as follows:
./configure --host=arm-unknown-linux --disable-static --build=x86-unknown-linux --enable-shared --without-zlib --without-bzip2 --without-pcre
I see the problem with a firefox client, and verified that X-Sendfile is still in the header that went out on the wire via Wireshark.
I verified this with both lighttpd v1.4.30 and 1.4.33.
Thanks in advance!
Updated by stbuehler over 11 years ago
- Status changed from New to Invalid
we certainly are not going to parse multipart mime responses; x-sendfile was never made to work that way.
yes, this means that you cannot implement "live" streaming with lighttpd, unless you write your own module (or change existing ones).
Updated by isthomps over 11 years ago
we certainly are not going to parse multipart mime responses; x-sendfile was never made to work that way.
Could you please explain what limits x-sendfile in this manner? Why is parsing multipart messages different from parsing standard messages? I'd very much appreciate your expert insight on the current implementation and its limitations in case I get the time to extend x-sendfile support in this way.
An efficient means for lighttpd to stream video would surely be popular, especially now that Cisco is making H.264 easier to license.
Thanks in advance!
Updated by stbuehler over 11 years ago
X-sendfile is just a normal HTTP header. Multipart is about the content of an HTTP response. We only look at the outer HTTP headers, not at the inner MIME/multipart headers.
Also usually your streaming server should get good feedback if (the connection to) a client is too slow for the current bitrate, and stop wasting resources server side; this is just not possible with the design in 1.4.x. In lighttpd2 (our development branch) we will try to fix this (it actually should already work), so backends should be able to stream. But we still won't parse multipart responses :)
Updated by isthomps over 11 years ago
Also usually your streaming server should get good feedback if (the connection to) a client is too slow for the current bitrate, and stop wasting resources server side; this is just not possible with the design in 1.4.x. In lighttpd2 (our development branch) we will try to fix this (it actually should already work), so backends should be able to stream. But we still won't parse multipart responses :)
Thank you for the details. It seems that this is something that might cause people to drop lighttpd for nginx, which is a shame IMHO. Why is parsing multipart responses over fastcgi all that difficult? It could be done as an option that one enables so as not to impact performance, and just is a matter of scanning a few more headers... Just trying to further understand.
Also available in: Atom