If-Modified-Since is ignored when the request goes via shared proxy
Added by anto over 14 years ago
Hello Everybody,
I am using lighttpd 1.4.23-3+b2 on my Debian Squeeze server.
What I understood is that when the HTTP request header contain "If-Modified-Since" and its value match with "Last-Modified", then lighttpd should in any case responses with 304. In my case, if the request does not go through a shared proxy, then lighttpd responses with 304. But if it goes through shared proxy, then lighttpd responses with 200, which only causes unnecessary use of bandwidth. And I can not workaround this using mod_magnet, because lighty.request["If-Modified-Since"] is not accessible within lua script.
Could anybody give me suggestions how to workaround this?
Thanks a lot in advance for your help. And please find below the requests from my Firefox browser and the responses of lighttpd, which I copied from HttpFox.
Kind regards,
Anto
----
First request ------------------------------------------------------------------------------------------------------------------ (Request-Line GET /about.html HTTP/1.1 Host thetwins.info User-Agent Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language en-gb Accept-Encoding gzip,deflate Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive 300 Proxy-Connection keep-alive Referer http://thetwins.info/home.html Cookie SESS160bb3c5c9abb9cdb69842bb19ad5339=5bc75584881aa70fff938b9b1ce262f8; has_js=1 If-Modified-Since Wed, 09 Sep 2009 06:48:02 GMT (Status-Line) HTTP/1.1 200 OK Content-Type text/html Cache-Control private, max-age=31104000, must-revalidate Last-Modified Wed, 09 Sep 2009 06:48:02 GMT Accept-Ranges bytes Expires Fri, 15 Nov 2002 09:45:00 GMT Date Thu, 10 Sep 2009 08:13:40 GMT Server lighttpd Content-Length 2432 Proxy-Connection Keep-Alive Connection Keep-Alive Content-Encoding gzip
Second request ------------------------------------------------------------------------------------------------------------------ (Request-Line) GET /about.html HTTP/1.1 Host thetwins.info User-Agent Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language en-gb Accept-Encoding gzip,deflate Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive 300 Proxy-Connection keep-alive Referer http://thetwins.info/home.html Cookie SESS160bb3c5c9abb9cdb69842bb19ad5339=5bc75584881aa70fff938b9b1ce262f8; has_js=1 If-Modified-Since Wed, 09 Sep 2009 06:48:02 GMT (Status-Line) HTTP/1.1 200 OK Content-Type text/html Cache-Control private, max-age=31104000, must-revalidate Last-Modified Wed, 09 Sep 2009 06:48:02 GMT Accept-Ranges bytes Expires Fri, 15 Nov 2002 09:45:00 GMT Date Thu, 10 Sep 2009 08:13:54 GMT Server lighttpd Content-Length 2432 Proxy-Connection Keep-Alive Connection Keep-Alive Content-Encoding gzip
Replies (6)
RE: If-Modified-Since is ignored when the request goes via shared proxy - Added by stbuehler over 14 years ago
Fix your proxy (or prove that lighty is the problem).
(Btw: the expire header looks strange... perhaps some misconfiguration?)
RE: If-Modified-Since is ignored when the request goes via shared proxy - Added by anto over 14 years ago
Thanks for your response, stbuehler. But I think your suggestion is far from practical. All users usually don't even know the setup of their shared proxy. What they know is usually just the server name/IP address of the proxy and the port that they have to enter in their web browser. That is the same for me. And I don't have the privilege to access the setup of our shared proxy as it is managed by our IT department.
From the responses that I provided before, it seems lighttpd does not response properly. I think that shows the prove that you asked. If not, please give me more detail of why you think lighttpd is correct by sending 200 response to the request with "If-Modified-Since" header set?
About the Expires header, I intentionally set it like that to ensure that any HTTP/1.0 browsers (e.g. bots) do not cache the pages. Its function is overwritten by Cache-Control max-age for HTTP/1.1 browsers.
Edit:
I don't want to raise a bug report yet, as maybe the problem is on my configuration.
RE: If-Modified-Since is ignored when the request goes via shared proxy - Added by anto over 14 years ago
Quote from HTTP 1.1 specification (http://www.ietf.org/rfc/rfc2616.txt)
c) If the variant has not been modified since a valid If-Modified-Since date, the server SHOULD return a 304 (Not Modified) response.
RE: If-Modified-Since is ignored when the request goes via shared proxy - Added by anto over 14 years ago
Below is the right response from lighttpd, for the request which does not goes via a shared proxy. Everybody should be able verify this by accessing the same url as my web site is on the internet. So is it a valid bug report or a misconfiguration on my server? If that would be a misconfiguration then where should I have a look?
I am starting to believe that this is a bug on lighttpd as lighttpd behaviour is against HTTP/1.1 specification, but I am not 100% sure. Or, is lighttpd not fully support HTTP/1.1?
(Request-Line) GET /about.html HTTP/1.1 Host thetwins.info User-Agent Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language en-gb Accept-Encoding gzip,deflate Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive 300 Connection keep-alive Referer http://thetwins.info/siteinfo.html Cookie SESS160bb3c5c9abb9cdb69842bb19ad5339=8352e5bf035da572968e811e4aefbaea; has_js=1 If-Modified-Since Wed, 09 Sep 2009 06:48:02 GMT (Status-Line) HTTP/1.1 304 Not Modified Content-Type text/html Cache-Control private, max-age=31104000, must-revalidate Last-Modified Wed, 09 Sep 2009 06:48:02 GMT Accept-Ranges bytes Expires Fri, 15 Nov 2002 09:45:00 GMT Date Thu, 10 Sep 2009 17:30:46 GMT Server lighttpd
RE: If-Modified-Since is ignored when the request goes via shared proxy - Added by stbuehler over 14 years ago
Lighty does the right thing as you showed us, it gives 304 for the request. So why do you think it is lighty's fault if you get 200 through a proxy?
Show us the request the proxy sends to lighty and the response the proxy gets...
RE: If-Modified-Since is ignored when the request goes via shared proxy - Added by anto over 14 years ago
I have to admit that you are right, stbuehler :)
There is nothing wrong with lighttpd. It is because of the behaviour of the proxy server (BlueCoat) which does not forward the "If-Modified-Since", due to the Cache-Control that I set to "private". I want it to be that way, i.e. I don't want anything from my website being cached by any proxy server, so I don't want to set it to "public". Lighttpd sends 304 response when I set Cache-Control to "public" and the proxy has the content on its cache.
It seems that there is no way to workaround this without bothering the proxy server.
Anyway, below are the requests and responses from HttpFox and server.errorlog that you requested.
Request from Firefox to the proxy (on HttpFox)
(Request-Line) GET /about.html HTTP/1.1 Host thetwins.info User-Agent Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language en-gb Accept-Encoding gzip,deflate Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive 300 Proxy-Connection keep-alive Referer http://thetwins.info/siteinfo.html Cookie SESS160bb3c5c9abb9cdb69842bb19ad5339=8352e5bf035da572968e811e4aefbaea; has_js=1 If-Modified-Since Wed, 09 Sep 2009 06:48:02 GMT
Request from proxy to lighttpd (on server.errorlog)
2009-09-10 21:55:08: (request.c.302) fd: 18 request-len: 548 GET /about.html HTTP/1.1 Host: thetwins.info User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-gb Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Referer: http://thetwins.info/siteinfo.html Cookie: SESS160bb3c5c9abb9cdb69842bb19ad5339=8352e5bf035da572968e811e4aefbaea; has_js=1 Cache-Control: max-stale=0 Connection: Keep-Alive X-BlueCoat-Via: 540F9D6095AC9DA8
Response from lighttpd to proxy (on server.errorlog)
2009-09-10 21:55:08: (response.c.126) Response-Header: HTTP/1.1 200 OK Content-Encoding: gzip Content-Type: text/html Cache-Control: private, max-age=31104000, must-revalidate Last-Modified: Wed, 09 Sep 2009 06:48:02 GMT Accept-Ranges: bytes Expires: Fri, 15 Nov 2002 09:45:00 GMT Content-Length: 2432 Date: Thu, 10 Sep 2009 19:55:08 GMT Server: lighttpd
Response forwarded by proxy to Firefox (on HttpFox)
(Status-Line) HTTP/1.1 200 OK Content-Type text/html Cache-Control private, max-age=31104000, must-revalidate Last-Modified Wed, 09 Sep 2009 06:48:02 GMT Accept-Ranges bytes Expires Fri, 15 Nov 2002 09:45:00 GMT Date Thu, 10 Sep 2009 19:55:08 GMT Server lighttpd Content-Length 2432 Proxy-Connection Keep-Alive Connection Keep-Alive Content-Encoding gzip