Project

General

Profile

If-Modified-Since is ignored when the request goes via shared proxy

Added by anto over 12 years ago

Hello Everybody,

I am using lighttpd 1.4.23-3+b2 on my Debian Squeeze server.

What I understood is that when the HTTP request header contain "If-Modified-Since" and its value match with "Last-Modified", then lighttpd should in any case responses with 304. In my case, if the request does not go through a shared proxy, then lighttpd responses with 304. But if it goes through shared proxy, then lighttpd responses with 200, which only causes unnecessary use of bandwidth. And I can not workaround this using mod_magnet, because lighty.request["If-Modified-Since"] is not accessible within lua script.

Could anybody give me suggestions how to workaround this?

Thanks a lot in advance for your help. And please find below the requests from my Firefox browser and the responses of lighttpd, which I copied from HttpFox.

Kind regards,

Anto
----

First request
------------------------------------------------------------------------------------------------------------------
(Request-Line           GET /about.html HTTP/1.1
Host                    thetwins.info
User-Agent              Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3
Accept                  text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language         en-gb
Accept-Encoding         gzip,deflate
Accept-Charset          ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive              300
Proxy-Connection        keep-alive
Referer                 http://thetwins.info/home.html
Cookie                  SESS160bb3c5c9abb9cdb69842bb19ad5339=5bc75584881aa70fff938b9b1ce262f8; has_js=1
If-Modified-Since       Wed, 09 Sep 2009 06:48:02 GMT

(Status-Line)           HTTP/1.1 200 OK
Content-Type            text/html
Cache-Control           private, max-age=31104000, must-revalidate
Last-Modified           Wed, 09 Sep 2009 06:48:02 GMT
Accept-Ranges           bytes
Expires                 Fri, 15 Nov 2002 09:45:00 GMT
Date                    Thu, 10 Sep 2009 08:13:40 GMT
Server                  lighttpd
Content-Length          2432
Proxy-Connection        Keep-Alive
Connection              Keep-Alive
Content-Encoding        gzip

Second request
------------------------------------------------------------------------------------------------------------------
(Request-Line)          GET /about.html HTTP/1.1
Host                    thetwins.info
User-Agent              Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3
Accept                  text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language         en-gb
Accept-Encoding         gzip,deflate
Accept-Charset          ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive              300
Proxy-Connection        keep-alive
Referer                 http://thetwins.info/home.html
Cookie                  SESS160bb3c5c9abb9cdb69842bb19ad5339=5bc75584881aa70fff938b9b1ce262f8; has_js=1
If-Modified-Since       Wed, 09 Sep 2009 06:48:02 GMT

(Status-Line)           HTTP/1.1 200 OK
Content-Type            text/html
Cache-Control           private, max-age=31104000, must-revalidate
Last-Modified           Wed, 09 Sep 2009 06:48:02 GMT
Accept-Ranges           bytes
Expires                 Fri, 15 Nov 2002 09:45:00 GMT
Date                    Thu, 10 Sep 2009 08:13:54 GMT
Server                  lighttpd
Content-Length          2432
Proxy-Connection        Keep-Alive
Connection              Keep-Alive
Content-Encoding        gzip

Replies (6)

RE: If-Modified-Since is ignored when the request goes via shared proxy - Added by stbuehler over 12 years ago

Fix your proxy (or prove that lighty is the problem).

(Btw: the expire header looks strange... perhaps some misconfiguration?)

RE: If-Modified-Since is ignored when the request goes via shared proxy - Added by anto over 12 years ago

Thanks for your response, stbuehler. But I think your suggestion is far from practical. All users usually don't even know the setup of their shared proxy. What they know is usually just the server name/IP address of the proxy and the port that they have to enter in their web browser. That is the same for me. And I don't have the privilege to access the setup of our shared proxy as it is managed by our IT department.

From the responses that I provided before, it seems lighttpd does not response properly. I think that shows the prove that you asked. If not, please give me more detail of why you think lighttpd is correct by sending 200 response to the request with "If-Modified-Since" header set?

About the Expires header, I intentionally set it like that to ensure that any HTTP/1.0 browsers (e.g. bots) do not cache the pages. Its function is overwritten by Cache-Control max-age for HTTP/1.1 browsers.

Edit:
I don't want to raise a bug report yet, as maybe the problem is on my configuration.

RE: If-Modified-Since is ignored when the request goes via shared proxy - Added by anto over 12 years ago

Quote from HTTP 1.1 specification (http://www.ietf.org/rfc/rfc2616.txt)

c) If the variant has not been modified since a valid If-Modified-Since date, the server SHOULD return a 304 (Not Modified) response.

RE: If-Modified-Since is ignored when the request goes via shared proxy - Added by anto over 12 years ago

Below is the right response from lighttpd, for the request which does not goes via a shared proxy. Everybody should be able verify this by accessing the same url as my web site is on the internet. So is it a valid bug report or a misconfiguration on my server? If that would be a misconfiguration then where should I have a look?

I am starting to believe that this is a bug on lighttpd as lighttpd behaviour is against HTTP/1.1 specification, but I am not 100% sure. Or, is lighttpd not fully support HTTP/1.1?

(Request-Line)          GET /about.html HTTP/1.1
Host                    thetwins.info
User-Agent              Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3
Accept                  text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language         en-gb
Accept-Encoding         gzip,deflate
Accept-Charset          ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive              300
Connection              keep-alive
Referer                 http://thetwins.info/siteinfo.html
Cookie                  SESS160bb3c5c9abb9cdb69842bb19ad5339=8352e5bf035da572968e811e4aefbaea; has_js=1
If-Modified-Since       Wed, 09 Sep 2009 06:48:02 GMT

(Status-Line)           HTTP/1.1 304 Not Modified
Content-Type            text/html
Cache-Control           private, max-age=31104000, must-revalidate
Last-Modified           Wed, 09 Sep 2009 06:48:02 GMT
Accept-Ranges           bytes
Expires                 Fri, 15 Nov 2002 09:45:00 GMT
Date                    Thu, 10 Sep 2009 17:30:46 GMT
Server                  lighttpd

RE: If-Modified-Since is ignored when the request goes via shared proxy - Added by stbuehler over 12 years ago

Lighty does the right thing as you showed us, it gives 304 for the request. So why do you think it is lighty's fault if you get 200 through a proxy?

Show us the request the proxy sends to lighty and the response the proxy gets...

RE: If-Modified-Since is ignored when the request goes via shared proxy - Added by anto over 12 years ago

I have to admit that you are right, stbuehler :)

There is nothing wrong with lighttpd. It is because of the behaviour of the proxy server (BlueCoat) which does not forward the "If-Modified-Since", due to the Cache-Control that I set to "private". I want it to be that way, i.e. I don't want anything from my website being cached by any proxy server, so I don't want to set it to "public". Lighttpd sends 304 response when I set Cache-Control to "public" and the proxy has the content on its cache.

It seems that there is no way to workaround this without bothering the proxy server.

Anyway, below are the requests and responses from HttpFox and server.errorlog that you requested.

Request from Firefox to the proxy (on HttpFox)

(Request-Line)          GET /about.html HTTP/1.1
Host                    thetwins.info
User-Agent              Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3
Accept                  text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language         en-gb
Accept-Encoding         gzip,deflate
Accept-Charset          ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive              300
Proxy-Connection        keep-alive
Referer                 http://thetwins.info/siteinfo.html
Cookie                  SESS160bb3c5c9abb9cdb69842bb19ad5339=8352e5bf035da572968e811e4aefbaea; has_js=1
If-Modified-Since       Wed, 09 Sep 2009 06:48:02 GMT

Request from proxy to lighttpd (on server.errorlog)

2009-09-10 21:55:08: (request.c.302) fd: 18 request-len: 548
GET /about.html HTTP/1.1
Host: thetwins.info
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-gb
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Referer: http://thetwins.info/siteinfo.html
Cookie: SESS160bb3c5c9abb9cdb69842bb19ad5339=8352e5bf035da572968e811e4aefbaea; has_js=1
Cache-Control: max-stale=0
Connection: Keep-Alive
X-BlueCoat-Via: 540F9D6095AC9DA8

Response from lighttpd to proxy (on server.errorlog)

2009-09-10 21:55:08: (response.c.126) Response-Header:
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Type: text/html
Cache-Control: private, max-age=31104000, must-revalidate
Last-Modified: Wed, 09 Sep 2009 06:48:02 GMT
Accept-Ranges: bytes
Expires: Fri, 15 Nov 2002 09:45:00 GMT
Content-Length: 2432
Date: Thu, 10 Sep 2009 19:55:08 GMT
Server: lighttpd

Response forwarded by proxy to Firefox (on HttpFox)

(Status-Line)           HTTP/1.1 200 OK
Content-Type            text/html
Cache-Control           private, max-age=31104000, must-revalidate
Last-Modified           Wed, 09 Sep 2009 06:48:02 GMT
Accept-Ranges           bytes
Expires                 Fri, 15 Nov 2002 09:45:00 GMT
Date                    Thu, 10 Sep 2009 19:55:08 GMT
Server                  lighttpd
Content-Length          2432
Proxy-Connection        Keep-Alive
Connection              Keep-Alive
Content-Encoding        gzip

    (1-6/6)