Bug #2221

responceText incorrect on 403

Added by lightning almost 4 years ago. Updated almost 4 years ago.

Status:InvalidStart date:2010-06-06
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:-
Missing in 1.5.x:No

Description

Due to some undetermined error in processing I get a (presumably correct) 403 . However, the responseText that is sent the responseXML content. This is clearly wrong and causes the browser to display an error about incorrect encoding.

Here is the content of responseText in this case:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <title>403 - Forbidden</title> </head> <body> <h1>403 - Forbidden</h1> </body> </html>

History

#1 Updated by lightning almost 4 years ago

What happened to the edit facility?

v 1.4.26

#2 Updated by icy almost 4 years ago

  • Status changed from New to Need Feedback

Can you rephrase what you think is wrong? The w3 validator says the 403 document is perfectly fine, not even a warning.

What browser displays an error?

#3 Updated by lightning almost 4 years ago

OK, it looks like lighttpd's output was correct. It was firefox perverting it.

FF had the good idea to insert the 403 responce inside an embedded svg object. This then triggered more DOM magic by FF as it found what it inserted was not valid image/svg-xml markup, hence the error messages and screwed about content.

Sorry for thinking this was the server.

I don't seem to have the option to close this , so please feel free to do so.

BTW, it seems the 403 was due to mod_evasive. Would it be useful to send some sort of message indicating a server limit (as is common with ftp connection limits) rather than a brutal and rather obscure 403.

403 Forbidded is not really appropriate since the URL is not forbidden, it's a throttling device. I suspect replying 403 to an url that is legit maybe a spec violation.

It confused the hell out of me for a while , I'm sure it would confuse someone using the site.

I have set more suitable values to avoid this but it raised the question of whether 403 was the correct server response here.

Thanks.

#4 Updated by Olaf-van-der-Spek almost 4 years ago

lightning wrote:

FF had the good idea to insert the 403 responce inside an embedded svg object.

What content-type header was sent?

#5 Updated by icy almost 4 years ago

  • Status changed from Need Feedback to Invalid

#6 Updated by lightning almost 4 years ago

What content-type header was sent?

either it was in response to :

function refresh_svg(){
// opera object does not have svgobj.location
if ((svgobj)&&(svgobj.location)) {
if (debug) alert("before reaload "+ svgobj.location);

svgobj.location.reload(true)                                                              
}
else {
if (debug) alert("full reload");
location.reload(true);
// NB this breaks the repeat loop since timer obj gets reloaded.
}
} // end refresh_svg

This reload normally works as expected and gets the current svg from the server.

or it is from the xmlhttp response function that called refresh_svg():

function display_response() {
if (xmlhttp.readyState==4) {
// alert(xmlhttp.responseText);
refresh_svg()
document.forms0.elements['resp'].value=xmlhttp.responseText
}
}

this function also gets a 403 and displays it in the form as expected. Viewing the same exchange in opera I see the svg replaced by "403 forbidden" BEFORE the reload(true). This makes me think that , for some reason, the response to the xmlhttp exchange is getting inserted into the html display in both browsers. Opera does this in a way that does not create an error.

I would need to program a series of different web pages to isolate what exactly is happening.

It has cost me more time than it should to track down why I was getting a 403. I'm not sure I have a much time to work out exactly why each browser does what it does when it gets refused the page.

Since all browsers now seem to screw around with the content by DOM and js, it's rather hard to determine what they did actually receive as a response when it goes wrong.

Do you have any thoughts on whether it is legitimate to return 403 to a valid URL in this context?

#7 Updated by Olaf-van-der-Spek almost 4 years ago

lightning wrote:

What content-type header was sent?

either it was in response to :

Use (something like) Wireshark to see what was actually send, there's no need to guess.

#8 Updated by lightning almost 4 years ago

  • Status changed from Invalid to Reopened

Hmm, maybe there is a bug here.

Here's firefox requesting the svgz when evasive.max-conns-per-ip = 3 . This rapidly causes a refusal.

0000  00 d0 69 40 b2 73 00 d0  b7 8e ac 39 08 00 45 00   ..i@.s.. ...9..E.
0010  02 20 e7 cc 40 00 40 06  cb 85 c0 a8 02 03 c0 a8   . ..@.@. ........
0020  02 32 97 55 1f 90 cd ce  a9 89 c9 76 35 96 80 18   .2.U.... ...v5...
0030  00 6c 23 b4 00 00 01 01  08 0a 0e a3 40 4f 01 0e   .l#..... ....@O..
0040  3b b9 47 45 54 20 2f 64  61 74 61 5f 69 6d 61 67   ;.GET /d ata_imag
0050  65 73 2f 32 30 30 74 61  6e 6b 2d 6d 6f 6e 69 74   es/200ta nk-monit
0060  6f 72 2e 73 76 67 7a 20  48 54 54 50 2f 31 2e 31   or.svgz  HTTP/1.1
0070  0d 0a 48 6f 73 74 3a 20  31 39 32 2e 31 36 38 2e   ..Host:  192.168.
0080  32 2e 35 30 3a 38 30 38  30 0d 0a 55 73 65 72 2d   2.50:808 0..User-
0090  41 67 65 6e 74 3a 20 4d  6f 7a 69 6c 6c 61 2f 35   Agent: M ozilla/5
00a0  2e 30 20 28 58 31 31 3b  20 55 3b 20 4c 69 6e 75   .0 (X11;  U; Linu
00b0  78 20 69 36 38 36 3b 20  65 6e 2d 55 53 3b 20 72   x i686;  en-US; r
00c0  76 3a 31 2e 39 2e 32 29  20 47 65 63 6b 6f 2f 32   v:1.9.2)  Gecko/2
00d0  30 31 30 30 32 31 32 0d  0a 41 63 63 65 70 74 3a   0100212. .Accept:
00e0  20 74 65 78 74 2f 68 74  6d 6c 2c 61 70 70 6c 69    text/ht ml,appli
00f0  63 61 74 69 6f 6e 2f 78  68 74 6d 6c 2b 78 6d 6c   cation/x html+xml
0100  2c 61 70 70 6c 69 63 61  74 69 6f 6e 2f 78 6d 6c   ,applica tion/xml
0110  3b 71 3d 30 2e 39 2c 2a  2f 2a 3b 71 3d 30 2e 38   ;q=0.9,* /*;q=0.8
0120  0d 0a 41 63 63 65 70 74  2d 4c 61 6e 67 75 61 67   ..Accept -Languag
0130  65 3a 20 65 6e 2c 66 72  2d 66 72 3b 71 3d 30 2e   e: en,fr -fr;q=0.
0140  37 2c 65 6e 2d 75 73 3b  71 3d 30 2e 33 0d 0a 41   7,en-us; q=0.3..A
0150  63 63 65 70 74 2d 45 6e  63 6f 64 69 6e 67 3a 20   ccept-En coding: 
0160  67 7a 69 70 2c 64 65 66  6c 61 74 65 0d 0a 41 63   gzip,def late..Ac
0170  63 65 70 74 2d 43 68 61  72 73 65 74 3a 20 49 53   cept-Cha rset: IS
0180  4f 2d 38 38 35 39 2d 31  2c 75 74 66 2d 38 3b 71   O-8859-1 ,utf-8;q
0190  3d 30 2e 37 2c 2a 3b 71  3d 30 2e 37 0d 0a 4b 65   =0.7,*;q =0.7..Ke
01a0  65 70 2d 41 6c 69 76 65  3a 20 31 31 35 0d 0a 43   ep-Alive : 115..C
01b0  6f 6e 6e 65 63 74 69 6f  6e 3a 20 6b 65 65 70 2d   onnectio n: keep-
01c0  61 6c 69 76 65 0d 0a 52  65 66 65 72 65 72 3a 20   alive..R eferer: 
01d0  68 74 74 70 3a 2f 2f 31  39 32 2e 31 36 38 2e 32   http://1 92.168.2
01e0  2e 35 30 3a 38 30 38 30  2f 70 72 6f 74 65 63 74   .50:8080 /protect
01f0  65 64 2f 67 6e 75 70 6c  6f 74 2e 68 74 6d 6c 0d   ed/gnupl ot.html.
0200  0a 50 72 61 67 6d 61 3a  20 6e 6f 2d 63 61 63 68   .Pragma:  no-cach
0210  65 0d 0a 43 61 63 68 65  2d 43 6f 6e 74 72 6f 6c   e..Cache -Control
0220  3a 20 6e 6f 2d 63 61 63  68 65 0d 0a 0d 0a         : no-cac he....  

0000  00 d0 b7 8e ac 39 00 d0  69 40 b2 73 08 00 45 00   .....9.. i@.s..E.
0010  02 24 08 0b 40 00 40 06  ab 43 c0 a8 02 32 c0 a8   .$..@.@. .C...2..
0020  02 03 1f 90 97 55 c9 76  35 96 cd ce ab 75 80 18   .....U.v 5....u..
0030  0f 80 6a 18 00 00 01 01  08 0a 01 0e 3b bb 0e a3   ..j..... ....;...
0040  40 4f 48 54 54 50 2f 31  2e 31 20 34 30 33 20 46   @OHTTP/1 .1 403 F
0050  6f 72 62 69 64 64 65 6e  0d 0a 43 6f 6e 74 65 6e   orbidden ..Conten
0060  74 2d 45 6e 63 6f 64 69  6e 67 3a 20 67 7a 69 70   t-Encodi ng: gzip
0070  0d 0a 43 6f 6e 74 65 6e  74 2d 54 79 70 65 3a 20   ..Conten t-Type: 
0080  74 65 78 74 2f 68 74 6d  6c 0d 0a 43 6f 6e 74 65   text/htm l..Conte
0090  6e 74 2d 4c 65 6e 67 74  68 3a 20 33 34 35 0d 0a   nt-Lengt h: 345..
00a0  44 61 74 65 3a 20 53 75  6e 2c 20 30 36 20 4a 75   Date: Su n, 06 Ju
00b0  6e 20 32 30 31 30 20 31  38 3a 33 33 3a 35 35 20   n 2010 1 8:33:55 
00c0  47 4d 54 0d 0a 53 65 72  76 65 72 3a 20 6c 69 67   GMT..Ser ver: lig
00d0  68 74 74 70 64 0d 0a 0d  0a 3c 3f 78 6d 6c 20 76   httpd... .<?xml v
00e0  65 72 73 69 6f 6e 3d 22  31 2e 30 22 20 65 6e 63   ersion=" 1.0" enc
00f0  6f 64 69 6e 67 3d 22 69  73 6f 2d 38 38 35 39 2d   oding="i so-8859-
0100  31 22 3f 3e 0a 3c 21 44  4f 43 54 59 50 45 20 68   1"?>.<!D OCTYPE h
0110  74 6d 6c 20 50 55 42 4c  49 43 20 22 2d 2f 2f 57   tml PUBL IC "-//W
0120  33 43 2f 2f 44 54 44 20  58 48 54 4d 4c 20 31 2e   3C//DTD  XHTML 1.
0130  30 20 54 72 61 6e 73 69  74 69 6f 6e 61 6c 2f 2f   0 Transi tional//
0140  45 4e 22 0a 20 20 20 20  20 20 20 20 20 22 68 74   EN".          "ht
0150  74 70 3a 2f 2f 77 77 77  2e 77 33 2e 6f 72 67 2f   tp://www .w3.org/
0160  54 52 2f 78 68 74 6d 6c  31 2f 44 54 44 2f 78 68   TR/xhtml 1/DTD/xh
0170  74 6d 6c 31 2d 74 72 61  6e 73 69 74 69 6f 6e 61   tml1-tra nsitiona
0180  6c 2e 64 74 64 22 3e 0a  3c 68 74 6d 6c 20 78 6d   l.dtd">. <html xm
0190  6c 6e 73 3d 22 68 74 74  70 3a 2f 2f 77 77 77 2e   lns="htt p://www.
01a0  77 33 2e 6f 72 67 2f 31  39 39 39 2f 78 68 74 6d   w3.org/1 999/xhtm
01b0  6c 22 20 78 6d 6c 3a 6c  61 6e 67 3d 22 65 6e 22   l" xml:l ang="en" 
01c0  20 6c 61 6e 67 3d 22 65  6e 22 3e 0a 20 3c 68 65    lang="e n">. <he
01d0  61 64 3e 0a 20 20 3c 74  69 74 6c 65 3e 34 30 33   ad>.  <t itle>403
01e0  20 2d 20 46 6f 72 62 69  64 64 65 6e 3c 2f 74 69    - Forbi dden</ti
01f0  74 6c 65 3e 0a 20 3c 2f  68 65 61 64 3e 0a 20 3c   tle>. </ head>. <
0200  62 6f 64 79 3e 0a 20 20  3c 68 31 3e 34 30 33 20   body>.   <h1>403 
0210  2d 20 46 6f 72 62 69 64  64 65 6e 3c 2f 68 31 3e   - Forbid den</h1>
0220  0a 20 3c 2f 62 6f 64 79  3e 0a 3c 2f 68 74 6d 6c   . </body >.</html
0230  3e 0a                                              >.       

I notice that the reply header claims gzip compression (which would be correct for svgz) but is not correct for the 403 response.

The following is required for FF to correctly receive svgz.

$HTTP["url"] =~ "\.svgz$" {                       
setenv.add-response-header = ( "Content-Encoding" => "gzip" ),
compress.filetype = ("")
}

Should lighttpd be either compressing the 403 reply or not declaring it to be compressed when it is text?

#9 Updated by Olaf-van-der-Spek almost 4 years ago

  • Target version set to 1.4.28

Ideally it would compress the error response, but that's another feature request.

#10 Updated by lightning almost 4 years ago

Feature requests aside , you can't send out a header that announces compressed data then send text in rest of the packet.

Does this imply that any use of setenv.add-response-header is unsafe in anything but http 200 ? Is this an invalid use of that feature?

Is there a way to make the additional header conditional on the file being available ready to send (mod-evasive notwithstanding)?

Maybe I need a more complex config for this file type.

#11 Updated by lightning almost 4 years ago

In fact this sort of regex matching on the URL is nothing better that file extension sniffing. Is there a way to set (or not set) the headers based on the content that is served rather than on the "file extension" of the request?

#12 Updated by lightning almost 4 years ago

FYI:
http://www.w3.org/TR/2008/WD-SVGMobile12-20080912/conform.html#ConformingSVGServers

the encoding header seems to be the right to conform to standards. So there is a definite need for a means to add this header without it becoming a default for error condition replies as well.

Suggestions?

#13 Updated by Olaf-van-der-Spek almost 4 years ago

lightning wrote:

Does this imply that any use of setenv.add-response-header is unsafe in anything but http 200 ? Is this an invalid use of that feature?

Maybe. What's the MIME type for .svgz?
.svgz looks weird, either it's .svg with gzip content encoding or it's .svgz (.svg.gz?) without gzip content encoding.

#14 Updated by lightning almost 4 years ago

Maybe what? Maybe bad usage or maybe setenv.add-response-header is unsafe?

mimetype.assign = (
".svg"=> "image/svg+xml",
".svgz"=> "image/svg+xml",
...
svgz is defined with the same mime type as svg. Content-Encoding , as shown in the w3.org link is gzip encoding for svgz format files. svgz is quite simply an svg file compressed with gzip.

It seems the fundamental problem here is defining a header based on the last few chars of the URL requested and not on the basis of the content returned by the server.

If client requests an image and server returns html error message, the headers must match what is sent and not what is asked for.

I think lighttpd needs to be coded to correctly respond to svgz requests without the need to artificially add this header by a manual config tweak that produces illegal output when a server error is sent.

This is a workaround for an incorrect default response but it not a stable solution as this case reveals.

#15 Updated by stbuehler almost 4 years ago

  • Status changed from Reopened to Invalid
  • Target version deleted (1.4.28)

lightning wrote:

[...]
I think lighttpd needs to be coded to correctly respond to svgz requests without the need to artificially add this header by a manual config tweak that produces illegal output when a server error is sent.

Yes, we certainly should add specific code to handle svgz requests... - is that really what you think? not going to happen.
Use mod_compress (this would 1. do the compression for you 2. detect what the browser supports) or write a new "mod_try_precompressed".

This is a workaround for an incorrect default response but it not a stable solution as this case reveals.

I think it is pretty clear that setenv.add-response-header is dangerous - use at your own risk.

We could ofc reset some headers in our error handling code, but there are probably always some headers we don't catch.

#16 Updated by lightning almost 4 years ago

  • Status changed from Invalid to Reopened

I agree setenv.add-response-header is bad, I already said it was a hack and I am looking for correct, clean solution.

from www consortium link above:

</quote>
Also, if the SVG file is compressed with gzip or deflate, Conforming SVG Servers must indicate this with the appropriate header, according to what the protocol supports. Specifically, for content compressed by the server immediately prior to transfer, the server must use the "Transfer-Encoding: gzip" or "Transfer-Encoding: deflate" headers as appropriate, and for content stored in a compressed format on the server (e.g. with the file extension "svgz"), the server must use the "Content-Encoding: gzip" or "Content-Encoding: deflate" headers as appropriate. </quote>

lighttpd claims to be standards compliant and in this case it is clearly not. The bug is not INVALID because you don't want to fix it. (Maybe you need to add WONTFIX to the status list )

I already have mod_compress loaded but I need to compress the files on creation not when requested. They are large , frequently updated data plots. Relying on mod_compress means some response times would be unacceptably long.

The changes needed would seem to be fairly trivial , so if you can give me some pointers at where I need to look to add the correct headers I'll see if I can provide a patch.

Please be as specific as possible since I don't have a huge amount of free time to spend getting familiar with large areas of the source code.

Thanks for your reply.

#17 Updated by stbuehler almost 4 years ago

  • Status changed from Reopened to Invalid

There is no standard conforming way of delivering .svgz, as not all http clients have to support compression.

I don't care that you can trigger bad behaviour with setenv.add-response-header, so there is no bug here (that is what this was all about).

If you want a new module "mod_precompressed_static_file", just go ahead and write it, i won't. And no, i don't think it is trivial, and i probably won't even have the time to review it. If you want you can try writing a mod_magnet handler instead...

#18 Updated by lightning almost 4 years ago

There is no standard conforming way of delivering .svgz, as not all http clients have to support compression.

They don't have to support transfer compression but if the client requests svgz (or any other compressed file) that it is not capable of handling that is not an issue for the server. Most browsers have a method of dealing with content that they cannot display as long as the headers correctly reflect what is being sent. I'm sure someone of your capabilities does not need me to explain the difference.

I don't care that you can trigger bad behaviour with setenv.add-response-header,

As I have already indicated, neither do I so why do you reiterate this every time you reply?!
My problem is that I have to use this to work around the incorrect headers that lighttpd is sending.

I did not clearly understand the root of the problem when I posted the bug but thanks to icy and Olef we now have a clear description of what is happening and it is wrong.

svgz does not get sent with the correct Content-Encoding header.

</quote>
for content stored in a compressed format on the server (e.g. with the file extension "svgz"), the server must use the "Content-Encoding: gzip" or "Content-Encoding: deflate" headers as appropriate.
</quote>

Can it be any clearer than that?

Since you seem entrenched in your position that lighttpd will remain defective and not standards conformant as it claims to be, I will leave the status incorrectly set as INVALID, but don't tell me this is not a bug. It is.

Thanks for clarifying your position, and thanks to icy and Olef for helping me understand exactly what was happening.

#19 Updated by stbuehler almost 4 years ago

lightning wrote:

</quote>
for content stored in a compressed format on the server (e.g. with the file extension "svgz"), the server must use the "Content-Encoding: gzip" or "Content-Encoding: deflate" headers as appropriate.
</quote>

Can it be any clearer than that?

I don't see that anywhere in a http rfc. There is one mechanism to describe the content type of files, and it is content-type. serving precompressed files with content-encoding is an optimization, and not supporting it is certainly not a bug.

#20 Updated by Olaf-van-der-Spek almost 4 years ago

It's probably a feature request to set more headers than just Content-Type based on file extension. Content-Encoding, but Content-Language and others come to mind.

#21 Updated by nitrox almost 4 years ago

Your problem is basically that lighty should perform an additional check if the file is available. For static files this can easily be done with a small lua-script, which checks if a file is present and sets the header accordingly. If not, it produces a 404 without content headers.

That content-encoding.lua looks like:

attr = lighty.stat(lighty.env["physical.path"])
if (attr) then
  lighty.header["Content-Encoding"] = "x-gzip" 
  else
   return 404
end

$HTTP["url"] =~ "\.svgz$" { magnet.attract-physical-path-to = ( "/path/to/content-encoding.lua" }

If a file needs more then content-type set to make a browser happy, its already faulty imho. Chrome, Safari, Opera, fx - they all handle this kind of files differently and there´s a list of bugs filed already. Please check yourself if this quick hack makes every browser happy or breaks others now.

I agree with stbuehler, that svgz needs an encoding too is not a lighty bug.

#22 Updated by nitrox almost 4 years ago

Olaf-van-der-Spek wrote:

It's probably a feature request to set more headers than just Content-Type based on file extension. Content-Encoding, but Content-Language and others come to mind.

I might agree. For upcoming lighty2 we should consider this and make it easier to include, but i doubt this features goes into 1.x. For static files we have lua, for dynamic stuff you´d need to set it yourself.

#23 Updated by icy almost 4 years ago

nitrox wrote:

For upcoming lighty2 we should consider this and make it easier to include

No "should", already possible :)

Also available in: Atom