Project

General

Profile

Feature #2474

Option to map send-file file-not-found error to normal 404

Added by Olaf-van-der-Spek almost 5 years ago. Updated 5 months ago.

Status:
Fixed
Priority:
Normal
Assignee:
-
Category:
mod_fastcgi
Target version:
Start date:
2013-02-13
Due date:
% Done:

100%

Estimated time:
Missing in 1.5.x:
No

Description

I've got tons of errors like this (don't ask):
2013-02-13 19:49:02: (mod_fastcgi.c.2641) send-file error: couldn't get stat_cache entry for: /srv/xwis.net/screenshots/619420_t.jpg

Is there an option to map those to normal 404s without logging?

fastcgi-404-always.diff (2.48 KB) fastcgi-404-always.diff patch: always sends 404 if file is not present bert, 2013-09-16 14:02
fastcgi-404-option.diff (4.48 KB) fastcgi-404-option.diff patch: config option to send 404 if file is not present bert, 2013-09-16 14:03

Associated revisions

Revision 3088 (diff)
Added by stbuehler over 1 year ago

[mod_fastcgi] 404 for X-Sendfile file not found (fixes #2474)

(slightly modified from patch by bert)

From: Glenn Strauss <>

Revision c80ae9b2 (diff)
Added by gstrauss over 1 year ago

[mod_fastcgi] 404 for X-Sendfile file not found (fixes #2474)

(slightly modified from patch by bert)

From: Glenn Strauss <>

git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@3088 152afb58-edef-0310-8abb-c4023f1b3aa9

Revision ddf33956 (diff)
Added by gstrauss 4 months ago

[core] server.error_handler_404 X-Sendfile ENOENT (#2474)

Better handling if server.error_handler_404 is a dynamic handler which
returns X-Sendfile pointing to a file which does not exist

(server.error_handler_404 historically did not reset con->file_started,
and for mod_fastcgi, an X-Sendfile failure in the error handler would
result in an empty response body.)

x-ref:
"Option to map send-file file-not-found error to normal 404"
https://redmine.lighttpd.net/issues/2474

History

#1

Updated by darix almost 5 years ago

call stat in your app before setting the x-sendfile header?

#2

Updated by Olaf-van-der-Spek almost 5 years ago

I'd prefer not to. Besides, that app might not have access to the file or even run on another box.

#3

Updated by stbuehler about 4 years ago

  • Target version changed from 1.4.33 to 1.4.34
#4

Updated by bert about 4 years ago

Attached are two patches to do this.

One sends a 404 instead of 502 if the file turns out to be missing when using X-Sendfile or X-Sendfile2.

The second patch introduces an option ("allow-x-send-file-404", a boolean) to control this behavior. The default is to keep the current mod_fastcgi behavior and send a 502.

#5

Updated by darix almost 4 years ago

  • Target version changed from 1.4.34 to 1.4.35
#6

Updated by stbuehler over 3 years ago

  • Target version changed from 1.4.35 to 1.4.36
#7

Updated by stbuehler over 2 years ago

  • Target version changed from 1.4.36 to 1.4.37
#8

Updated by stbuehler about 2 years ago

  • Target version changed from 1.4.37 to 1.4.38
#9

Updated by stbuehler about 2 years ago

  • Target version changed from 1.4.38 to 1.4.39
#10

Updated by stbuehler almost 2 years ago

  • Target version changed from 1.4.39 to 1.4.40
#11

Updated by gstrauss almost 2 years ago

If a FastCGI application returned X-Sendfile (or related) and the file does not exist on the local server running lighttpd, then 404 Not Found does seem like a valid error response. 502 Bad Gateway imparts slightly less information, but if a backend FastCGI server is generating paths and is unable to validate existence of a file, there is little difference between 502 and 404.

stbuehler: would you consider this if resubmitted as a pull request?

#12

Updated by stbuehler almost 2 years ago

I'm fine with replacing 502 with 404 in this case, I don't see a good reason to add an option for it; adding an option just increases complexity.

I just never got around reviewing it, but I appreciate any help with this.

#13

Updated by gstrauss almost 2 years ago

https://github.com/lighttpd/lighttpd1.4/pull/19

I slightly modified Olaf's patch to retain the error trace when stat_cache_get_entry() fails. If it should be removed for X-Sendfile, then it should also be removed for X-Sendfile2

#14

Updated by stbuehler over 1 year ago

  • Status changed from Patch Pending to Fixed
  • % Done changed from 0 to 100

Applied in changeset r3088.

#15

Updated by Olaf-van-der-Spek 5 months ago

I slightly modified Olaf's patch

Patch wasn't mine ;)

Just updated to 1.4.45, now I see this in error.log:
2017-06-17 14:35:05: (http-header-glue.c.539) not a regular file: /ra2/screenshots -> /srv/xwis.net/screenshots/10311.bin

And I see this in the response body:
Expires: 31-Dec-2020 12:00:00 GMT
Last-Modified: 01 Jan 2000 12:00:00 GMT
Content-Type: image/jpeg
X-Sendfile: /srv/xwis.net/screenshots/10311.bin

Isn't a non-existent file supposed to return a clean 404 without log entry in error.log?

#16

Updated by gstrauss 5 months ago

If a backend is returning X-Sendfile, which is a privilege that must be enabled by the admin, why wouldn't an error be desired if there was a failure to send the requested file from the backend?

Would you provide more details as to what your request is, and what is handling it, that you see X-Sendfile in the response? All code paths should go through http_response_write_header() which should strip out X-Sendfile before sending response.

#17

Updated by Olaf-van-der-Spek 5 months ago

I'm confused, isn't the whole point of this feature request not to log the error?

Anyway, some strace:

read(40, "\1\6\0\1\0\231\0\0Expires: 31-Dec-2020 12:00:00 GMT\r\nLast-Modified: 01 Jan 2000 12:00:00 GMT\r\nContent-Type: image/jpeg\r\nX-Sendfile: /srv/xwis.net/screenshots/10311.bin\r\n\r\n\1\6\0\1\0\0\0\0\1\3\0\1\0\0\0\0\0\0\0\0\0\0\0\0", 1087) = 185
stat("/srv/xwis.net/screenshots/10311.bin", 0x7fffdc94b700) = -1 ENOENT (No such file or directory)
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2945, ...}) = 0
write(7, "2017-06-17 16:26:32: (http-header-glue.c.539) not a regular file: /ra2/screenshots -> /srv/xwis.net/screenshots/10311.bin \n", 123) = 123
...
writev(39, [{"HTTP/1.1 200 OK\r\nContent-Length: 153\r\nDate: Sat, 17 Jun 2017 14:26:32 GMT\r\nServer: lighttpd/1.4.45\r\n\r\n", 102}, {"Expires: 31-Dec-2020 12:00:00 GMT\r\nLast-Modified: 01 Jan 2000 12:00:00 GMT\r\nContent-Type: image/jpeg\r\nX-Sendfile: /srv/xwis.net/screenshots/10311.bin\r\n\r\n", 153}], 2) = 255

After the file isn't found, it seems to call the backend again (it's the 404 handler too)..

#18

Updated by gstrauss 5 months ago

I'm confused, isn't the whole point of this feature request not to log the error?

Just because it was requested does not mean it is a good idea. I suggested above that the opposite, providing error trace, is more in line with what people might expect. You have provided no justification for why this might be a good idea or not. "I've got tons of errors like this (don't ask):" is not very convincing.

The title of this request is "Option to map send-file file-not-found error to normal 404" and that is what the patch I committed does. In your description, you asked "Is there an option to map those to normal 404s without logging?". The answer to that question is: no, there is not an option to do that. Mapping senf-file file-not-found error to 404 allows the 404 error handler to respond to these errors, which is the feature that was implemented.

After the file isn't found, it seems to call the backend again (it's the 404 handler too)..

This sounds like the source of your overall problem. Your 404 handler ought to be simpler and more robust than your default handlers.

[edit: the following sentence is incorrect; see further posts below]
For server.error-handler-404, the historical behavior is to send 200 OK and the verbatim output of running the script, which in your case includes a set of headers including X-Sendfile. Look closely at the writev() you provided and look for the \r\n\r\n which separates response headers from response body. This is another reason why your server.error-handler-404 should be different from your other handlers.

You might want to look at the more sane (IMHO) server.error-handler directive (lighttpd 1.4.40), which, unlike server.error-handler-404, does not force a 200 OK response.

#19

Updated by Olaf-van-der-Spek 5 months ago

gstrauss wrote:

I'm confused, isn't the whole point of this feature request not to log the error?

Just because it was requested does not mean it is a good idea.

True

I suggested above that the opposite, providing error trace, is more in line with what people might expect. You have provided no justification for why this might be a good idea or not. "I've got tons of errors like this (don't ask):" is not very convincing.

The title of this request is "Option to map send-file file-not-found error to normal 404" and that is what the patch I committed does.

Normal 404s don't cause an entry in error.log do they?

In your description, you asked "Is there an option to map those to normal 404s without logging?". The answer to that question is: no, there is not an option to do that. Mapping senf-file file-not-found error to 404 allows the 404 error handler to respond to these errors, which is the feature that was implemented.

After the file isn't found, it seems to call the backend again (it's the 404 handler too)..

This sounds like the source of your overall problem. Your 404 handler ought to be simpler and more robust than your default handlers.

In what way are my handlers not robust?

For server.error-handler-404, the historical behavior is to send 200 OK and the verbatim output of running the script, which in your case includes a set of headers including X-Sendfile.

That's not what 1.4.39 appears to be doing.

Look closely at the writev() you provided and look for the \r\n\r\n which separates response headers from response body. This is another reason why your server.error-handler-404 should be different from your other handlers.

You might want to look at the more sane (IMHO) server.error-handler directive (lighttpd 1.4.40), which, unlike server.error-handler-404, does not force a 200 OK response.

I'll have a look.

#20

Updated by gstrauss 5 months ago

For server.error-handler-404, the historical behavior is to send 200 OK and the verbatim output of running the script, which in your case includes a set of headers including X-Sendfile.

That's not what 1.4.39 appears to be doing.

I took a look through the history and you are correct. lighttpd 1.4.39 would return an empty page if server.error_handler_404 pointed to a FastCGI which returned X-Sendfile to a non-existent file. lighttpd 1.4.40 incorrectly returns the entire response from the backend if server.error_handler_404 is a FastCGI program which happens to return X-Sendfile pointing to a non-existent file.

Also available in: Atom