Lighttpd

call stat in your app before setting the x-sendfile header?

I'd prefer not to. Besides, that app might not have access to the file or even run on another box.

If a FastCGI application returned X-Sendfile (or related) and the file does not exist on the local server running lighttpd, then 404 Not Found does seem like a valid error response. 502 Bad Gateway imparts slightly less information, but if a backend FastCGI server is generating paths and is unable to validate existence of a file, there is little difference between 502 and 404.

@stbuehler: would you consider this if resubmitted as a pull request?

I'm fine with replacing 502 with 404 in this case, I don't see a good reason to add an option for it; adding an option just increases complexity.

I just never got around reviewing it, but I appreciate any help with this.

https://github.com/lighttpd/lighttpd1.4/pull/19

I slightly modified Olaf's patch to retain the error trace when stat_cache_get_entry() fails. If it should be removed for X-Sendfile, then it should also be removed for X-Sendfile2

I slightly modified Olaf's patch

Patch wasn't mine ;)

Just updated to 1.4.45, now I see this in error.log:
2017-06-17 14:35:05: (http-header-glue.c.539) not a regular file: /ra2/screenshots -> /srv/xwis.net/screenshots/10311.bin

And I see this in the response body:
Expires: 31-Dec-2020 12:00:00 GMT
Last-Modified: 01 Jan 2000 12:00:00 GMT
Content-Type: image/jpeg
X-Sendfile: /srv/xwis.net/screenshots/10311.bin

Isn't a non-existent file supposed to return a clean 404 without log entry in error.log?

If a backend is returning X-Sendfile, which is a privilege that must be enabled by the admin, why wouldn't an error be desired if there was a failure to send the requested file from the backend?

Would you provide more details as to what your request is, and what is handling it, that you see X-Sendfile in the response? All code paths should go through http_response_write_header() which should strip out X-Sendfile before sending response.

I'm confused, isn't the whole point of this feature request not to log the error?

Anyway, some strace:

read(40, "\1\6\0\1\0\231\0\0Expires: 31-Dec-2020 12:00:00 GMT\r\nLast-Modified: 01 Jan 2000 12:00:00 GMT\r\nContent-Type: image/jpeg\r\nX-Sendfile: /srv/xwis.net/screenshots/10311.bin\r\n\r\n\1\6\0\1\0\0\0\0\1\3\0\1\0\0\0\0\0\0\0\0\0\0\0\0", 1087) = 185
stat("/srv/xwis.net/screenshots/10311.bin", 0x7fffdc94b700) = -1 ENOENT (No such file or directory)
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2945, ...}) = 0
write(7, "2017-06-17 16:26:32: (http-header-glue.c.539) not a regular file: /ra2/screenshots -> /srv/xwis.net/screenshots/10311.bin \n", 123) = 123
...
writev(39, [{"HTTP/1.1 200 OK\r\nContent-Length: 153\r\nDate: Sat, 17 Jun 2017 14:26:32 GMT\r\nServer: lighttpd/1.4.45\r\n\r\n", 102}, {"Expires: 31-Dec-2020 12:00:00 GMT\r\nLast-Modified: 01 Jan 2000 12:00:00 GMT\r\nContent-Type: image/jpeg\r\nX-Sendfile: /srv/xwis.net/screenshots/10311.bin\r\n\r\n", 153}], 2) = 255

After the file isn't found, it seems to call the backend again (it's the 404 handler too)..

I'm confused, isn't the whole point of this feature request not to log the error?

Just because it was requested does not mean it is a good idea. I suggested above that the opposite, providing error trace, is more in line with what people might expect. You have provided no justification for why this might be a good idea or not. "I've got tons of errors like this (don't ask):" is not very convincing.

The title of this request is "Option to map send-file file-not-found error to normal 404" and that is what the patch I committed does. In your description, you asked "Is there an option to map those to normal 404s without logging?". The answer to that question is: no, there is not an option to do that. Mapping senf-file file-not-found error to 404 allows the 404 error handler to respond to these errors, which is the feature that was implemented.

After the file isn't found, it seems to call the backend again (it's the 404 handler too)..

This sounds like the source of your overall problem. Your 404 handler ought to be simpler and more robust than your default handlers.

[edit: the following sentence is incorrect; see further posts below]
For server.error-handler-404, the historical behavior is to send 200 OK and the verbatim output of running the script, which in your case includes a set of headers including X-Sendfile. Look closely at the writev() you provided and look for the \r\n\r\n which separates response headers from response body. This is another reason why your server.error-handler-404 should be different from your other handlers.

You might want to look at the more sane (IMHO) server.error-handler directive (lighttpd 1.4.40), which, unlike server.error-handler-404, does not force a 200 OK response.

gstrauss wrote:

I'm confused, isn't the whole point of this feature request not to log the error?

Just because it was requested does not mean it is a good idea.

True

I suggested above that the opposite, providing error trace, is more in line with what people might expect. You have provided no justification for why this might be a good idea or not. "I've got tons of errors like this (don't ask):" is not very convincing.

The title of this request is "Option to map send-file file-not-found error to normal 404" and that is what the patch I committed does.

Normal 404s don't cause an entry in error.log do they?

In your description, you asked "Is there an option to map those to normal 404s without logging?". The answer to that question is: no, there is not an option to do that. Mapping senf-file file-not-found error to 404 allows the 404 error handler to respond to these errors, which is the feature that was implemented.

After the file isn't found, it seems to call the backend again (it's the 404 handler too)..

This sounds like the source of your overall problem. Your 404 handler ought to be simpler and more robust than your default handlers.

In what way are my handlers not robust?

For server.error-handler-404, the historical behavior is to send 200 OK and the verbatim output of running the script, which in your case includes a set of headers including X-Sendfile.

That's not what 1.4.39 appears to be doing.

Look closely at the writev() you provided and look for the \r\n\r\n which separates response headers from response body. This is another reason why your server.error-handler-404 should be different from your other handlers.

You might want to look at the more sane (IMHO) server.error-handler directive (lighttpd 1.4.40), which, unlike server.error-handler-404, does not force a 200 OK response.

I'll have a look.

For server.error-handler-404, the historical behavior is to send 200 OK and the verbatim output of running the script, which in your case includes a set of headers including X-Sendfile.

That's not what 1.4.39 appears to be doing.

I took a look through the history and you are correct. lighttpd 1.4.39 would return an empty page if server.error_handler_404 pointed to a FastCGI which returned X-Sendfile to a non-existent file. lighttpd 1.4.40 incorrectly returns the entire response from the backend if server.error_handler_404 is a FastCGI program which happens to return X-Sendfile pointing to a non-existent file.