Lighttpd

"The error 404 handler didn't previously catch 403 responses generated by url.access-deny in this way."

You are wrong. You could of course say that it is a bug that the 404-handler also handles 403, but the one who coded it wanted it that way (and if you think it is a bug, please open a new ticket with only that - but i guess you will get a wontfix for that, as it is a stable release).

And the changed behaviour was a bug fix, and the current behaviour is wanted.

I wrote "The error 404 handler didn't previously catch 403 responses generated by url.access-deny in this way." and you responded to this asserting "You are wrong."

Sorry but this behaviour has genuinely changed. The 404-handler did not previously catch 403 responses generated by url.access-deny in a `HOST["url"]` block (prior to revision 2080), and does catch them now. Here is an explicit demonstration of the difference:

$ HOST=127.0.0.1 /tmp/lighttpd-1.4.x-r2079/bin/lighttpd -f /tmp/lighttpd.conf
$ curl http://127.0.0.1:8080/forbidden/index.html 2>/dev/null \
  | grep -iq forbidden && echo forbidden || echo allowed; echo
forbidden

$ HOST=127.0.0.2 /tmp/lighttpd-1.4.x-r2080/bin/lighttpd -f /tmp/lighttpd.conf
$ curl http://127.0.0.2:8080/forbidden/index.html 2>/dev/null \
  | grep -iq forbidden && echo forbidden || echo allowed; echo
allowed

$ cat /tmp/lighttpd.conf
server.bind = env.HOST
server.port = 8080
server.errorlog = "/dev/null" 
server.modules = (
  "mod_access",
)
server.document-root = "/tmp/www" 
server.error-handler-404 = "/index.html" 
$HTTP["url"] =~ "^/forbidden/" {
  url.access-deny = ("")
}
$

I am aware that some other sources of 403 were previously caught by the 404-handler, such as those generated from a static file with no read permissions. These are unaffected by the change in revision 2080, of course.

It's certainly cleaner for the 404-handler to receive every 403 response rather than just some of them and I'm not going to argue that sending them to the 404-handler is wrong per se. However, this is a fairly major change in behaviour which could affect quite a few users. 404-handlers are still the route that the Rails and Django projects suggest for overlaying dynamic content on top of the static tree. For instance, Rails ships with the following default lighttpd.conf:

http://github.com/rails/rails/tree/master/railties/configs/lighttpd.conf

As a result, the change will bite Rails and Django users who have masked off access to areas of their sites in the intuitively obvious way, with url.access-deny inside a `$HTTP'URL'` block. This configuration will have worked up to and including 1.4.18, albeit unintentionally from your point of view! We have a large number of Rails users who've configured their lighttpd instances based on the default Rails lighttpd.conf, and who have masked off areas with url.access-deny in a `$HTTP'url'`-matching block. They weren't expecting the upgrade to 1.4.19 to open up all the areas of their sites to which access had been disabled.

So, given this is the intended behaviour, how should I correctly be advising these users to deny access to areas that would otherwise be covered by a server.error-handler-404? I'd suggest turning off the server.error-handler-404 inside the `$HTTPurl` block, but there doesn't appear to be a way to do this. (Setting it to an empty string isn't the same thing as unsetting it, and doesn't do the right thing.) If there's a canonical recipe, I can document it both for our users and on the lighttpd wiki page for error-handler-404, as I'm sure our users won't be the only people caught by this.

Ok, let me explain it (thx for not reopening this again):

Before 1.4.19, the error-handler-404 triggered on your forbidden url, but as the condition cache was not reset, that handler got forbidden too. That is, a part of your config (the condition block) was active when it shouldn't have been.
As the error-handler-404 is only a rewrite and must be treated like a normal request in any other way (except when it generates an error again), as you may have some conditional fastcgi/... handling, this was clearly a bug and was therefore fixed.

The recommended way is to use mod_magnet, but for the forbidden part you could setup a real error-handler-404, which displays a forbidden message (static document would be fine), or just use a document which is forbidden too (like "/forbidden/index.html").

"Before 1.4.19, the error-handler-404 triggered on your forbidden url, but as the condition cache was not reset, that handler got forbidden too. That is, a part of your config (the condition block) was active when it shouldn't have been."

Yes, okay, that's certainly not a valid path for the error-handler-404 to end up being skipped in older lighttpd.

I'll document a warning about this sort of configuration and guide our users in the direction of something like

  $HTTP["url"] =~ "^/forbidden" {
    url.access-deny = ("")
    server.error-handler-404 = "/forbidden/" 
  }

so that the attempt to access the 404-handler generates the expected 403 response. (Another possibility is to use mod_auth/mod_redirect to mask off areas of the site that aren't supposed to be accessible.)