Project

General

Profile

Feature #2997

closed

mod_auth: Rate limit failures

Added by helmut 10 months ago. Updated 9 months ago.

Status:
Invalid
Priority:
Normal
Category:
mod_auth
Target version:
ASK QUESTIONS IN Forums:

Description

It would be nice if lighttpd provided a means to rate limit authentication failures.

Rationale: Why not use existing solutions such as fail2ban?
fail2ban works by inspecting a log acquired from the relevant service. However, lighttpd does not emit the relevant information in the log. A digest authentication request can be rejected due to a missing/stale nonce or due to an actual authentication failure. The relevant log entry does not distinguish these situations. However, rejecting digest authentication attempts due to missing nonces is quite common when using stateless clients. As such, limiting the number of 401 returns is not a useful measure. For this reason, any authentication failure rate limiting requires more support from lighttpd. In principle, it could be implemeted through PAM, SASL or LDAP, but doing so is quite heavy weight.

RFC-PATCH: Add new configuration items to provide such rate limiting at mod_auth

I know that your first response to my idea was not very encouraging. I'm posting the patch anyway, because we've done the work and I can share the resulting patch for others to use. It is copyright Intenta GmbH and suitably BSD-3 licensed as the rest of lighttpd. The primary author of the patch is Benjamin Wozniak.

Please comment on the approach taken. If you reject the feature, that's fine. This report mainly documents the need for this feature and a possible solution.


Files


Related issues

Related to Feature #2976: Digest auth nonces are not validatedFixedActions
#1

Updated by gstrauss 10 months ago

  • Status changed from New to Need Feedback
The patch does not look ready for production, and I would recommend against its use. After reviewing for only a few mins, I see that:
  • It may result in a tight loop of auth failures, eating CPU and bandwidth as the client retries and the server keeps rejecting.
  • Too broad. It is trivially DoS'able to reject all auth for the HTTP Auth realm
    - It does not limit the counts per username
    - It does not limit the block per IP address
    - (and it clears the consecutive failure count upon any successful auth)
  • The feature request talks about digests and nonces, but is also (partially?) applied to Basic auth.

Regarding CLOCK_MONOTONIC, if a server has poor time keeping on the level of a second or more, then that is a separate problem to be solved with NTP.
If Mr. Wozniak implemented this for daylight-saving time switches twice a year, then he might simply do a graceful restart once a year when time "moves backwards", or, in the current patch implementation, reset the counters with a single successful auth. (It is also trivially detectable if srv->cur_ts is earlier than the last rejected failure, if the time of last failure was saved, too.)

What are the goals of the patch? What is it actually trying to do or protect against?

Rationale: Why not use existing solutions such as fail2ban?
fail2ban works by inspecting a log acquired from the relevant service. However, lighttpd does not emit the relevant information in the log. A digest authentication request can be rejected due to a missing/stale nonce or due to an actual authentication failure. The relevant log entry does not distinguish these situations.

Why did the patch not focus on adding such desired information to enable use of fail2ban? mod_accesslog.c can log the 401 HTTP status. If additional pieces of information are required in the log, then I might consider a patch where mod_auth adds those pieces of information to the request environment, which can be logged in mod_accesslog.

#2

Updated by gstrauss 10 months ago

  • Related to Feature #2976: Digest auth nonces are not validated added
#3

Updated by gstrauss 10 months ago

BTW, using lighttpd mod_accesslog today, you can already log the "Authorization" request header along with the IP address of client and HTTP status code of the response. If you use HTTP Digest auth and log the Authentication header, then a simple post-processing could determine whether or not the request was sent with an Authentication header and a nonce. Similarly, you could log the "WWW-Authenticate" header and look for stale=true.

#4

Updated by gstrauss 10 months ago

Here's an untested sample patch against the HEAD of my development which prints the IP address of failed HTTP Digest attempts in the case of 401 Unauthorized response, except if Authorization header is not provided or if the nonce is stale. (In the next version of lighttpd, some interfaces may be changing, but you should get the picture and can change log_error() to log_error_write() along with some arguments)

diff --git a/src/mod_auth.c b/src/mod_auth.c
index cf4754d0..221621a1 100644
--- a/src/mod_auth.c
+++ b/src/mod_auth.c
@@ -976,7 +976,7 @@ static handler_t mod_auth_check_digest(connection *con, void *p_d, const struct
        }

        if (NULL == vb) {
-               return mod_auth_send_401_unauthorized_digest(con, require, 0);
+               return mod_auth_send_401_unauthorized_digest(con, require, -1);
        }

        if (!buffer_eq_icase_ssn(vb->ptr, CONST_STR_LEN("Digest "))) {
@@ -1187,6 +1187,10 @@ static handler_t mod_auth_check_digest(connection *con, void *p_d, const struct

 static handler_t mod_auth_send_401_unauthorized_digest(connection *con, const struct http_auth_require_t *require, int nonce_stale) {
        buffer * const tb = con->srv->tmp_buf;
+       if (0 == nonce_stale)
+               log_error(con->conf.errh, __FILE__, __LINE__,
+                 "digest: auth failed (IP: %s)", con->dst_addr_buf->ptr);
+       if (nonce_stale < 0) nonce_stale = 0; /*(no Authorization request header)*/
        mod_auth_digest_www_authenticate(tb, log_epoch_secs, require, nonce_stale);
        http_header_response_set(con, HTTP_HEADER_OTHER, CONST_STR_LEN("WWW-Authenticate"), CONST_BUF_LEN(tb));

Given the simplicity of the above, which should be easy to plug into fail2ban (possibly with slight modification to the error message), I do not see much benefit to any further work on the patch proposed by Mr. Wozniak. In the future, please consider opening a discussion in the lighttpd Development forum before posting to the issue tracker. Thank you.

#5

Updated by gstrauss 9 months ago

What are the goals of the patch? What is it actually trying to do or protect against?

Rationale: Why not use existing solutions such as fail2ban?
fail2ban works by inspecting a log acquired from the relevant service. However, lighttpd does not emit the relevant information in the log. A digest authentication request can be rejected due to a missing/stale nonce or due to an actual authentication failure. The relevant log entry does not distinguish these situations.

Why did the patch not focus on adding such desired information to enable use of fail2ban? mod_accesslog.c can log the 401 HTTP status. If additional pieces of information are required in the log, then I might consider a patch where mod_auth adds those pieces of information to the request environment, which can be logged in mod_accesslog.

.

Still awaiting feedback here. I may soon mark this issue invalid and suggest a discussion in the forums instead.

#6

Updated by helmut 9 months ago

Thank you for looking into the feature request and the patch.

I somehow missed the forum as the appropriate communication medium for RFC patches and will use it next time. Should we move the discussion there and close this ticket or continue here?

You asked about the goals of the patch and you observed that it is trivially DoSable. That is correct. In our application DoS is considered unprotectable. The patch defends against password brute force however and thus makes unauthorized access harder at the cost of making DoS simpler. That can be a reasonable trade-off. In the particular environment we are targeting, ip spoofing needs to be considered, so making the blocks by ip address would have been a trivial circumvention technique. Nevertheless, I see that limiting such blocks per ip would be useful for others.

You asked about why we did not focus on improving the integration with fail2ban. Compared to lighttpd, fail2ban is quite heavy in resource consumption and thus unsuitable for our application. We'd have to reimplement a stripped down version of fail2ban.
Plugging this functionality into lighttpd seemed like the easiest option to us and the size of the patch indicates that our expectation was reasonable.

When submitting it, I was aware that it was solving a niche use-case and I was kinda expecting you to reject it as is. I did see two benefits from submitting it anyway:
1. Document the use case and why it could make sense to have lighttpd support it.
2. Make a simple solution available for others.

Given that you seem to fundamentally disagree with adding brute force defense to lighttpd itself (as opposed to enabling brute force defense in another layer such as fail2ban), I suggest closig the ticket as invalid. Thank you for considering it anyway and presenting your reasons.

#7

Updated by gstrauss 9 months ago

  • Status changed from Need Feedback to Invalid

When submitting it, I was aware that it was solving a niche use-case and I was kinda expecting you to reject it as is. I did see two benefits from submitting it anyway:
1. Document the use case and why it could make sense to have lighttpd support it.
2. Make a simple solution available for others.

Some reasons that this belongs in the forums:
  • trivially DoS'able ==> not acceptable for general use
  • niche use-case (not disclosed in original post)
  • it was not clearly communicated why you thought fail2ban was not an option in your niche use-case (please review the original post)
  • similarly, your niche use-case does not implement common mitigations that would be useful to the general population.
  • even rejecting common mitigations such as IP blocking, the patch omits the very obvious mitigation of rate limiting per username.
    (You might be attempting to protect a single account, but have not detailed that. In such a case, rate limiting per username would not help your use-case)

I do not recommend production use of this patch to any general population.

Should we move the discussion there and close this ticket or continue here?

Yes, please open a discussion in the Development forum and link it here if you would like to continue to the discussion, disclosing more information about what you are trying to achieve and why common mitigations do not apply. There may be changes you can make to your architecture to better protect the device. Given what you have described so far, you might consider rate limiting all unauthorized requests, in which case your custom patch might be to sleep 1 second ever time WWW-Authenticate needs to be sent. While that would pause the entire server, and is not recommended, it might be the simplest one-line patch for your niche use-case. If you are trying to protect a single account, perhaps you should allow the account name to be changed, e.g. from something other than 'admin'. Then, rate limiting per username might be more applicable.

#8

Updated by gstrauss 9 months ago

helmut: I believe that your statement about fail2ban is incorrect.

Rationale: Why not use existing solutions such as fail2ban?
fail2ban works by inspecting a log acquired from the relevant service. However, lighttpd does not emit the relevant information in the log. A digest authentication request can be rejected due to a missing/stale nonce or due to an actual authentication failure. The relevant log entry does not distinguish these situations.

I took a look at fail2ban filter config/filter.d/lighttpd-auth.conf and it (correctly) picks up only the IP for password comparison failure for Basic and Digest, not other failures. It does not flag missing or stale nonce. lighttpd does emit relevant information in the error log for this to work. Please detail why you think this is not the case.

When submitting it, I was aware that it was solving a niche use-case and I was kinda expecting you to reject it as is. I did see two benefits from submitting it anyway:
1. Document the use case and why it could make sense to have lighttpd support it.
2. Make a simple solution available for others.

BTW, another limiting assumption of Mr. Wozniak's patch is that it assumes no valid account. Given a valid account, even with Mr. Wozniak's patch, it is possible to brute-force attack another account (e.g. 'admin') simply by periodically logging in with the valid account in order to reset counters in Mr. Wozniak's patch.

==> The patch is so limited in its niche-use case that I strongly recommend against anybody using it, including you.

Further discussion in https://redmine.lighttpd.net/boards/3/topics/8885

#9

Updated by helmut 9 months ago

I stand corrected. I was looking at the access log, but the error log contains everything that is needed for fail2ban to work correctly.

Also available in: Atom