Lighttpd

No need to include the core.

The assertion is an explicit assertion in lighttpd.
configfile-glue.c.273: assertion failed: sizeof(matches) >= srv>config_context->used

This is the code leading to configfile-glue.c line 273

    unsigned char matches[4096];   /*directives matches (4k is way too many!)*/
    unsigned short contexts[4096]; /*conditions matches (4k is way too many!)*/
    uint32_t n = 0;
    int rc = 1; /* default is success */
    force_assert(sizeof(matches) >= srv->config_context->used);

The code sets an arbitrary limit of 4096 conditions to set an upper bound on stack usage for this routine.

While you can safely double the limit, such a large number of conditions is not expected, and the memory usage of lighttpd per connection is likely much larger than with expected usage of a much smaller set of conditions.

All that is to say that there are probably better ways to achieve what you are trying to do.

• One option is to combine the regexes into a large regex with alternations (|)
• Another option is to use mod_magnet and check all those regexes in custom lua code.

Looking at spamreferrers.conf, I would recommend the first option. Doing so should result in a measurable reduction in memory usage in lighttpd per connection, as well as a performance increase going through the regex engine once rather than 6948 times (spamreferrers.conf)

While you could do similar for badbotipv4.conf (9985 lines), here I would recommend the second option to implement the policy comparison in custom lua code and run with mod_magnet

Creating two tables in lua, one for referrer and another for IPs, and then look up the referrer and IP. Doing so would likely be much faster than what you are doing now. It appears that you are interested in exact string matches instead of regexes.

I'll think further about your use case and may post more later.

[edit]
update: concatenating referrers into a single string hits default limits for PCRE. While this could be done in a handful of PCRE expressions with slightly shorter (but still very long) strings, a better answer is lua tables.

Thank you for the explanation and suggestions. This is hosting some personal git repositories so performance was furthest from my mind. I lazily converted the lists and 1.4.55 just happened to accept them.

lighttpd 1.4.59 with some very large regexes (but not too long) accepted by PCRE library can push > 24000 requests per second in the same basic tests I used above, and using slightly more than 7 MB resident memory. By comparison, your original (ab)use of the lighttpd config syntax took 40x (4000%) the memory and was much slower. The combined regexes are 160x (16000%) faster.

I took the very large lists of referrers and IPs and batched them into multiple very long regexes (but not too long). I then used include "spamre" and include "botsre" in my lighttpd.conf. YMMV.

# (note: implementation details below assume no '|' in input data since we scan and split large string on '|')

perl -e '@a = <>; chomp @a; $str = join("|", map { quotemeta($_) } @a); while ($str ne "") { $pos = index($str, "|", 34000); if ($pos != -1) { $part = substr($str, 0, $pos); substr($str, 0, $pos+1, ""); } else { $part = $str; $str = ""; } print "\$HTTP[\"referer\"] =~ \"(?:\\.|^)(?i:", $part, ")\$\" { url.access-deny = ( \"\" ) }\n"; }' spamreferrers.list > spamre

perl -e '@a = <>; chomp @a; $str = join("|", map { quotemeta($_) } @a); while ($str ne "") { $pos = index($str, "|", 34000); if ($pos != -1) { $part = substr($str, 0, $pos); substr($str, 0, $pos+1, ""); } else { $part = $str; $str = ""; } print "\$HTTP[\"remote-ip\"] =~ \"^(?i:", $part, ")\$\" { url.access-deny = ( \"\" ) }\n"; }' badbotipv4.list > botsre

The combined regex alternations is the performance winner for quick solutions.

Native C structures could be much faster but are less flexible, as a lighttpd module would need to be written for this special purpose. (Not difficult, just not as easy for immediate use.)

For a flexible solution, I coded up some lua at AbsoLUAtion and linked reject-bad-actors.lua. Using an mcdb constant database for lookups, I was able to achieve > 14000 requests per second. Separating the data into a database allows it to be updated, or for lighttpd to query the updated data without the need to restart lighttpd (see comments in reject-bad-actors.lua). Using lua allows arbitrary policy extensions to be applied by the lua script writer, and can be maintained independently from lighttpd.

I posted a messy and long perl one-liner to mod_access documentation to provide sample code to parse https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/conf.d/globalblacklist.conf into lighttpd rules. Feedback appreciated.