Lighttpd

Lightspeed tests are failing abysmally, worse than without any kind of compression.

Would you provide some more details? What is your configuration? How are you running your tests? What is the command line?

running through fastcgi server (127.0.0.1:9000) PHP 7.4.3
the tests are through lighthouse (not lightspeed as stated sorry) in the browser (both edge and chrome)
without mod_deflate
!

!
with mod_deflate
!

!

This should be a benchmark of lighttpd, not PHP. You have not shared your complete config. Please make sure you're using PHP-FPM, not PHP as CGI.

lighttpd can serve many tens of thousands of requests per second for a static file of 10's of KB, even with mod_deflate.

That you are seeing multiple seconds suggests that this is many requests and a large size, but I'm not going to spend any more time on this guessing, so please provide more details on what you are measuring and how.

I am using PHP-FPM - here's the config

server.modules = ("mod_fastcgi", "mod_openssl", "mod_deflate", "mod_expire", "mod_access", "mod_compress", "mod_redirect", "mod_rewrite", "mod_setenv", "mod_auth", "mod_accesslog", "mod_maxminddb", "mod_magnet", "mod_alias", "mod_wstunnel", "mod_cgi")
accesslog.filename = "/var/log/lighttpd/access.log" 
server.errorlog             = "/var/log/lighttpd/error.log" 
server.pid-file             = "/var/run/lighttpd.pid" 
server.username             = "www-data" 
server.groupname            = "www-data" 
server.max-fds              = 2048
server.max-worker           = 8
server.max-keep-alive-idle = 4
server.max-keep-alive-requests = 4
server.max-read-idle = 30
server.max-write-idle = 120
server.event-handler        = "linux-sysepoll" 
server.network-backend      = "linux-sendfile" 
server.stat-cache-engine    = "disable" 
server.stream-response-body = 1
server.feature-flags = ( "server.h2proto" => "enable", "server.h2c" => "enable" )
static-file.etags           = "enable" 
mimetype.use-xattr          = "enable" 
maxminddb.activate = "enable" 
mimetype.assign   = (".svg" => "image/svg+xml", ".css" => "text/css", ".js" => "text/javascript")
maxminddb.db = "/var/lib/GeoIP/GeoLite2-Country.mmdb" #"/usr/share/GeoIP/GeoLite2-Country.mmdb" 
#maxminddb.env = ( "GEOIP_COUNTRY_CODE" => "country/iso_code", "GEOIP_COUNTRY_NAME" => "country/names/en" )

server.http-parseopts = (
  "header-strict"           => "enable",# default
  "host-strict"             => "enable",# default
  "host-normalize"          => "enable",# default
  "url-normalize-unreserved"=> "enable",# recommended highly
  "url-normalize-required"  => "enable",# recommended
  "url-ctrls-reject"        => "enable",# recommended
  "url-path-2f-decode"      => "enable",# recommended highly (unless breaks app)
  "url-path-dotseg-remove"  => "enable",# recommended highly (unless breaks app)
)

#dir-listing.activate        = "disable" 
index-file.names            = ( "index.php", "index.html", "index.lighttpd.html" )
url.access-deny             = ( "~", ".inc" )
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi", ".html" )
fastcgi.map-extensions = ( ".html" => ".php", ".php3" => ".php", ".htm" => ".php")

server.document-root = "/var/www/" 
server.port = 80
$SERVER["socket"] == "[::]:80" { }
ssl.pemfile = "/etc/lighttpd/server.pem" 
$SERVER["socket"] == ":443" { ssl.engine = "enable" }
$SERVER["socket"] == "[::]:443" { ssl.engine = "enable" }

$HTTP["scheme"] == "http" {
    $HTTP["remote-ip"] != "127.0.0.1" {
        $HTTP["remote-ip"] != "[::1]" {
            url.redirect = ("" => "https://${url.authority}${url.path}${qsa}")
        }
    }
}

url.rewrite-if-not-file = ( "^/(.)(\?.)?$" => "/index.html?fwd=$1" )
fastcgi.server = (
    ".html" => (
        "localhost" => (
            #"socket"                => "/var/run/lighttpd/php7-fpm.sock",
            "host"                  => "127.0.0.1",
            "port"                  => "9000",
            "broken-scriptfilename" => "enable" 
      )),
)

deflate.mimetypes = ("text/html", "text/plain", "text/css", "text/javascript", "text/xml", "image/svg+xml")
deflate.allowed-encodings = ( "brotli", "gzip", "deflate" )

What is the performance difference with maxminddb.activate = "disable" ?

server.max-fds = 2048

That's low for a busy server, but fine for most uses. Increase this if there are errors in the lighttpd error log which suggest that you ran out of fds. If you're doing a performance benchmark test, you may need to increase this value.

server.max-worker = 8

lighttpd is very fast. A single process is all that most people need, especially when PHP-FPM is the likely bottleneck. You should comment out server.max-worker unless it measurably improves performance.

server.max-keep-alive-requests = 4

That's way too low. The default in lighttpd, as well as in Apache is 100.

server.stat-cache-engine = "disable"

That's a huge performance hit and you should not use that on any high performance site. If some of your files are changing too quickly to be cachable, put those frequently-modified files behind PHP, and let the rest of the static files get the performance of the stat cache.

mimetype.use-xattr = "enable"

This should be disabled unless you actually use it. If you do not know what it is, you should disable it.

There are so many performance pessimizations in your configuration that I question the performance of the site before lighttpd 1.4.56.

gstrauss wrote in RE: mod_deflate speed:

What is the performance difference with maxminddb.activate = "disable" ?

With all else the same, not much - performance went to 29%

server.max-fds = 2048
server.max-worker = 8
server.max-keep-alive-requests = 4
server.stat-cache-engine = "disable"
mimetype.use-xattr = "enable"

Settings are now:
server.max-fds - commented out
server.max-worker - commented out
server.max-keep-alive-requests commented out
server.stat-cache-engine = "simple"
mimetype.use-xattr - commented out

when I re-run the test again:
1. With mod_deflate enabled

!!

2. With mod_deflate commented out

!!

So it does not seem to have had any effect.
PS: Most of the settings were inherited from previous admin as "optimisations" - since they did not seem to break the site, I have kept them there and only added / removed where necessary (most additions are actually not in the above config as they deal with re-writes which are being abandoned with the new site in development).

I have cleaned up the config but still get similar results (as in OP).
Note that I can actually further improve the performance without mod_deflate by compressing text output via PHP output buffering (I have that for html, css, svg and js files) but since it is a bit of a hack, that is why I wanted to compare mod_deflate performance in 1.4.56

Here's the cleaned up config:

server.modules = ("mod_fastcgi", "mod_openssl", "mod_deflate", "mod_expire", "mod_access", "mod_compress", "mod_redirect", "mod_rewrite", "mod_setenv", "mod_accesslog", "mod_maxminddb", "mod_magnet", "mod_alias")
accesslog.filename = "/var/log/lighttpd/access.log" 
server.errorlog             = "/var/log/lighttpd/error.log" 
server.pid-file             = "/var/run/lighttpd.pid" 
server.username             = "www-data" 
server.groupname            = "www-data" 
server.bind            = "172.78.153.71" 
server.event-handler        = "linux-sysepoll" 
server.network-backend      = "linux-sendfile" 
server.feature-flags = ( "server.h2proto" => "enable", "server.h2c" => "enable" )

mimetype.assign   = (".svg" => "image/svg+xml", ".css" => "text/css", ".js" => "text/javascript")

server.http-parseopts = (
  "header-strict"           => "enable",# default
  "host-strict"             => "enable",# default
  "host-normalize"          => "enable",# default
  "url-normalize-unreserved"=> "enable",# recommended highly
  "url-normalize-required"  => "enable",# recommended
  "url-ctrls-reject"        => "enable",# recommended
  "url-path-2f-decode"      => "enable",# recommended highly (unless breaks app)
  "url-path-dotseg-remove"  => "enable",# recommended highly (unless breaks app)
 )

index-file.names            = ( "index.php", "index.html", "index.lighttpd.html" )
url.access-deny             = ( "~", ".inc" )
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi", ".html" )
fastcgi.map-extensions = ( ".html" => ".php", ".php3" => ".php", ".htm" => ".php")

server.document-root = "/var/www/" 
server.port = 80
$SERVER["socket"] == "[::]:80" { }
ssl.pemfile = "/etc/lighttpd/server.pem" 
$SERVER["socket"] == ":443" { ssl.engine = "enable" }
$SERVER["socket"] == "[::]:443" { ssl.engine = "enable" }

$HTTP["scheme"] == "http" {
    $HTTP["remote-ip"] != "127.0.0.1" {
        $HTTP["remote-ip"] != "[::1]" {
            url.redirect = ("" => "https://${url.authority}${url.path}${qsa}")
        }
    }
}
url.rewrite-if-not-file = ( "^/(.)(\?.)?$" => "/index.html?fwd=$1" )
fastcgi.server = (
    ".html" => (
        "localhost" => (
            #"socket"                => "/var/run/lighttpd/php7-fpm.sock",
            "host"                  => "127.0.0.1",
            "port"                  => "9000",
            "broken-scriptfilename" => "enable" 
      )),
)

deflate.mimetypes = ("text/html", "text/plain", "text/css", "text/javascript", "text/xml", "image/svg+xml")
deflate.allowed-encodings = ( "brotli", "gzip", "deflate" )

I wrote:

There are so many performance pessimizations in your configuration that I question the performance of the site before lighttpd 1.4.56.

You seem to have missed the hint. Before suggesting that the problem is with lighttpd 1.4.56, you should run the same tests with what you inherited.

PS: Most of the settings were inherited from previous admin as "optimisations" - since they did not seem to break the site, I have kept them there and only added / removed where necessary (most additions are actually not in the above config as they deal with re-writes which are being abandoned with the new site in development).

It is unlikely that the person before you tested the "performance tweaks" that he or she made.

Similarly, your site is its own thing. Have you run lighthouse tests against lighttpd when the site is a simple "hello world" PHP program?

gstrauss wrote in RE: mod_deflate speed:

I wrote:

There are so many performance pessimizations in your configuration that I question the performance of the site before lighttpd 1.4.56.

You seem to have missed the hint. Before suggesting that the problem is with lighttpd 1.4.56, you should run the same tests with what you inherited.

I took the hint and purged most of the settings from the configuration file, clearly I went the wrong way. Further hints?

PS: Most of the settings were inherited from previous admin as "optimisations" - since they did not seem to break the site, I have kept them there and only added / removed where necessary (most additions are actually not in the above config as they deal with re-writes which are being abandoned with the new site in development).

It is unlikely that the person before you tested the "performance tweaks" that he or she made.

Similarly, your site is its own thing. Have you run lighthouse tests against lighttpd when the site is a simple "hello world" PHP program?

I actually did a hello world with a very simple page that has both css and js links and ran it with my pared down config- site performance in lighthouse with mod_deflate is way lower than without any compression, I could even see the load on the server using htop as I loaded the page with mod_deflate enabled.

Right - I did some more fiddling this morning after a good night's sleep.
I believe mod_deflate is sequentially compressing resources (blocking while compressing), not to mention that the compression is also discernibly slower that if done by PHP. It may be that I could tune the compression level for brotli to improve speed, but if my it is sequentially processing then there would still be a hit.

As for optimising the config - I am open to suggestions, suffice to say I used a very basic, un-optimised configuration for this morning's test, and as yesterday, the lighthouse performance results for mod_deflate in its current iteration are multiple times slower than without any compression (and worse with PHP compression).