Docs Performance » History » Revision 29
Revision 28 (gstrauss, 2021-02-04 16:08) → Revision 29/46 (gstrauss, 2021-07-16 04:51)
{{>toc}} h1. Performance Tuning Improvements . *Module: core* h4. important performance tuning rules {{>toc}} # *Prefer h2. Description - Performance Issues lighttpd defaults unless you have a reason to change a setting, and unless you test that changing the setting is beneficial to you.* # *Proper functionality is more optimized into varying directions. The most important than marginal increases in performance; a web server that does not function as intended direction is not useful. Do not sacrifice security or desired operational functioning for marginal performance improvements.* # *Performance tuning is not magic. performance. The recommended appoach is that one change be made at a time, that the change be tested and benchmarked, and that operation system has two major facilities to help lighttpd deliver its best performance. h2. HTTP Keep-Alive Disabling keep-alive might help your server if the change does not have a measurable and positive impact in real-world scenarios, that the change be reverted.* lighttpd is generally pretty snappy. Most of the following are micro-optimizations. No changes are required unless you have suffer from a specific performance issue that you must address. large (Please refer to [[Docs_ConfigurationOptions|configuration options]] for details about any number of the options listed below.) open file descriptors. h4. lighttpd configuration performance tuning (technical guidelines) * less is more (and is often simpler, too) ** rely on The defaults where possible to reduce unnecessary (duplicative) config processing (at runtime) to process configuration directives which were already set to for the default values server are: ** set config options in the global scope rather than repeating in sub-scopes. lighttpd optimizes configuration settings in the global scope and makes those settings the defaults ** TLS configuration can be set in the global scope and inherited by multiple @$SERVER["socket"]@ @ssl.pemfile <pre> server.max-keep-alive-requests = "..."@ @ssl.privkey 100 server.max-keep-alive-idle = "..."@ @$SERVER["socket"] == ":443" { ssl.engine 5 server.max-read-idle = "enable" }@ @$SERVER["socket"] == "[::]:443" { ssl.engine 60 server.max-write-idle = "enable" }@ 360 </pre> Handling 100 keep-alive requests in a row on a single connection, waiting 5 seconds ** list only the modules actually used and enabled in @server.modules@; comment out the others - each loaded module registers itself into lighttpd hooks and before an unused keep-alive connection gets a chance to dropped by lighttpd. If you handle each request, which is is unnecessary if several connections at once under a module is loaded but not otherwise configured to be used - @server.compat-module-load high load (let's assume 500 connections in parallel for 24h) you might run into the out-of-fd problem described below. :: <pre> server.max-keep-alive-requests = "disable"@ skips loading 4 server.max-keep-alive-idle = 4 </pre> It would release the default modules ([[Docs_ModIndexFile|mod_indexfile]], [[Docs_ModDirListing|mod_dirlisting]], [[Docs_ModStaticFile|mod_staticfile]]), connections earlier and you can then explicitly add one or more to @server.modules@ to use them would free file descriptors without a ** tweaks to remove optional functionality - @server.tag = ""@ skips sending "Server: lighttpd/1.4.xx" in responses; alternatively, use: @server.tag = "lighttpd"@ to hide detrimental performance loss. Disabling keep-alive completely is the lighttpd version - @server.range-requests = "disable"@ can be used last resort if all server responses you are small files, but otherwise it still short on file descriptors: :: <pre> server.max-keep-alive-requests = 0 </pre> h2. Event Handlers The first one is recommended to be left enabled ** review the default lighttpd config provided by your distro - configs provided by distros aim to be newbie friendly but can introduce complexity Event Handler which takes care of yet another config framework - configs provided by distros are often out-dated and then kept for historic compatibility, rather than current best practices - example: ~20 years ago some widely used versions notifying the server that one of Adobe Acrobat reader plugin PDF clients misbehaved with range requests. Unfortunately, the config setting connections is ready to disable range requests for PDFs has been cargo-culted into configs since then. Prefer to comment out send or remove: @$HTTP["url"] =~ "\.pdf$" { server.range-requests = "disable" }@ receive. ** @server.max-connections@ limits Every OS has at least the maximum number of simultaneous connections select() call which has some limitations. See [[lighttpd:Server_event-handlerDetails|server.event-handler]] for details on how to handle set an event handler in lighty. For more information on this topic take a look at http://www.kegel.com/c10k.html and also affects memory usage http://libev.schmorp.de/bench.html h2. Network Handlers The basic network interface for the connection cache - default is (about) 1365 which is oversized for all but platforms at the largest systems. Embedded systems might set @server.max-connections = 16@ or lower syscalls read() and ** @server.max-worker = 0@ should generally be left unset (or "0"), write(). Every modern OS provides its own syscall to help network servers transfer files as CPU bottlenecks are usually elsewhere ** @server.follow-symlink = "enable"@ (default) should be left enabled. fast as possible. If such restrictions are required, prefer you want to run send out a separate lighttp instance under a separate user account, and enforce more restrictive file access permissions. from the webserver, it doesn't make any sense ** @ssl.read-ahead = "disable"@ (default) is strongly recommended for slower, embedded systems which process TLS packets more slowly than network wire-speed. For faster systems, *test* if @ssl.read-ahead = "enable"@ improves performance (or not) ** prefer to configure [[Docs_ModExtForward|mod_extforward]] @extforward.hap-PROXY@ for lighttpd instances behind HAProxy or load balancers supporting copy the HAProxy PROXY protocol * minimize conditional processing (but not at the cost of proper functionality) ** more conditions means more config processing at runtime ** more conditions means more memory used by config per request ** avoid repeating conditions and its opposite by joining them file into if/else @<condition> { ... } else { ... }@ @<condition> { ... } else <condition> { ... } else { ... }@ ** sometimes it may take fewer config lines to set a config option once in the global scope and then, where necessary, webserver just to unset the option in write() it back into a small number of conditions rather than leaving the default socket in the global scope and enabling next step. sendfile() minimizes the config option work in many more conditions ** having no config conditions will be among the fastest configs to be processed, but config processing at runtime is fast application and is not typically pushes a bottleneck file directly * dynamic backends ([[Docs_ModProxy|mod_proxy]], [[Docs_ModFastCGI|mod_fastcgi]], [[Docs_ModSCGI|mod_scgi]], [[Docs_ModAJP13|mod_ajp13]], ...) into the network card (ideally). lighttpd supports all major platform-specific calls: |_.OS |_.Method |_.Config Value | ** prefer |all |write |write | |Unix |writev |writev | |Linux 2.4+ |sendfile |linux-sendfile | |Linux 2.6+ |sendfile64 |linux-sendfile | |Solaris |sendfilev |solaris-sendfilev| |FreeBSD |sendfile |freebsd-sendfile | The best backend is selected at compile time. In case you want to use unix domain sockets (instead of TCP sockets) for connections from lighttpd to backends running on the same host another backend set: ** lighttpd can listen on a unix domain socket (@server.bind <pre> server.network-backend = "/path/to/lighttpd.sock"@) and lighttpd [[Docs_ModProxy|mod_proxy]] "writev" </pre> You can act as a reverse-proxy to a find more information about network backend in: http://blog.lighttpd.net/articles/2005/11/11/optimizing-lighty-for-high-concurrent-large-file-downloads h2. Max Connections As lighttpd server. Use with [[Docs_ModExtForward|mod_extforward]] to preserve client remote address for is a single-threaded server, its main resource limit is the backend. * [[Docs_ModFastCGI|mod_fastcgi]]: use PHP-FPM number of file descriptors, which is set to manage PHP 1024 by default (on most systems). If you are running a high-traffic site you might want to increase this limit ** by setting: <pre> server.max-fds = 2048 </pre> This only works if lighttpd provides mechanisms for is started as root. Note that SELinux can prevent lighttpd to start up PHP backends, and that works well, but PHP-FPM is from setting the modern and recommended mechanism to manage PHP backends * [[Docs_ModRewrite|mod_rewrite]] and [[Docs_ModRedirect|mod_redirect]]: short-circuiting (when using a sequence maximum number of regexes) ** consider putting static open file matches (passed through unmodified) first, descriptions, at least on Fedora and using RHEL/EPEL5. To create a blank target SELinux module allowing lighttpd to indicate no modification set server.max-fds: <pre> ** consider using a blank match as a catch-all, rather than "^(.*)", which will still match all, but without the regex @url.rewrite-once = (@ @ # /usr/sbin/semodule -DB # service auditd restart # service lighttpd restart # grep lighttpd /var/log/audit/audit.log | audit2allow -M lighttpdmaxfds # semodule -i lighttpdmaxfds.pp # service lighttpd start Starting lighttpd: [ "^/static/|\.(?:css|jpg)$" => "",@ @ OK "" => "/index.php${url.path}${qsa}"@ @)@ ] * [[Docs_ModIndexFile|mod_indexfile]]: reduce the # /usr/sbin/semodule -B </pre> You can look for lighttpd's maximum number of entries open file descriptors in index-file.names, if [[Docs_ModIndexFile|mod_indexfile]] is enabled /proc: <pre> ** @index-file.names = ("index.html")@ as # cat /proc/`ps ax | grep lighttpd | grep -v grep | awk -F " " '{print $1}'`/limits |grep "Max open files" Max open files 2048 2048 files </pre> h2. Out-of-fd condition Since file descriptors are used for TCP/IP sockets, files and directories, a list of one or two entries rather than simple request for a list of, say, 10 differenent PHP page might result in using 3 file extensions descriptors: # the TCP/IP socket to the client * cache tuning ** stat_cache: default @server.stat_cache-engine = "simple"@ works well for typical usage # the TCP/IP and caches @stat()@ results for 1-2 seconds Test with @server.stat-cache-engine = "inotify"@ or @server.stat-cache-engine = "kqueue"@ for @stat()@ results Unix domain socket to be cached longer (16 seconds) the FastCGI process ** [[Docs_ModAuth|mod_auth]]: set @auth.cache = ("max-age" => "600")@ # the filehandle to cache passwords (default disabled), but acknowledge changes the file in the document root to your security posture check if enabling the cache. (since it exists If lighttpd 1.4.56) runs out of file descriptors, it will stop accepting new ** [[Docs_ModDeflate|mod_deflate]]: set @deflate.cache-dir@ connections for awhile to cache (and reuse) compressed static assets based on ETag (since lighttpd 1.4.56) use the existing file descriptors to handle the ** [[Docs_ModDirListing|mod_dirlisting]]: set @dir-listing.cache = ( ... )@ to configure caching currently-running requests. If more than 90% of generated directory listings (since lighttpd 1.4.60) * do not sacrifice security to save a few CPU cycles ** @server.http-parseopts*@ option defaults the file descriptors are recommended, and are very fast used then the handling of new ** disabling @server.http-parseopts*@ might save a few CPU cycles, but connections is an anti-pattern for secure configurations disabled. If it drops below 80% again new connections will ** @server.http-parseopts*@ options should be modified only when accepted again. Under some circumstances you will see <pre> ... accept() failed: Too many open files </pre> in the functionality needs to be tuned for proper site operation error log. This tells you there were too many new requests at once ** ETag response headers are used in HTTP/1.1 conditional caching. ETag response headers are also required for [[Docs_ModDeflate|mod_deflate]] and strongly recommended with [[Docs_ModWebDAV|mod_webdav]]. While lighttpd ETag generation for static content can be disabled for micro-benchmarking purposes, ETag generation (default enabled) is recommended for production use (@etag.use-inode@, @etag.use-mtime@, @etag.use-size@) could not disable the incoming connections soon enough. The * compile lighttpd with mmap support (@./configure --enable-mmap@) to improve [[Docs_ModDeflate|mod_deflate]] performance h4. lighttpd configuration for use of operating system (OS) features lighttpd generally chooses optimal defaults for connection was dropped and the OS on which it client received an error message like 'connection failed'. This is running. Prefer lighttpd defaults unless something is not functioning correctly. (Please report bugs very rare and include your platform information if might only occur in test setups. Increasing the lighttpd OS defaults are not working correctly.) ``server.max-fds`` limit will reduce the probability of this * @server.event-handler@ (e.g. epoll, kqueue, event ports, devpoll, poll, ...) * @server.network-backend@ (e.g. sendfile, writev, write) h4. lighttpd configuration tuning for high-traffic sites with a large number of connections problem. * test with @server.max-fds = 16384@ (or higher) h2. stat() cache A stat(2) can be expensive; caching it saves time and OS system and/or per-user @ulimit -Hn@ might need to be adjusted to allow this or higher values. For each 4k increase in @server.max-fds@, lighttpd uses an additional ~100 kb context switches. Instead of memory for internal structures, not including memory used by each active connection. (In other words, there is a marginal cost for using very high values when there are not nearly so many simultaneous open connections). @server.max-connections@ is calculated stat() every time to be 1/3 check for the existence of @server.max-fds@ if @server.max-connections@ a file you can stat() it once and monitor the directory the file is not configured. h4. lighttpd configuration tuning in for low-memory systems * test with @server.max-fds = 128@ (or lower) * test with @server.max-connections = 16@ (or lower) * test with @server.listen-backlog = 16@ (or lower) * (default) @server.stat_cache-engine = "simple"@ * (default) @ssl.read-ahead = "disable"@ * support for modifications. As long as the HTTP/2 protocol (enabled by default directory doesn't change, the files in lighttpd 1.4.59) uses more memory than HTTP/1.1; low-memory systems might choose to disable HTTP/2 protocol support: @server.feature-flags += ("server.h2proto" => "disable")@ h4. lighttpd configuration tuning for traffic shapping (download rate-limiting) [[Docs_TrafficShaping|Traffic Shaping]] it @connection.kbytes-per-second@ @server.kbytes-per-second@ h4. lighttpd configuration tuning for timeouts must all still be the same. To free up connections more quickly, tune down With the idle timeouts for how long lighttpd waits to read help of FAM or write gamin you can use kernel events to the client (when lighttpd assure that your stat cache is trying up to read or write), or how long lighttpd waits for date. Alternatively you can use the next keep-alive request, and "simple" method, which caches results for how many keep-alive requests, before lighttpd closes up to one second. Enabling the connection. A value of 0 disables an idle timeout and is not recommended. @server.max-read-idle stat cache can significantly improve performance on heavily-loaded servers: <pre> server.stat-cache-engine = 60@ "fam" # either fam, simple or disabled </pre> See http://oss.sgi.com/projects/fam/faq.html for information about FAM. @server.max-write-idle = 360@ @server.max-keep-alive-idle = 5@ @server.max-keep-alive-requests = 100@ Generally, @server.max-keep-alive-requests@ should not be set to 0 since setting up a new TCP connection takes more resources than keeping an open idle fd, especially if the connection is over TLS. h4. See http://www.gnome.org/~veillard/gamin/overview.html for information about gamin. See also: http://trac.lighttpd.net/trac/wiki/server.stat-cache-engineDetails h2. Platform-Specific Notes *Note:* the following is old and possibly out-dated. Please consider only as a starting point for further testing. h5. h3. Linux For Linux 2.4.x you should think about compiling lighttpd with the option ``--disable-lfs`` to disable the support for files larger than 2GB. lighttpd will fall back to the ``writev() + mmap()`` network calls which is ok, but not as fast as possible but support files larger than 2GB. Disabling the TCP options reduces the overhead of each TCP packet and might help to get the last few percent of performance out of the server. Be aware that disabling these options most likely decreases performance for high-latency and lossy links. * net.ipv4.tcp_sack = 0 * net.ipv4.tcp_timestamps = 0 *Note:* Be carefull with net.ipv4.tcp_timestamps. It caused massive problems for me under benchmark load with a high count of concurrent connections. Increasing the TCP send and receive buffers will increase the performance a lot if (and only if) you have a lot of large files to send. * net.ipv4.tcp_wmem = 4096 65536 524288 * net.core.wmem_max = 1048576 If you have a lot of large file uploads, increasing the receive buffers will help. * net.ipv4.tcp_rmem = 4096 87380 524288 * net.core.rmem_max = 1048576 Some things that a high-traffic site found useful: <pre> # These ensure that TIME_WAIT ports either get reused or closed fast. net.ipv4.tcp_fin_timeout = 1 net.ipv4.tcp_tw_recycle = 1 # TCP memory net.core.rmem_max = 16777216 net.core.rmem_default = 16777216 net.core.netdev_max_backlog = 262144 net.core.somaxconn = 262144 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_max_orphans = 262144 net.ipv4.tcp_max_syn_backlog = 262144 net.ipv4.tcp_synack_retries = 2 net.ipv4.tcp_syn_retries = 2 # you shouldn't be using conntrack on a heavily loaded server anyway, but these are # suitably high for our uses, insuring that if conntrack gets turned on, the box doesn't die net.ipv4.netfilter.ip_conntrack_max = 1048576 net.nf_conntrack_max = 1048576 </pre> Keep in mind that every TCP connection uses the configured amount of memory for socket buffers. If you've got many connections this can quickly drain the available memory. See http://www.acc.umu.se/~maswan/linux-netperf.txt for more information on these parameters. h5. h3. FreeBSD On FreeBSD you might gain some performance by enabling accept filters. Just compile your kernel with: <pre> options ACCEPT_FILTER_HTTP </pre> or just load it using <pre> kldload accf_http </pre> For more ideas about tuning FreeBSD read: tuning(7) Reducing the recvspace should always be ok if the server only handles HTTP requests without large uploads. Increasing the sendspace would reduce the system load if you have a lot of large files to be sent, but keep in mind that you have to provide the memory in the kernel for each connection. 1024 * 64KB would mean 64MB of kernel RAM. Keep this in mind. * net.inet.tcp.recvspace = 4096