DevelProblemAndSolution » History » Revision 17
Revision 16 (gstrauss, 2018-10-31 11:09) → Revision 17/19 (gstrauss, 2021-08-25 03:17)
h1. Development Problems And Solutions {{>toc}} This page is meant to list development challenges, problems, Problems and (perhaps) solutions Solutions implemented in, or analyzed for lighttpd. Feature Any simple feature requests without solution should go to "ticket system":http://redmine.lighttpd.net/projects/lighttpd/issues/new. h1. Redundant FastCGI/SCGI etc h2. Problem In lighttpd 1.4, most of the source code of mod_cgi, mod_scgi and mod_proxy were cut'n'pasted from mod_fastcgi. Why problem: Redundant code make it hard to maintain. h2. Solution Combine them into mod_proxy_core and add protocols around it ** the core provides ***** config handling ***** connect/retry on failure ***** fork/restart worker child on dead. (easier to improve native win32 support) ***** balancing ***** x-sendfile ** the protocol backends take care of ***** preparing the environment (most cgi env code can be posted shared) ***** encode/decode data ***** handle io Why: scgi/fastcgi/ejb/http or whatever, are just protocols that we can use to send requests to and get response from backend. h2. Status Implemented in 1.4.46 Implemented in 1.5.0 (abandoned) h1. Redundant and simple config handling h2. Problem The current config handling is done in core and plugin. The core parse the "lighttpd Forums":https://redmine.lighttpd.net/projects/lighttpd/boards config, convert it into a conditional tree, and the plugin pick up some of the values (use PATCH) and use it. Why problem: It's sure easier to fix the 1 core instead of all the plugins, the plugin is still handling too much config handling logical and it is hard to improve. h2. Solution Combine most of config handling into core, including: * alloc/free plugin data * init default values * insert values from config (into plugin data) * patch(pick) values from plugin data for discussion. connection. this is done by calling a function ptr, but we can kill the string comparisons, lower or higher performance? h2. .htaccess-like functionality Status h3. not yet implemented h1. .htaccess support h2. Problem support .htaccess-like functionality Many users want .htaccess or whatever named config file(s) sit(s) in document root or its sub dir. Needs: Neets: # vhost users can do no harm creating such file # limited config settings # has to be dynamic loaded, i.e. takes loaded. i.e., it makes affect when the file is updated (without without restarting lighttpd) lighttpd Improvements: # high performance, e.g.: "cached" # robust, e.g.: syntax error won't stop the server h3. Discussion h2. Solution No full solution yet from the view of the current source state, but anyway the following precondition should be supported first: # clean config parsing with no memory leak (already) # dynamic config updating # versioned config: multiple clients may still referenced to current config nodes while the new one is being loaded Cautions: * avoid blocking operation (e.g.: disk io) reloading the file h3. Solutions h2. Status * Option: ** configure standalone application server and have the application implement the .htaccess policy e.g. via PHP-FPM and per-application php.ini [[TutorialLighttpdAndPHP#Per-directory-PHP-Config|PHP htscanner]] Feature requested. h1. c10k problem h2. Problem * Option: ** convert .htaccess into file included in lighttpd.conf ** user-triggered immediate graceful restart (reload config while continuing to handling existing requests in a background process) @server.feature-flags += ("server.graceful-restart-bg" => "enable")@ ([[Server_feature-flagsDetails|server.feature-flags]]) @server.systemd-socket-activation = "enable"@ See "c10k problem":http://www.kegel.com/c10k.html. * Option: ** convert .htaccess into custom lua script run with [[Docs_ModMagnet|mod_magnet]] Common use of .htaccess for url-rewriting and access control can be written into a custom lua script. @$HTTP["url"] =~ "^/app(?:/|$)" { magnet.attract-raw-url-to = "/path/to/app/.htaccess.lua" }@ [[Docs_ModMagnet|mod_magnet]] scripts are cached and are dynamically reloaded when modified. TODO: add more notes here h2. Solution If non-admin users are writing lighttpd.conf include files or lua scripts, then a lighttpd frontend should be configured with [[Docs_ModProxy|mod_proxy]] to reverse proxy to a per-application (or per-user) lighttpd backend instance, e.g. running [[Docs_ModMagnet|mod_magnet]], and potentially permissioned to let See "c10k problem":http://www.kegel.com/c10k.html. TODO: add more notes here h2. Status Implemented since the application owner (or user) configure lighttpd.conf and trigger restart first release of the per-application (or per-user) lighttpd instance. h2. by jan. (While other projects may use "libevent":http://monkey.org/~provos/libevent/) h1. Difficult coding for single threaded h3. h2. Problem Lighttpd was a proof of concept for "c10k problem":http://www.kegel.com/c10k.html, for which reason it was made single processed/threaded. Why problem: Single threaded is hard to code because everywhere you u come to a an syscall that would block, you have to u gotta make it async. h3. Discussion h2. Solution Move some complex jobs: # into a new thread # into a forked child process # as a backend (e.g. fcgi backend), which mean it is built into standalone executable. Rules And things can be done in another thread or process: # Static file request/response handling should stay in the main io thread # Leave most of the current implemention as-is as is unless it cause problem. # For auto reload config from file, vhosts settings from mysql etc, should we do them in another thread or just use fam? # "fastcgized magnet, lua + FastCGI":http://jan.kneschke.de/projects/lua # Move out mod_rrdtool (was deprecated by jan in 1.5 (abandoned), but should be moved instead of removed imho) # mod_webdav TODO: more notes here Why: Multi thread/process is not evil, fastcgi/php were make into standalone/backend process already. Neither multi-thread/process was the key problem nor single threaded was the key solution to c10k. The problem was 1:1 model (I'm not talking about thread scheduling model): every 1 connection have to be taken care of by 1 thread or process, and similarly 1 php instance opens up at least 1 persistent connection to mysql. This consume lots of memory and requires lots of processes/threads until untill the OS can't handle. With the current implementation of lighttpd, using TCP for HTTP, because of http have to be tcp, you still have to 1 vs 1 for connection vs file handler, it's far more better than too many threads/processes, threades/processes, and it's lucky that we have O(1) OS impl for it like linux epoll etc. h2. Status Unknown h1. Buffer whole POST request or not h2. Problem h2. Solution h2. Status lighttpd buffers all POST data in memory for each request *fixed* in 1.4.40 (not fixed in 1.5 (abandoned))