Development Problems And Solutions¶

• Table of contents
• Development Problems And Solutions
  • .htaccess-like functionality
    • Problem
    • Discussion
    • Solutions
  • Difficult coding for single threaded
    • Problem
    • Discussion

This page is meant to list development challenges, problems, and (perhaps) solutions analyzed for lighttpd.
Feature requests should be posted in the lighttpd Forums for discussion.

.htaccess-like functionality¶

Problem¶

support .htaccess-like functionality

Needs:

1. vhost users can do no harm creating such file
2. limited config settings
3. has to be dynamic loaded, i.e. takes affect when the file is updated (without restarting lighttpd)
   Improvements:
4. high performance, e.g.: "cached"
5. robust, e.g.: syntax error won't stop the server

Discussion¶

No full solution yet from the view of the current source state, but anyway the following precondition should be supported first:

1. clean config parsing with no memory leak (already)
2. dynamic config updating
3. versioned config: multiple clients may still referenced to current config nodes while the new one is being loaded
   Cautions:
• avoid blocking operation (e.g.: disk io) reloading the file

Solutions¶

• Option:
  • configure standalone application server and have the application implement the .htaccess policy
    e.g. via PHP-FPM and per-application php.ini PHP htscanner

• Option:
  • convert .htaccess into file included in lighttpd.conf
  • user-triggered immediate graceful restart (reload config while continuing to handling existing requests in a background process)
    server.feature-flags += ("server.graceful-restart-bg" => "enable") (server.feature-flags)
    server.systemd-socket-activation = "enable"

• Option:
  • convert .htaccess into custom lua script run with mod_magnet
    Common use of .htaccess for url-rewriting and access control can be written into a custom lua script.
    $HTTP["url"] =~ "^/app(?:/|$)" { magnet.attract-raw-url-to = "/path/to/app/.htaccess.lua" }
    mod_magnet scripts are cached and are dynamically reloaded when modified.

If non-admin users are writing lighttpd.conf include files or lua scripts, then a lighttpd frontend should be configured with mod_proxy to reverse proxy to a per-application (or per-user) lighttpd backend instance, e.g. running mod_magnet, and potentially permissioned to let the application owner (or user) configure lighttpd.conf and trigger restart of the per-application (or per-user) lighttpd instance.

Difficult coding for single threaded¶

Problem¶

Lighttpd was a proof of concept for c10k problem, for which reason it was made single processed/threaded.
Why problem: Single threaded is hard to code because everywhere you come to a syscall that would block, you have to make it async.

Discussion¶

Move some complex jobs:

1. into a new thread
2. into a forked child process
3. as a backend (e.g. fcgi backend), which mean it is built into standalone executable.

Rules And things can be done in another thread or process:

1. Static file request/response handling should stay in the main io thread
2. Leave most of the current implemention as-is unless it cause problem.
3. For auto reload config from file, vhosts settings from mysql etc, should we do them in another thread or just use fam?
4. fastcgized magnet, lua + FastCGI
5. Move out mod_rrdtool (was deprecated by jan in 1.5 (abandoned), but should be moved instead of removed imho)
   TODO: more notes here

Why: Multi thread/process is not evil, fastcgi/php were make into standalone/backend process already.
Neither multi-thread/process was the key problem nor single threaded was the key solution to c10k.
The problem was 1:1 model (I'm not talking about thread scheduling model): every 1 connection have to be taken care of by 1 thread or process, and similarly 1 php instance opens up at least 1 persistent connection to mysql.
This consume lots of memory and requires lots of processes/threads until the OS can't handle.
With the current implementation of lighttpd, using TCP for HTTP, you still have to 1 vs 1 for connection vs file handler, it's far better than too many threads/processes, and it's lucky that we have O(1) OS impl for it like linux epoll etc.