PageOutline

Developmenet Problem And Solution

This page is meant to list Problem And Solutions implemented in, or analyzed for lighttpd. Any simple feature requests without solution should goes to ticket system.

Redundant FastCGI/SCGI etc

Problem

In lighttpd 1.4, most of the source code of mod_cgi, mod_scgi and mod_proxy were cut'n'pasted from mod_fastcgi.

Why problem: Redundant code make it hard to maintain.

Solution

Combine them into mod_proxy_core and add protocols around it
  • the core provides
    • config handling
    • connect/retry on failure
    • fork/restart worker child on dead. (easier to improve native win32 support)
    • balancing
    • x-sendfile
  • the protocol backends take care of
    • preparing the environment (most cgi env code can be shared)
    • encode/decode data
    • handle io

Why: scgi/fastcgi/ejb/http or whatever, are just protocols that we can use to send requests to and get response from backend.

Status

Implemented in 1.5.0

Redundant and simple config handling

Problem

The current config handling is done in core and plugin. The core parse the config, convert it into a conditional tree, and the plugin pick up some of the values (use PATCH) and use it.

Why problem: It's sure easier to fix the 1 core instead of all the plugins, the plugin is still handling too much config handling logical and i't hard to improve.

Solution

Combine most of config handling into core, including:
  • alloc/free plugin data
  • init default values
  • insert values from config (into plugin data)
  • patch(pick) values from plugin data for connection. this is done by calling a function ptr, but we can kill the string comparisons, lower or higher performance?

Status

Scheduled but not done in 1.5.0, and is likely to be post pound to next major release.

.htaccess support

Problem

Many users want .htaccess or whatever named config file(s) sit(s) in document root or its sub dir.

Neets:
  1. vhost users can do no harm creating such file
  2. limited config settings
  3. has to be dynamic loaded. i.e., it makes affect when the file is updated without restarting lighttpd
    Improvements:
  4. high performance, e.g.: "cached"
  5. robust, e.g.: syntax error won't stop the server

Solution

No full solution yet from the view of the current source state, but anyway the following precondition should be supported first:
  1. clean config parsing with no memory leak (already)
  2. dynamic config updating
  3. versioned config: multiple clients may still referenced to current config nodes while the new one is being loaded
    Cautions:
    • avoid blocking operation (e.g.: disk io) reloading the file

Status

Feature requested.

c10k problem

Problem

See c10k problem.

TODO: add more notes here

Solution

See c10k problem.

TODO: add more notes here

Status

Implemented since the first release of lighttpd by jan. (While other projects may use libevent)

Difficult coding for single threaded

Problem

Lighttpd was a proof of concept for c10k problem, for which reason it was made single processed/threaded.

Why problem: Single threaded is hard to code because everywhere u come to an syscall that would block, u gotta make it async.

Solution

Move some complex jobs:
  1. into a new thread
  2. into a forked child process
  3. as a backend (e.g. fcgi backend), which mean it is built into standalone executable.
Rules And things can be done in another thread or process:
  1. Static file request/response handling should stay in the main io thread
  2. Leave most of the current implemention as is unless it cause problem.
  3. For auto reload config from file, vhosts settings from mysql etc, should we do them in another thread or just use fam?
  4. fastcgized magnet, lua + FastCGI
  5. Move out mod_rrdtool (was deprecated by jan in 1.5, but should be moved instead of removed imho)
  6. mod_webdav
    TODO: more notes here

Why: Multi thread/process is not evil, fastcgi/php were make into standalone/backend process already.
Neither multi-thread/process was the key problem nor single threaded was the key solution to c10k.
The problem was 1:1 model (I'm not talking about thread scheduling model): every 1 connection have to be taken care of by 1 thread or process, and similarly 1 php instance opens up at least 1 persistent connection to mysql.
This consume lots of memory and requires lots of processes/threads untill the OS can't handle.
With the current implementation of lighttpd, because of http have to be tcp, you still have to 1 vs 1 for connection vs file handler, it's far more better than too many threades/processes, and it's lucky that we have O(1) OS impl for it like linux epoll etc.

Status

Unknown

Buffer whole POST request or not

Problem

Solution

Status

both 1.4 and 1.5 uses buffer all post data for each request currently.