Lighttpd

The problem¶

Blocking operations are real problem for event based servers. Blocking I/O such as stat() is a good example here.

Fobax on #lighttpd brought up the idea to use fastcgi to speed up the blocking stat() for us.

The idea¶

Lighttpd passes the request to a simple fastcgi app that does a stat() call.
Then the app sends back a X-LIGHTTPD-send-file header with the original filename.
If lighttpd now stats the same file, the information is already hot in the kernel cache and we get
result without waiting for the disk.

On the one hand, this adds the overhead of fastcgi to static file serving, on the other hand it really speeds up the stat() calls. For workloads where the bottleneck is harddrive seek time, this is a worthwhile trade off.

This works best for servers which have a very high stat() count like for all the servers which have to send alot small files (thumbnails, images, ads, ...). For servers with a single drive, it won't make a big difference, but for servers with many drives, it'll make a big difference. One benchmark on a machine with 7 drives showed an 8 times improvement over lighttpd alone. Similar results can be achieved by using lighttpds max-workers feature, though this avoids the problems with having multiple processes, and is lighter.

The code¶

Fobax provided a working example code for us. I just added support to modify the thread count at runtime.
For easier support with spawn-fcgi we abused the PHP_FCGI_CHILDREN variable for that.

/*
  compile with: 

  $ gcc -lfcgi -lpthread fcgi-stat-accel.c -o fcgi-stat-accel

  fcgi-stat-accel will use the PHP_FCGI_CHILDREN environment variable to set the thread count.

  The default value, if spawned from lighttpd, is 20.
*/

#include "fcgi_config.h" 

#include <pthread.h>
#include <sys/types.h>
#include <unistd.h>
#include "fcgiapp.h" 
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>

#include <stdlib.h>
#include <stdio.h>

#define THREAD_COUNT 20

#define FORBIDDEN(stream) \
    FCGX_FPrintF(stream, "Status: 403 Forbidden\r\nContent-Type: text/html\r\n\r\n<h1>403 Forbidden</h1>\n");
#define NOTFOUND(stream, filename) \
    FCGX_FPrintF(stream, "Status: 404 Not Found\r\nContent-Type: text/html\r\n\r\n<h1>404 Not Found</h1>\r\n%s", filename);
#define SENDFILE(stream, filename) \
    FCGX_FPrintF(stream, "X-LIGHTTPD-send-file: %s\r\n\r\n", filename); 

static void *doit(void *a){
    FCGX_Request request;
    int rc;
    char *filename;
    FILE *fd;

    FCGX_InitRequest(&request, 0, FCGI_FAIL_ACCEPT_ON_INTR);

    while(1){
        //Some platforms require accept() serialization, some don't. The documentation claims it to be thread safe
//        static pthread_mutex_t accept_mutex = PTHREAD_MUTEX_INITIALIZER;
//        pthread_mutex_lock(&accept_mutex);
        rc = FCGX_Accept_r(&request);
//        pthread_mutex_unlock(&accept_mutex);

        if(rc < 0)
            break;

    //get the filename
        if((filename = FCGX_GetParam("SCRIPT_FILENAME", request.envp)) == NULL){
            FORBIDDEN(request.out);
    //don't try to open directories
        }else if(filename[strlen(filename)-1] == '/'){
            FORBIDDEN(request.out);
    //open the file
        }else if((fd = fopen(filename, "r")) == NULL){
            NOTFOUND(request.out, filename);
    //no error, serve it
        }else{
            SENDFILE(request.out, filename);

            fclose(fd);
        }

        FCGX_Finish_r(&request);
    }
    return NULL;
}

int main(void){
    int i,j,thread_count;
    pthread_t* id;
    char* env_val;

    FCGX_Init();

    thread_count = THREAD_COUNT;
    env_val = getenv("PHP_FCGI_CHILDREN");
    if (env_val != NULL) {
        j = atoi(env_val);
        if (j != 0) {
            thread_count = j;
        };
    };

    id = malloc(sizeof(*id) * thread_count);

    for (i = 0; i < thread_count; i++) {
        pthread_create(&id[i], NULL, doit, NULL);
    }

    doit(NULL);
    free(id);
    return 0;
}

lighty config¶

To make this work, add something similar to the following to your lighttpd.conf:

   ## for all files
   fastcgi.server = ( "" =>
                  ( "fastcgi-stat-local" =>
                    (
                      "socket" => "/var/tmp/fcgi-stat-accel.socket",
                      "bin-path" => "/usr/local/bin/fcgi-stat-accel",
                      "bin-environment" => (
                        "PHP_FCGI_CHILDREN" => "2",
                      ),
                      "max-procs" => 1,
                      "disable-time" => 2,
                      "check-local" => "disable",
                      "allow-x-send-file" => "enable",
                    )
                  )
               )