Lighttpd

TracNav(DocsToc)

#!rst
==============================
Optimizing FastCGI performance
==============================

.. contents::

Overview
========

If you've recently asked yourself

* "How many PHP backends do I need for my load?" or
* "Why is my application returning an error 500 from time to time?" 

then you'll want to read this article very carefully.

How many PHP processes do I need?
=================================

That's the question you're looking for the answer to, and to answer it, it's probably easiest to use an example.

lighty is managing a pipe. On one side are your users with their web browsers, on the other side is PHP. If
you have more incoming requests than your backends can handle, lighty will queue them up and will push the new
requests to the backends when they are free again. If you have so many requests that the queue fills up, it will burst and the next 
requests to this backend will be denied, and you'll see a message in the error log like this: ::

  ... load = 380 ...

To calculate the number of backends you need, consider this:

* you have 100 PHP requests per second
* the average request time on the PHP side is 0.1sec

In the average case you need: ::

  100 PHP requests/sec * 0.1sec/PHP request = 10 PHP processes

Since you probably can't control the number of incoming PHP requests, the best you can do is to reduce the average 
request time spent in the PHP process. Some ideas:

* Use a byte-code cache like http://xcache.lighttpd.net/ or http://pecl.php.net/APC
* Add caching to your application 
* Tune your database queries

Measuring the average request time is not that easy, so output like "fastcgi.backend.0.load: 22" is an indicator
of how many PHP processes would be used right now.

Measuring the load
==================

Load the status-module and enable the statistics: ::

  server.modules = {..., "mod_status", ... }

  status.statistics-url = "/server-counters" 

The counters page lists serveral counters of the fastcgi module:

* the total number of requests handled by the module
* the total number of requests waiting to be handled
* the number of requests waiting to be handled per backend

.. note::

  If you have more than one backend, you should name each backend individually. ::

    fastcgi.server = ( 
      ".php" => (
        "backend1" => ( "host" => "php-srv1", ... ),
        "backend2" => ( "host" => "php-srv2", ... ),
      )
    ) 

You might get this output: ::

  fastcgi.active-requests: 22
  fastcgi.backend.0.0.connected: 5639
  fastcgi.backend.0.0.died: 0
  fastcgi.backend.0.0.disabled: 0
  fastcgi.backend.0.0.load: 11
  fastcgi.backend.0.0.overloaded: 0
  fastcgi.backend.0.1.connected: 7724
  fastcgi.backend.0.1.died: 0
  fastcgi.backend.0.1.disabled: 0
  fastcgi.backend.0.1.load: 11
  fastcgi.backend.0.1.overloaded: 0
  fastcgi.backend.0.load: 22
  fastcgi.requests: 13363

We have 2 backends (max-procs = 2) and a current load of 22 (fastcgi.backend.0.load: 22). The load is equally 
distributed over the two backends (fastcgi.backend.0.0.load: 11, fastcgi.backend.0.1.load: 11).

Using rrdtool to monitor the load
---------------------------------

Enable mod_rrdtool in your config, and add the following config entries (adapt these to your actual setup): ::

   rrdtool.binary = "/usr/bin/rrdtool" 
   rrdtool.db-name = "/var/www/lighttpd/lighttpd-web.rrd" 

Read the documentation on rrd on how to generate a graph from the .rrd file.

Installing XCache
=================

XCache is one of the cachers that speeds up fastcgi processing in case you're using PHP. Pick the lastest version from http://trac.lighttpd.net/xcache/wiki/ReleaseArchive.

To install: ::

  ~/src $ wget http://... (the release url)
  ~/src $ tar -zxf xcache-*.tar.gz
  ~/src $ cd xcache
  ~/src/xcache $ phpize
  ~/src/xcache $ ./configure --enable-xcache --enable-xcache-coverager
  ~/src/xcache $ make
  ~/src/xcache $ su
  ~/src/xcache # make install
  ~/src/xcache # cat xcache.ini >> /etc/php.ini
  ~/src/xcache # $EDITOR /etc/php.ini

Set xcache.size=64M, and set up your xcache.admin.pass.

Setting up the web interface: ::

  alias.url += ("/xcache-admin/" => "/usr/share/xcache/admin/")

Check it out by pointing your browser to http://localhost/xcache-admin/

Tuning the database 
===================

This is a very short intro to tuning MySQL. First check your my.cnf:

.. note::

  *Read up before changing anything on your server, especially if you are not on a dedicated MySQL box with spare memory.*

::

  [mysqld]
  ## default is 100, might need to raise it

  max-connections = 200
  ## if you use innodb alot, increase the pool-size 
  ## default is 8M, far too low.

  innodb_buffer_pool_size = 512M

  ## for MyISAM it is 
  key_buffer_size = 128M

  query_cache_size = 32M

  ## logs are good

  log-slow-queries

  long-query-time = 2

  log-queries-not-using-indexes

Restart the MySQL server and check for 'hostname-slow.log' in the datadir. It will list all queries which 

* take longer than 2 seconds to execute, or
* are not using an index

On all these queries, run an EXPLAIN and add an index when necessary. You'll want to concentrate on queries which

* are run often, or
* have a query-time > 2

If the number of examined rows is several times larger than the number of sent rows, add an index.

Another thing to check for is: ::

  mysql> SHOW GLOBAL STATUS;
  ...
  | Created_tmp_disk_tables | 88    |
  | Created_tmp_files       | 2     |
  | Created_tmp_tables      | 39079 |
  ...
  Created_tmp_disk_tables should as small as 
  possible compared to Created_tmp_tables 

  | Com_select              | 39004 |
  | Select_scan             | 39004 |
  Oops, all SELECT statements are Table-Scans, very bad.

  ...
  | Table_locks_immediate   | 3     |
  | Table_locks_waited      | 0     |
  This is good, we never had to wait for a table lock.

.. note:: 

  In MySQL before 5.0.x it is SHOW STATUS.

Benchmarking
============

By measuring the response time before and after the optimizations, you can get an idea of how an increased load
will affect you in the future.

Several tools are available to measure the response time:

* ab/ab2 is part of the Apache Server package

  * handles max 1024 connections
  * hammers a single URL only
  * is select()-based

* flood is an apache project
* siege

  * handles about 100 parallel connections (before it dies due to lack of memory)
  * is threaded
  * can generate random load

* http_load

  * can generate random load
  * allows throttling

* httpperf

When you are using a benchmark tool which only queries a single URL several times, you won't see problems caused by:

* dirty caches (MySQL Query Cache, Byte-Code Cache, ...)
* locking (Table Locks, File Locks, ...)

If you use siege for the random load and ab for the single-URL load, you should get useful results.

(to be continued)

Can I have too many PHP processes?
==================================

Having too many PHP processes can be bad too. If you have more than you need, the PHP processes will use up all the available memory
and will start using your swap space, which is much slower than real memory.

If you are using a config like this: ::

  fastcgi.server = ( ".php" =>
      (( "socket" => "/tmp/php-fastcgi.socket",
          "bin-path" => "/usr/bin/php-cgi",
          "max-procs" => 10,
          "bin-environment" => (
              "PHP_FCGI_CHILDREN" => "16",
              "PHP_FCGI_MAX_REQUESTS" => "1000" 
          ),
          "broken-scriptfilename" => "enable" 
      ))
  )

you will have (following the famous formula)::

  num-procs = max-procs * ( 1 + PHP_FCGI_CHILDREN ) 

  10 * (16 + 1) = 170 procs

A single PHP 5.1.5 process running with APC 3.0.11 takes about 13MB (RSZ) for itself: :: 

  $ ps axu | grep php
    400 web       16   0  152m  13m 6804 S    1  0.7   0:01.99 php-fcgi     

  13MB * 170 processes = 2,16GB RAM

Not all of them will be using 13MB right from the start, but you start to see the problem. Running into swap space is counter-productive.

Why is my PHP application returning an error 500 from time to time?
=====================================================================

This problem seems to stem from a little-known issue with PHP: PHP stops accepting new FastCGI connections after handling 500 requests; unfortunately, there is a potential race condition during the PHP cleanup code in which PHP can be shutting down but still have the socket open, so lighty can send request number 501 to PHP and have it "accepted", but then PHP appears to simply exit, causing a 500 return from lighty.

To limit this occurance, set PHP_FCGI_MAX_REQUESTS to 500.

Also, when configuring lighty to manage php-fcgi processes, it is better to have more processes and less children than less processes and more children. For example: ::

  fastcgi.server = ( ".php" =>
      (( "socket" => "/tmp/php-fastcgi.socket",
          "bin-path" => "/usr/bin/php-cgi",
          "max-procs" => 10,
          "bin-environment" => (
              "PHP_FCGI_CHILDREN" => "10",
              "PHP_FCGI_MAX_REQUESTS" => "500" 
          ),
          "broken-scriptfilename" => "enable" 
      ))
  )

...is better than... ::

  fastcgi.server = ( ".php" =>
      (( "socket" => "/tmp/php-fastcgi.socket",
          "bin-path" => "/usr/bin/php-cgi",
          "max-procs" => 2,
          "bin-environment" => (
              "PHP_FCGI_CHILDREN" => "50",
              "PHP_FCGI_MAX_REQUESTS" => "500" 
          ),
          "broken-scriptfilename" => "enable" 
      ))
  )

...because, when a backend "dies", all the children from that one FastCGI process become unavailable. Having many processes means that if one dies the rest can share the load. If you have 2 processes and both are loaded equally then if/when one dies the other one will suddenly become overloaded and die itself, causing lighty to throw 500 errors.

Note: If you are using php with bytecode caches every proc uses it's own bytecode cache. So if you have max-procs 2 there are 2 bytecode caches taking memory. In this case it is better to have max-procs 1 and use PHP_FCGI_CHILDREN to scale number of php processes and hope that backend won't "die".

Remote Balancing
================

If all this doesn't help, you can still take a few servers, install a shared filesystem like NFS and run 
PHP on these servers. Just add their IPs to the "host" field in the fastcgi.server setting and lighty
will balance the load across the PHP servers.

I use Perl, Ruby, Python, or another language
=============================================

PHP was only used as an example. Most of these suggestions apply to other languages as well.