Lighttpd needs to restart occassionally

Hi,

my lighttpd setup seems to need to restart on occassion. What could be the cause?

I have a site which uses lighttpd to serve static pages - after some time, when loading the site, I just see a 503 error come up. Service Temporarily Unavailable. Apache is still running.

running on lighttpd 1.4.24
apache 2
php 5

Normally this happens when server experiences a lot of requests. Also, I need to stop apache first before restarting lighttppd so that the lighttpd configuration settings will take effect - is this normal?

Thanks in advance for any solution to this problem.

Replies (5)

RE: Lighttpd needs to restart occassionally - Added by carpii over 15 years ago

h3000 wrote:

Hi,

my lighttpd setup seems to need to restart on occassion. What could be the cause?

I have a site which uses lighttpd to serve static pages - after some time, when loading the site, I just see a 503 error come up. Service Temporarily Unavailable. Apache is still running.

running on lighttpd 1.4.24
apache 2
php 5

Normally this happens when server experiences a lot of requests. Also, I need to stop apache first before restarting lighttppd so that the lighttpd configuration settings will take effect - is this normal?

Thanks in advance for any solution to this problem.

I dont think you've given us enough details foe anyone to really answer with any authority.

HTTP 503 means the web server cant take any more requests.

This could be caused by a few things, but youre using PHP, so Ill assume you're also using MySQL. See whats happening on your mySQL server when this happens. It may be that theres various queries locking tables, meaning PHP scripts are being blocked, meaning lighttpd connections are being held up until the lock is released.

Im not sure restarting lighty is necessarily fixing the problem, its just killing all those blocked connections, perhaps by which time whatever was causing the lock up has completed.

Ultimately though, you need to provide more details ;-)

RE: Lighttpd needs to restart occassionally - Added by h3000 over 15 years ago

so I set up a script that monitors when this happens and output the processlist whenever the server returns a 503 error.

the problem happened again and this is the processlist at the time:

+---------+-------+-----------+-------+---------+------+-------+------------------+
| Id      | User  | Host      | db    | Command | Time | State | Info             |
+---------+-------+-----------+-------+---------+------+-------+------------------+
| 2316398 | szp6  | localhost | szp6c | Sleep   | 0    |       |                  |
| 2316399 | szp6  | localhost | szp6c | Sleep   | 0    |       |                  |
| 2316400 | szp6  | localhost | szp6c | Sleep   | 0    |       |                  |
| 2316401 | admin | localhost |       | Query   | 0    |       | show processlist |
+---------+-------+-----------+-------+---------+------+-------+------------------+

not sure what other details are needed.

the site is a Drupal based site using the Boost module to improve performance. Lighttpd is set up to serve image and html files from certain directories.

the following are further details:
Drupal 6.14
Boost 6.x-1.14
Lighttpd 1.4.24
Apache 2
PHP 5
Mysql 5

the Boost module was only installed recently and that was probably the reason why the 503 error was noticed because the html files were resulting in 503 errors when this thing happened.

I have also played with the MaxClients settings for apache the following are the current settings:

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
    StartServers          8
    MinSpareServers       8
    MaxSpareServers      8
    MaxClients          50
    MaxRequestsPerChild   500
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_worker_module>
    StartServers          2
    MaxClients          150
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadsPerChild      25
    MaxRequestsPerChild   0
</IfModule>

these are also my lighttpd settings:

server.modules              = (
            "mod_expire",
#            "mod_fastcgi",
            "mod_access",
            "mod_alias",
            "mod_accesslog",
#            "mod_rewrite",
            "mod_redirect",
            "mod_setenv",
            "mod_status",
#           "mod_evhost",
            "mod_compress",
#           "mod_usertrack",
#           "mod_rrdtool",
#           "mod_webdav",
#           "mod_flv_streaming",
#           "mod_evasive",
#            "mod_magnet" 
 )

server.document-root       = "/var/www/documents/" 

server.errorlog            = "/var/log/lighttpd/error.log" 
index-file.names = ( "index.php", "index.html", "index.htm" )
#magnet.attract-physical-path-to = ( "/etc/lighttpd/drupal.lua" )

## bind to port (default: 80)
server.port                = 81

## bind to localhost (default: all interfaces)
server.bind                = "127.0.0.1" 

## set the event-handler (read the performance section in the manual)
#server.event-handler = "freebsd-kqueue" # needed on OS X

# timing parameters 10/2/08 2:26 AM
server.max-keep-alive-requests = 16
server.max-keep-alive-idle = 1
server.max-read-idle = 15
server.max-write-idle = 360

#### status module
status.status-url          = "/3server-status" 
status.config-url          = "/3server-config" 

mimetype.assign             = (
  ".pdf"          =>      "application/pdf",
  ".sig"          =>      "application/pgp-signature",
  ".spl"          =>      "application/futuresplash",
  ".class"        =>      "application/octet-stream",
  ".ps"           =>      "application/postscript",
  ".torrent"      =>      "application/x-bittorrent",
  ".dvi"          =>      "application/x-dvi",
  ".gz"           =>      "application/x-gzip",
  ".pac"          =>      "application/x-ns-proxy-autoconfig",
  ".swf"          =>      "application/x-shockwave-flash",
  ".tar.gz"       =>      "application/x-tgz",
  ".tgz"          =>      "application/x-tgz",
  ".tar"          =>      "application/x-tar",
  ".zip"          =>      "application/zip",
  ".mp3"          =>      "audio/mpeg",
  ".m3u"          =>      "audio/x-mpegurl",
  ".wma"          =>      "audio/x-ms-wma",
  ".wax"          =>      "audio/x-ms-wax",
  ".ogg"          =>      "application/ogg",
  ".wav"          =>      "audio/x-wav",
  ".gif"          =>      "image/gif",
  ".jpg"          =>      "image/jpeg",
  ".jpeg"         =>      "image/jpeg",
  ".png"          =>      "image/png",
  ".xbm"          =>      "image/x-xbitmap",
  ".xpm"          =>      "image/x-xpixmap",
  ".xwd"          =>      "image/x-xwindowdump",
  ".css"          =>      "text/css",
  ".html"         =>      "text/html; charset=utf-8",
  ".htm"          =>      "text/html",
  ".js"           =>      "text/javascript",
  ".asc"          =>      "text/plain",
  ".c"            =>      "text/plain",
  ".cpp"          =>      "text/plain",
  ".log"          =>      "text/plain",
  ".conf"         =>      "text/plain",
  ".text"         =>      "text/plain",
  ".txt"          =>      "text/plain",
  ".dtd"          =>      "text/xml",
  ".xml"          =>      "text/xml",
  ".mpeg"         =>      "video/mpeg",
  ".mpg"          =>      "video/mpeg",
  ".mov"          =>      "video/quicktime",
  ".qt"           =>      "video/quicktime",
  ".avi"          =>      "video/x-msvideo",
  ".asf"          =>      "video/x-ms-asf",
  ".asx"          =>      "video/x-ms-asf",
  ".wmv"          =>      "video/x-ms-wmv",
  ".bz2"          =>      "application/x-bzip",
  ".tbz"          =>      "application/x-bzip-compressed-tar",
  ".tar.bz2"      =>      "application/x-bzip-compressed-tar" 
 )

#accesslog.filename         = "/var/log/lighttpd/access.log" 
url.access-deny            = ( "~", ".inc", ".module", ".install", ".info", ".engine", ".profile", ".po", ".sh", ".sql", ".theme", ".tpl.php", ".xtmpl" )
server.pid-file            = "/var/run/lighttpd.pid" 
server.dir-listing          = "disable" 
static-file.exclude-extensions = ( ".fcgi", ".php", ".rb", "~", ".inc" )
server.username            = "www-data" 
server.groupname           = "www-data" 
compress.cache-dir          = "/var/cache/lighttpd/compress/" 
compress.filetype           = ("text/plain", "text/html", "text/xml", "application/xhtml+xml", "text/css", "application/xml", "image/svg+xml", "application/rss+xml", "application/atom_xml", "application/x-javascript", "application/x-httpd-php", "application/x-httpd-fastphp", "application/x-httpd-eruby", "text/javascript")

$HTTP["url"] =~ "\.(png|jpe?g|gif|js|css)$" {
     expire.url = ( "" => "access 1 months" )
}

    # Change expires header for boost functionality
    #    Header add Expires "Sun, 19 Nov 1978 05:00:00 GMT" 
$HTTP["url"] =~ "^/cache" {
     setenv.add-response-header = ( "Expires" => "Sun, 19 Nov 1978 05:00:00 GMT" )
}

static-file.etags = "disable" 

# Lighttpd redirect www to no www
# to lighttpd redirect www by kenzie on Jan 31, 2008
# Catch all requests for domain.tld and redirect to www.domain.tld

$HTTP["host"] =~ "^\.(.*)" {
   url.redirect = ( "^/(.*)" => "http://www.%1/$1" )
}

Thanks.

RE: Lighttpd needs to restart occassionally - Added by h3000 over 15 years ago

here is another record of the processlist when lighttpd was not working again:

+--------+-------+-----------+-------+---------+------+--------------+------------------------------------------------------------------------------------------------------+
| Id     | User  | Host      | db    | Command | Time | State        | Info                                                                                                 |
+--------+-------+-----------+-------+---------+------+--------------+------------------------------------------------------------------------------------------------------+
| 133297 | szp6  | localhost | szp6c | Query   | 1    | Sending data | SELECT node.nid AS nid, node.sticky AS node_sticky, node.created AS node_created FROM node no        |
| 133337 | szp6  | localhost | szp6c | Query   | 1    | Sending data | SELECT COUNT(*) FROM (SELECT node.nid AS nid FROM node node LEFT JOIN node n ON node.nid = n.nid     |
| 133347 | szp6  | localhost | szp6c | Query   | 1    | Sending data | SELECT COUNT(*) FROM (SELECT node.nid AS nid FROM node node LEFT JOIN node n ON node.nid = n.nid     |
| 133352 | szp6  | localhost | szp6c | Sleep   | 0    |              |                                                                                                      |
| 133353 | admin | localhost |       | Query   | 0    |              | show processlist                                                                                     |
+--------+-------+-----------+-------+---------+------+--------------+------------------------------------------------------------------------------------------------------+

this is also the info on the proxy pass thru directives in the apache vhosts config file

...
<IfModule proxy_module>
   ProxyRequests Off
   ProxyPreserveHost On
      # Added next to lines to stop "Error reading remote server" with lighttpd communication 10/30/08 7:32 AM
   SetEnv force-proxy-request-1.0 1
   SetEnv proxy-nokeepalive 1

<IfModule mod_rewrite.c>
        RewriteEngine on
        RewriteLog "/var/log/apache2/rewrite.log" 
        RewriteLogLevel 9

        # Pass image requests for drupal imagecache module images to lighttpd only if files exist
        # otherwise pass request to apache so that we can generate the image
        RewriteCond %{REQUEST_URI} ^/files/imagecache
        RewriteCond %{DOCUMENT_ROOT}/files/imagecache/$1/$2/$3 -f
        RewriteRule ^/files/imagecache/([-_a-zA-Z0-9]+)/files/([-_/a-zA-Z0-9]+)/([-_a-zA-Z0-9\ @\=\(\)]+\.(png|gif|jpg|jpeg))$ http://0.0.0.0:81/files/imagecache/$1/$2/$3 [NC,L,P]

        # Pass image requests for drupal imagecache module images to lighttpd only if files exist
        # otherwise pass request to apache so that we can generate the image
        RewriteCond %{REQUEST_URI} ^/audio
        RewriteCond %{DOCUMENT_ROOT}/files/audio/$2 -f
        RewriteRule ^/audio/download/([-_a-zA-Z0-9]+)/([-_a-zA-Z0-9\ @\=\(\)]+\.(mp3||m4a|wav))$ http://0.0.0.0:81/files/audio/$2 [NC,L,P]

        RewriteCond %{REQUEST_URI} ^/files/imagecache
        RewriteCond %{DOCUMENT_ROOT}$1 -f
        RewriteRule (.*) http://0.0.0.0:81$1 [NC,L,P]

        # Combined CSS files must be generated on the fly before serving don't serve through lighttpd if file does not exist        
        RewriteCond %{REQUEST_URI} ^/files/css
        RewriteCond %{DOCUMENT_ROOT}$1 -f
        RewriteRule (.*) http://0.0.0.0:81$1 [NC,L,P]

        # Combined JS files must be generated on the fly before serving don't serve through lighttpd if file does not exist        
        RewriteCond %{REQUEST_URI} ^/files/js
        RewriteCond %{DOCUMENT_ROOT}$1 -f
        RewriteRule (.*) http://0.0.0.0:81$1 [NC,L,P]

        # Disable this to debug rewrite rules
        RewriteLogLevel 1 
</IfModule>

   # Exclude imagecache directory as these images are generated dynamically by php if they don't exist
   ProxyPass /files/imagecache !
   ProxyPass /files/css !
   ProxyPass /files/js !
   # Pass these directories to lighttpd
   ProxyPass /files http://0.0.0.0:81/files
   ProxyPass /images http://0.0.0.0:81/images
# Pass boost cache requests to lighttpd
   ProxyPass /cache http://0.0.0.0:81/cache
   ProxyPassReverse / http://0.0.0.0:81/
</IfModule>
...

btw, lighttpd is not running php or mysql - it is just serving static files

thanks in advance for any info on this problem.

RE: Lighttpd needs to restart occassionally - Added by h3000 over 15 years ago

Hi all,

I am still having this problem - lighttpd crashes once or twice a day and needs to be restarted each time.

It seems to be during high server load.

Anyone experiencing the same problem?

this one is an output of processlists in mysql during one of the times the 503 errors come up

*----------------------------------------*
Sun Nov 29 15:26:01 HKT 2009 [ERROR 503] start -- 
+--------+-------+-----------+-------+---------+------+-------+-----------------------+
| Id     | User  | Host      | db    | Command | Time | State | Info                  |
+--------+-------+-----------+-------+---------+------+-------+-----------------------+
| 582298 | szp6  | localhost | szp6c | Query   | 0    |       |                       |
| 582300 | szp6  | localhost | szp6c | Sleep   | 0    |       |                       |
| 582301 | admin | localhost |       | Query   | 0    |       | show full processlist |
+--------+-------+-----------+-------+---------+------+-------+-----------------------+
Uptime: 161257  Threads: 3  Questions: 86893113  Slow queries: 1705  Opens: 61738  Flush tables: 1  Open tables: 477  Queries per second avg: 538.849
Sun Nov 29 15:26:13 HKT 2009 [ERROR 503] end 
*----------------------------------------*

thanks for any info on the problem

RE: Lighttpd needs to restart occassionally - Added by h3000 over 15 years ago

Hi,

Project

General

Profile

Lighttpd