Project

General

Profile

Bug #895

lighty 1.4.13 + php/fastcgi hangs after a short time of running

Added by Anonymous over 13 years ago. Updated almost 12 years ago.

Status:
Invalid
Priority:
Urgent
Category:
mod_fastcgi
Target version:
ASK QUESTIONS IN Forums:

Description

I tried lighty running on a quite busy site. After about 15 minutes, lighty seems hanging like this:


# top

last pid:  5144;  load averages:  0.11,  2.03,  4.77                                                        up 7+06:46:13  15:25:54
108 processes: 1 running, 107 sleeping
CPU states:     % user,     % nice,     % system,     % interrupt,     % idle
Mem: 330M Active, 784M Inact, 262M Wired, 13M Cache, 112M Buf, 2123M Free
Swap: 4095M Total, 4892K Used, 4090M Free

  PID USERNAME   THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
94291 www          1   4    0 47248K 46972K kqread 2   2:31  0.83% lighttpd
45996 www          1  96    0  7052K  3040K select 0  20:52  0.00% smbd
94228 www          1   4    0 40264K 32348K accept 2   2:20  0.00% php
94223 www          1   4    0 37028K 28916K accept 2   2:20  0.00% php
94235 www          1   4    0 26792K 18712K accept 0   2:19  0.00% php
94227 www          1   4    0 32452K 24304K accept 2   2:17  0.00% php
94233 www          1   4    0 31020K 22984K accept 2   2:16  0.00% php
94230 www          1   4    0 38032K 30180K accept 2   2:16  0.00% php
94232 www          1   4    0 24548K 16648K accept 2   2:16  0.00% php
94234 www          1   4    0 29584K 21828K accept 0   2:16  0.00% php
94226 www          1   4    0 40260K 32464K accept 0   2:16  0.00% php
94231 www          1   4    0 35348K 27452K accept 0   2:16  0.00% php
94225 www          1   4    0 32852K 24880K accept 0   2:14  0.00% php
94229 www          1   4    0 35512K 27660K accept 2   2:14  0.00% php
94236 www          1   4    0 32888K 24956K accept 0   2:12  0.00% php
94237 www          1   4    0 31896K 23952K accept 2   2:11  0.00% php
94224 www          1   4    0 27108K 19172K accept 0   2:11  0.00% php
94238 www          1   4    0 29352K 21516K accept 0   2:08  0.00% php
  438 root         1  96    0  3548K   928K select 2   0:48  0.00% nmbd
94294 www          1   4    0 10868K 10580K kqread 2   0:24  0.00% lighttpd
94295 www          1  -8    0  3116K  2188K piperd 2   0:04  0.00% ab1
  378 root         1  96    0  1300K   616K select 0   0:04  0.00% syslogd
94292 www          1   4    0  6000K  5688K kqread 2   0:03  0.00% lighttpd
  566 root         1   8    0  1312K   536K nanslp 0   0:02  0.00% cron
94293 www          1   4    0  5760K  5448K kqread 2   0:02  0.00% lighttpd

# netstat -an|grep stream|wc -l
    1061

Is it a little bit high?

My conf is:


server.max-worker = 4
server.username = "www" 
server.groupname = "www" 

$HTTP["host"] == "www.mysite.com" {
  server.document-root = "/usr/local/www/htdocs" 
  fastcgi.server = (
  ".php" => (
    ( "bin-path" => "/usr/local/bin/php",
      "socket" => "/tmp/php1.socket",
      "max-procs" => 1,
      "bin-environment" => (
         "PHP_FCGI_CHILDREN" => "16",
         "PHP_FCGI_MAX_REQUESTS" => "1000" 
      ),
      "bin-copy-environment" => (
         "PATH", "SHELL", "USER"               
      ),
      "broken-scriptfilename" => "enable",                 
      "idle-timeout" => 20
    )
  )
  )               
  server.max-write-idle = 15
  magnet.attract-raw-url-to = ( "/usr/local/www/bin/redirect.lua" )
}   

I'm running ea-0.9.5.

All php requests will result in a 500 error. A truss result is attached. It looks like some resource is eaten up. I've adjusted sysctl's as:


kern.timecounter.hardware=TSC
kern.ipc.somaxconn=2048
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.tcp.msl=5000
net.inet.tcp.sendspace=8192
net.inet.tcp.recvspace=8192
kern.maxfiles=99328
kern.maxfilesperproc=98304
kern.ipc.maxsockbuf=4000000

Is there some other sysctl's to tune?


# truss -p 94291
(null)()                                         = 1 (0x1)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 556392},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
accept(0x3,0xbfbfe8b0,0xbfbfe8ac)                ERR#35 'Resource temporarily unavailable'
gettimeofday({1161415562 565573},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(150,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x96,0xa773000,0x5bf)                       = 1452 (0x5ac)
write(25,0xa773000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 574098},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
accept(0x3,0xbfbfe8b0,0xbfbfe8ac)                ERR#35 'Resource temporarily unavailable'
gettimeofday({1161415562 584401},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(150,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x96,0xa773000,0x5bf)                       = 1452 (0x5ac)
write(25,0xa773000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 589518},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(175,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xaf,0x9a83000,0x47f)                       = 403 (0x193)
gettimeofday({1161415562 625972},0x0)            = 0 (0x0)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0xaf,0xa69d1e0,0x2)                       = 515 (0x203)
ioctl(175,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xaf,0x9a83000,0x47f)                       ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 626856},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(195,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xc3,0xa479000,0x4ff)                       = 662 (0x296)
gettimeofday({1161415562 641336},0x0)            = 0 (0x0)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0xc3,0xa69d1e0,0x2)                       = 515 (0x203)
ioctl(195,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xc3,0xa479000,0x4ff)                       ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 642341},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(1164,FIONREAD,0xbfbfe4fc)                  = 0 (0x0)
read(0x48c,0x9460000,0x5bf)                      = 1460 (0x5b4)
write(58,0x9460000,1460)                         = 1460 (0x5b4)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 656188},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(1164,FIONREAD,0xbfbfe4fc)                  = 0 (0x0)
read(0x48c,0x9460000,0x5bf)                      = 1460 (0x5b4)
write(58,0x9460000,1460)                         = 1460 (0x5b4)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 657195},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
write(15,0x6823706c,1586)                        = 1586 (0x632)
munmap(0x68231000,0x669e)                        = 0 (0x0)
close(28)                                        = 0 (0x0)
ioctl(15,FIONREAD,0xbfbfe4fc)                    = 0 (0x0)
read(0xf,0xa79f000,0x4ff)                        ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 664979},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
accept(0x3,0xbfbfe8b0,0xbfbfe8ac)                ERR#35 'Resource temporarily unavailable'
gettimeofday({1161415562 667978},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 684158},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(150,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x96,0xa773000,0x5bf)                       = 1452 (0x5ac)
write(25,0xa773000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 686152},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 2 (0x2)
ioctl(108,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x6c,0x90f4000,0x47f)                       = 290 (0x122)
gettimeofday({1161415562 696154},0x0)            = 0 (0x0)
gettimeofday({1161415562 696326},0x0)            = 0 (0x0)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0x6c,0xa69d1e0,0x1)                       = 237 (0xed)
open("/usr/local/www/htdocs/p.swf",0x0,01031047720) = 28 (0x1c)
fcntl(28,F_SETFD,0x1)                            = 0 (0x0)
mmap(0x0,26270,(0x1)PROT_READ,(0x1)MAP_SHARED,28,0x0) = 1747128320 (0x68231000)
write(108,0x68231000,26270)                      = 8403 (0x20d3)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 697609},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 698360},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(538,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x21a,0xa7bd000,0xb7f)                      = 1452 (0x5ac)
write(26,0xa7bd000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 700883},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(538,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x21a,0xa7bd000,0xb7f)                      = 1452 (0x5ac)
write(26,0xa7bd000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 722641},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(538,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x21a,0xa7bd000,0xb7f)                      = 1452 (0x5ac)
write(26,0xa7bd000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 743801},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(202,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xca,0xa6f6000,0x53f)                       = 745 (0x2e9)
gettimeofday({1161415562 747520},0x0)            = 0 (0x0)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0xca,0xa69d1e0,0x2)                       = 515 (0x203)
ioctl(202,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xca,0xa6f6000,0x53f)                       ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 748380},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(150,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x96,0xa773000,0x5bf)                       = 1452 (0x5ac)
write(25,0xa773000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 772818},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(1164,FIONREAD,0xbfbfe4fc)                  = 0 (0x0)
read(0x48c,0x9460000,0x5bf)                      = 1460 (0x5b4)
write(58,0x9460000,1460)                         = 1460 (0x5b4)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 773602},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(163,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xa3,0xa76d800,0x4bf)                       = 379 (0x17b)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0xa3,0xa69d1e0,0x2)                       = 515 (0x203)
ioctl(163,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xa3,0xa76d800,0x4bf)                       ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 784113},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(150,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x96,0xa773000,0x5bf)                       = 1452 (0x5ac)
write(25,0xa773000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 788715},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(8,FIONREAD,0xbfbfe4fc)                     = 0 (0x0)
read(0x8,0x9007000,0x47f)                        = 543 (0x21f)
gettimeofday({1161415562 810095},0x0)            = 0 (0x0)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0x8,0xa69d1e0,0x2)                        = 515 (0x203)
ioctl(8,FIONREAD,0xbfbfe4fc)                     = 0 (0x0)
read(0x8,0x9007000,0x47f)                        ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 810856},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(87,FIONREAD,0xbfbfe4fc)                    = 0 (0x0)
read(0x57,0x89cf800,0x4ff)                       = 574 (0x23e)
gettimeofday({1161415562 854431},0x0)            = 0 (0x0)
gettimeofday({1161415562 854669},0x0)            = 0 (0x0)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0x57,0xa69d1e0,0x1)                       = 237 (0xed)
open("/usr/local/www/htdocs/p.swf",0x0,01053313320) = 181 (0xb5)
fcntl(181,F_SETFD,0x1)                           = 0 (0x0)
mmap(0x0,26270,(0x1)PROT_READ,(0x1)MAP_SHARED,181,0x0) = 1747533824 (0x68294000)
write(87,0x68294000,26270)                       = 8403 (0x20d3)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 855958},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
accept(0x3,0xbfbfe8b0,0xbfbfe8ac)                ERR#35 'Resource temporarily unavailable'
gettimeofday({1161415562 862176},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
accept(0x3,0xbfbfe8b0,0xbfbfe8ac)                ERR#35 'Resource temporarily unavailable'
gettimeofday({1161415562 870102},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(175,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xaf,0x9a83000,0x47f)                       = 403 (0x193)
gettimeofday({1161415562 876680},0x0)            = 0 (0x0)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0xaf,0xa69d1e0,0x2)                       = 515 (0x203)
ioctl(175,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xaf,0x9a83000,0x47f)                       ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 877630},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(129,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x81,0x91f4800,0x53f)                       = 552 (0x228)
gettimeofday({1161415562 879434},0x0)            = 0 (0x0)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0x81,0xa69d1e0,0x2)                       = 515 (0x203)
ioctl(129,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x81,0x91f4800,0x53f)                       ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 880348},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(538,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x21a,0xa7bd000,0xb7f)                      = 1452 (0x5ac)
write(26,0xa7bd000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 885411},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(150,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x96,0xa773000,0x5bf)                       = 1452 (0x5ac)
write(25,0xa773000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 907141},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(538,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x21a,0xa7bd000,0xb7f)                      = 1452 (0x5ac)
write(26,0xa7bd000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 907887},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(171,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xab,0x92d4800,0x47f)                       = 303 (0x12f)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0xab,0xa69d1e0,0x2)                       = 515 (0x203)
ioctl(171,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xab,0x92d4800,0x47f)                       ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 911431},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(166,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xa6,0x846e000,0x3ff)                       = 303 (0x12f)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0xa6,0xa69d1e0,0x2)                       = 515 (0x203)
ioctl(166,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xa6,0x846e000,0x3ff)                       ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 921200},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(103,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x67,0x89d0800,0x57f)                       = 593 (0x251)
gettimeofday({1161415562 921586},0x0)            = 0 (0x0)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0x67,0xa69d1e0,0x2)                       = 515 (0x203)
ioctl(103,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x67,0x89d0800,0x57f)                       ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 922422},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
accept(0x3,0xbfbfe8b0,0xbfbfe8ac)                ERR#35 'Resource temporarily unavailable'
gettimeofday({1161415562 923229},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 923859},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(538,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x21a,0xa7bd000,0xb7f)                      = 1452 (0x5ac)
write(26,0xa7bd000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 929167},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
write(30,0x68241386,17176)                       = 4306 (0x10d2)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 960445},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(150,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x96,0xa773000,0x5bf)                       = 1452 (0x5ac)
write(25,0xa773000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 961977},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
accept(0x3,0xbfbfe8b0,0xbfbfe8ac)                ERR#35 'Resource temporarily unavailable'
gettimeofday({1161415562 974207},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(150,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x96,0xa773000,0x5bf)                       = 1452 (0x5ac)
write(25,0xa773000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415562 977678},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
accept(0x3,0xbfbfe8b0,0xbfbfe8ac)                ERR#35 'Resource temporarily unavailable'
gettimeofday({1161415563 28233},0x0)             = 0 (0x0)
wait4(0x1700e,0xbfbfe8c8,0x1,0x0)                ERR#10 'No child processes'
shutdown(0x8c,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(140)                                       = 0 (0x0)
shutdown(0x90,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(144)                                       = 0 (0x0)
shutdown(0x29,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(41)                                        = 0 (0x0)
shutdown(0x108,0x1)                              = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(264)                                       = 0 (0x0)
shutdown(0x7c,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(124)                                       = 0 (0x0)
shutdown(0xdf,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(223)                                       = 0 (0x0)
shutdown(0xd9,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(217)                                       = 0 (0x0)
shutdown(0x7a,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(122)                                       = 0 (0x0)
shutdown(0xfb,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(251)                                       = 0 (0x0)
shutdown(0x77,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(119)                                       = 0 (0x0)
shutdown(0x5e,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(94)                                        = 0 (0x0)
shutdown(0x5a,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(90)                                        = 0 (0x0)
shutdown(0xf9,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(249)                                       = 0 (0x0)
shutdown(0x2d,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(45)                                        = 0 (0x0)
shutdown(0xed,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(237)                                       = 0 (0x0)
shutdown(0x42,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(66)                                        = 0 (0x0)
shutdown(0x87,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(135)                                       = 0 (0x0)
shutdown(0x99,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(153)                                       = 0 (0x0)
shutdown(0x66,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(102)                                       = 0 (0x0)
shutdown(0xee,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(238)                                       = 0 (0x0)
shutdown(0x49,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(73)                                        = 0 (0x0)
shutdown(0x28,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(40)                                        = 0 (0x0)
shutdown(0x27,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(39)                                        = 0 (0x0)
write(4,0x92aa000,199)                           = 199 (0xc7)
write(5,0xa7e6000,2612)                          = 2612 (0xa34)
close(132)                                       = 0 (0x0)
munmap(0x6825b000,0x669e)                        = 0 (0x0)
shutdown(0x7f,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(127)                                       = 0 (0x0)
shutdown(0x60,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(96)                                        = 0 (0x0)
shutdown(0x24,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(36)                                        = 0 (0x0)
shutdown(0x14,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(20)                                        = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(147,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x93,0xa3e7000,0x47f)                       = 533 (0x215)
gettimeofday({1161415563 58746},0x0)             = 0 (0x0)
stat("/usr/local/www/bin/redirect.lua",0xbfbfe300) = 0 (0x0)
open("/usr/local/www/bin/redirect.lua",0x0,00)   = 20 (0x14)
close(20)                                        = 0 (0x0)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0x93,0xa681940,0x2)                       = 515 (0x203)
ioctl(147,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x93,0xa3e7000,0x47f)                       ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 59943},0x0)             = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(179,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xb3,0x9fb0000,0x3ff)                       = 282 (0x11a)
gettimeofday({1161415563 60354},0x0)             = 0 (0x0)
stat("/usr/local/www/htdocs/watch/undefined.html",0xbfbfe410) ERR#2 'No such file or directory'
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0xb3,0xa681940,0x2)                       = 479 (0x1df)
ioctl(179,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xb3,0x9fb0000,0x3ff)                       ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 61340},0x0)             = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(138,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x8a,0x99d9400,0x3ff)                       = 675 (0x2a3)
gettimeofday({1161415563 77630},0x0)             = 0 (0x0)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0x8a,0xa681940,0x2)                       = 515 (0x203)
ioctl(138,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x8a,0x99d9400,0x3ff)                       ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 78459},0x0)             = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
close(181)                                       = 0 (0x0)
munmap(0x68294000,0x669e)                        = 0 (0x0)
shutdown(0x57,0x1)                               = 0 (0x0)
kevent(0x6,0xbfbfe4b0,0x1,0x0,0x0,0xbfbfe4a8)    = 0 (0x0)
close(87)                                        = 0 (0x0)
gettimeofday({1161415563 86534},0x0)             = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
accept(0x3,0xbfbfe8b0,0xbfbfe8ac)                ERR#35 'Resource temporarily unavailable'
gettimeofday({1161415563 103387},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
accept(0x3,0xbfbfe8b0,0xbfbfe8ac)                ERR#35 'Resource temporarily unavailable'
gettimeofday({1161415563 105894},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
accept(0x3,0xbfbfe8b0,0xbfbfe8ac)                ERR#35 'Resource temporarily unavailable'
gettimeofday({1161415563 115967},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 118757},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
accept(0x3,0xbfbfe8b0,0xbfbfe8ac)                ERR#35 'Resource temporarily unavailable'
gettimeofday({1161415563 123430},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
accept(0x3,0xbfbfe8b0,0xbfbfe8ac)                ERR#35 'Resource temporarily unavailable'
gettimeofday({1161415563 124427},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
stat("/usr/local/www/htdocs/p.swf",0xbfbfe400)   = 0 (0x0)
open("/usr/local/www/htdocs/p.swf",0x0,015005501501) = 20 (0x14)
close(20)                                        = 0 (0x0)
write(159,0x6828114b,17747)                      = 4617 (0x1209)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 135354},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(150,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x96,0xa773000,0x5bf)                       = 1452 (0x5ac)
write(25,0xa773000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 150197},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
accept(0x3,0xbfbfe8b0,0xbfbfe8ac)                ERR#35 'Resource temporarily unavailable'
gettimeofday({1161415563 152295},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
accept(0x3,0xbfbfe8b0,0xbfbfe8ac)                ERR#35 'Resource temporarily unavailable'
gettimeofday({1161415563 155517},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(150,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x96,0xa773000,0x5bf)                       = 1452 (0x5ac)
write(25,0xa773000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 166086},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(150,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x96,0xa773000,0x5bf)                       = 1452 (0x5ac)
write(25,0xa773000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 182755},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
write(108,0x682330d3,17867)                      = 4557 (0x11cd)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 183175},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
write(108,0x682342a0,13310)                      = 2880 (0xb40)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 186427},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(194,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xc2,0x9fd6000,0x37f)                       = 534 (0x216)
gettimeofday({1161415563 204060},0x0)            = 0 (0x0)
stat("/usr/local/www/htdocs/",0xbfbfe410)        = 0 (0x0)
stat("/usr/local/www/htdocs/index.php",0xbfbfe390) = 0 (0x0)
open("/usr/local/www/htdocs/index.php",0x0,027757761450) = 20 (0x14)
close(20)                                        = 0 (0x0)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0xc2,0xa681940,0x2)                       = 515 (0x203)
ioctl(194,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xc2,0x9fd6000,0x37f)                       ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 205433},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(157,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x9d,0x99cf800,0x43f)                       = 343 (0x157)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0x9d,0xa681940,0x2)                       = 515 (0x203)
ioctl(157,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x9d,0x99cf800,0x43f)                       ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 213700},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(538,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x21a,0xa7bd000,0xb7f)                      = 1452 (0x5ac)
write(26,0xa7bd000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 214275},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(145,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x91,0x9442400,0x3bf)                       = 219 (0xdb)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0x91,0xa681940,0x2)                       = 515 (0x203)
ioctl(145,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x91,0x9442400,0x3bf)                       ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 220315},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(62,FIONREAD,0xbfbfe4fc)                    = 0 (0x0)
read(0x3e,0x897a800,0x57f)                       = 343 (0x157)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0x3e,0xa681940,0x2)                       = 515 (0x203)
ioctl(62,FIONREAD,0xbfbfe4fc)                    = 0 (0x0)
read(0x3e,0x897a800,0x57f)                       ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 221401},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(1164,FIONREAD,0xbfbfe4fc)                  = 0 (0x0)
read(0x48c,0x9460000,0x5bf)                      = 320 (0x140)
write(58,0x9460000,320)                          = 320 (0x140)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 224699},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 225781},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(538,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x21a,0xa7bd000,0xb7f)                      = 1452 (0x5ac)
write(26,0xa7bd000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 235453},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(150,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x96,0xa773000,0x5bf)                       = 1452 (0x5ac)
write(25,0xa773000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 252412},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(538,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x21a,0xa7bd000,0xb7f)                      = 1452 (0x5ac)
write(26,0xa7bd000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 255330},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
accept(0x3,0xbfbfe8b0,0xbfbfe8ac)                ERR#35 'Resource temporarily unavailable'
gettimeofday({1161415563 263432},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(63,FIONREAD,0xbfbfe4fc)                    = 0 (0x0)
read(0x3f,0x8887800,0x3ff)                       = 775 (0x307)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0x3f,0xa681940,0x2)                       = 515 (0x203)
ioctl(63,FIONREAD,0xbfbfe4fc)                    = 0 (0x0)
read(0x3f,0x8887800,0x3ff)                       ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 265124},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(150,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x96,0xa773000,0x5bf)                       = 1452 (0x5ac)
write(25,0xa773000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 268442},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(150,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x96,0xa773000,0x5bf)                       = 1452 (0x5ac)
write(25,0xa773000,1452)                         = 1452 (0x5ac)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 285308},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(155,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x9b,0x955f000,0x4ff)                       = 563 (0x233)
gettimeofday({1161415563 317241},0x0)            = 0 (0x0)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0x9b,0xa681940,0x2)                       = 515 (0x203)
ioctl(155,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0x9b,0x955f000,0x4ff)                       ERR#35 'Resource temporarily unavailable'
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
kevent(0x6,0xbfbfe4d0,0x1,0x0,0x0,0xbfbfe4c8)    = 0 (0x0)
gettimeofday({1161415563 318077},0x0)            = 0 (0x0)
kevent(0x6,0x0,0x0,0x80f5000,0x17ff7,0xbfbfe950) = 1 (0x1)
ioctl(171,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xab,0x92d4800,0x47f)                       = 303 (0x12f)
__sysctl(0xbfbfe448,0x2,0xbfbfe440,0xbfbfe444,0x0,0x0) = 0 (0x0)
writev(0xab,0xa681940,0x2)                       = 515 (0x203)
ioctl(171,FIONREAD,0xbfbfe4fc)                   = 0 (0x0)
read(0xab,0x92d4800,0x47f)                       ERR#35 'Resource temporarily unavailable'

-- namosys

#1

Updated by Anonymous over 13 years ago

It seemed that lighty failed to connect to php-fastcgi. I saw

switch(fcgi_write_request(srv, hctx)) {
case HANDLER_ERROR:
:
:
:
if (hctx->reconnects < 5) {
fcgi_reconnect(srv, hctx);
joblist_append(srv, con); /* in case we come from the event-handler */
return HANDLER_WAIT_FOR_FD;
} else {
fcgi_connection_close(srv, hctx);
buffer_reset(con->physical.path);
con->mode = DIRECT;
con->http_status = 500;
joblist_append(srv, con); /* in case we come from the event-handler */
return HANDLER_FINISHED;
}

-- namosys

#2

Updated by Anonymous over 13 years ago

I closed zlib output compression in php.ini, after a restart, lighty runs perfectly for about TWO hours w/o any problems.

-- namosys

#3

Updated by Anonymous over 13 years ago

After running 6hrs, lighty halted again w/o php's zlib output compression.

I did a @`@tail -f access.log | grep ' 500 '`` trace. When lighty halted, there would be tons of '500's in the access log and FEW normal replies (200,206,301,304, and even 404, etc.).

It seems the communication between lighty and php/fastcgi has some trouble.

-- namosys

#4

Updated by Anonymous over 13 years ago

I added extra error logs in mod_fastcgi.c where it sends 500 to the client, and in lighty.error.log I finally found:


[ honeymoon period ]
2006-10-22 01:30:48: (mod_fastcgi.c.3005) ndx==-1  
2006-10-22 01:30:49: (mod_fastcgi.c.3005) ndx==-1  
2006-10-22 01:30:58: (mod_fastcgi.c.3005) ndx==-1  
2006-10-22 01:31:05: (mod_fastcgi.c.3005) ndx==-1  
[ tragedy period ]
[ I restarted lighty then ]
[ honeymoon period ]
2006-10-22 10:15:51: (mod_fastcgi.c.3005) ndx==-1  
2006-10-22 10:15:52: (mod_fastcgi.c.3005) ndx==-1 
[ tragedy period again ]

So, the problem perhaps happens at:


        if (hctx->host == NULL) {
                size_t k;
                int ndx, used = -1;

                /* get best server */
                for (k = 0, ndx = -1; k < hctx->ext->used; k++) {
                        host = hctx->ext->hosts[k];

                        /* we should have at least one proc that can do something */
                        if (host->active_procs == 0) continue;

                        if (used == -1 || host->load < used) {
                                used = host->load;

                                ndx = k;
                        }
                }       

                /* found a server */
                if (ndx == -1) {
                        /* all hosts are down */

                        fcgi_connection_close(srv, hctx);

                        con->http_status = 500;
                        con->mode = DIRECT;

log_error_write(srv,__FILE__,__LINE__,"ss","ndx==-1");

                        return HANDLER_FINISHED;
                }

Maybe lighty cannot find a fcgi host to work because


                        if (host->active_procs == 0) continue;

All hosts are not activated in lighty's eyes?

-- namosys

#5

Updated by Anonymous over 13 years ago

I've tried the reconnect thresold from 5 to 50000 and disable_time from 60 to 1.

-- namosys

#6

Updated by Anonymous over 13 years ago

Failed again. The error log reads:


2006-10-22 16:15:25: (mod_fastcgi.c.1705) connect failed: Connection refused on unix:/tmp/php1.socket-0
2006-10-22 16:15:25: (mod_fastcgi.c.2817) backend died, we disable it for a 5 seconds and send the request to another backend instead: reconnects: 0 load: 1447
2006-10-22 16:15:25: (mod_fastcgi.c.3040) FCGI_STATE_INIT FCGI_STATE_CONNECT_DELAYED, active_procs--
2006-10-22 16:15:25: (mod_fastcgi.c.3048) HANDLER_ERROR, reconnects<50000 -2078608690
2006-10-22 16:15:25: (mod_fastcgi.c.3006) ndx==-1             
2006-10-22 16:15:25: (mod_fastcgi.c.3430) host==NULL
2006-10-22 16:15:25: (mod_fastcgi.c.3436) all handlers for  /w.php on .php are down.
2006-10-22 16:15:25: (mod_fastcgi.c.3430) host==NULL
2006-10-22 16:15:25: (mod_fastcgi.c.3430) host==NULL
2006-10-22 16:15:25: (mod_fastcgi.c.3430) host==NULL    
2006-10-22 16:15:25: (mod_fastcgi.c.3430) host==NULL
2006-10-22 16:15:25: (mod_fastcgi.c.3430) host==NULL    
2006-10-22 16:15:25: (mod_fastcgi.c.3430) host==NULL

-- namosys

#7

Updated by Anonymous over 13 years ago

Is there a backup mechanism to re-create fastcgi servers when current servers are dead?

-- namosys

#8

Updated by over 13 years ago

During testing, do not enable kern.timecounter.hardware=TSC, TSC on SMP is a little unreliable and quirky. Also, paste your dmesg so we can see what the kernel thinks.

#9

Updated by stbuehler about 12 years ago

  • Status changed from New to Fixed
  • Resolution set to invalid

Your backends died - so you get 500 of course. Lighttpd should restart the backends after 5 secs.

#10

Updated by stbuehler almost 12 years ago

  • Status changed from Fixed to Invalid

Also available in: Atom