Project

General

Profile

FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!!

Added by francisco almost 8 years ago

Hi All.

I need some help here.

I used to use Apache 1.3, but now, that I have a new server,
I migrate my website to Lighty, BUT IT IS FREEZING !!
One, two, three times a day it locks down on lighty rising CPU usage to 100%!

It seems to be something related to lighttpd + php fast-cgi.

On 2009-08-24, it stopped at 19:30 according to CACTI graphs that shows 100% CPU Utilization
- it jumped from normal 2-5 cpu (at 19:25) to 100% (at 19:30).

When the server locks, I try to KILL -9 lighttpd process, but it just dont work!!!
"Waiting for PIDS: 7748 7748 7748 7748 7748 7748 7748 7748 7748 7748 7748..."
(and never stops)

-------------------------------------------------
Some info about my scenario:
-------------------------------------------------

FreeBSD 7.2-RELEASE #0: Tue Jul 28 19:36:22 BRT 2009
root@server.com :/usr/src/sys/amd64/compile/www
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU E5410 @ 2.33GHz (2327.51-MHz K8-class CPU)

Origin = "GenuineIntel"  Id = 0x1067a  Stepping = 10
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8
 ,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36
 ,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x40ce3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST
  ,TM2,SSSE3,CX16,xTPR,PDCM,DCA,<b19>,XSAVE>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>

Cores per package: 4
usable memory = 4276776960 (4078 MB)
avail memory  = 4104232960 (3914 MB)
ACPI APIC Table: <DELL   PE_SC3  >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
 cpu4 (AP): APIC ID:  4
 cpu5 (AP): APIC ID:  5
 cpu6 (AP): APIC ID:  6
 cpu7 (AP): APIC ID:  7
-------------------------------------------------
mysql-server-5.1.36
xcache-1.2.2
php5-5.2.10
lighttpd-1.4.23
-------------------------------------------------

When I restarted lighttpd, FREE cmd shows:

mem_wire:         507826176 (    484MB) [ 12%]
mem_active:  +    308924416 (    294MB) [  7%]
mem_inactive:+    360247296 (    343MB) [  8%]
mem_cache:   +      5332992 (      5MB) [  0%]
mem_free:    +   2943963136 (   2807MB) [ 71%]
mem_gap_vm:  +       487424 (      0MB) [  0%]

17 hours after last LOCK DOWN, free cmd shows:

mem_wire:         671326208 (    640MB) [ 16%]
mem_active:  +    472195072 (    450MB) [ 11%]
mem_inactive:+   1835315200 (   1750MB) [ 44%]
mem_cache:   +      2838528 (      2MB) [  0%]
mem_free:    +   1144553472 (   1091MB) [ 27%]
mem_gap_vm:  +       552960 (      0MB) [  0%]

Server-status and Server-statistics reports:

Uptime 16 hours 53 s 
Started at 2009-08-24 21:36:35 

absolute (since start) 
Requests 245 kreq 
Traffic 3.46 Gbyte 

average (since start) 
Requests 4 req/s 
Traffic 62.99 kbyte/s 

average (5s sliding average) 
Requests 18 req/s 
Traffic 161.91 kbyte/s 
--------------------------------
fastcgi.active-requests: 0
fastcgi.backend.localhost.0.connected: 676
fastcgi.backend.localhost.0.died: 0
fastcgi.backend.localhost.0.disabled: 0
fastcgi.backend.localhost.0.load: 0
fastcgi.backend.localhost.0.overloaded: 0
fastcgi.backend.localhost.1.connected: 2921
fastcgi.backend.localhost.1.died: 0
fastcgi.backend.localhost.1.disabled: 0
fastcgi.backend.localhost.1.load: 0
fastcgi.backend.localhost.1.overloaded: 0
fastcgi.backend.localhost.2.connected: 13352
fastcgi.backend.localhost.2.died: 0
fastcgi.backend.localhost.2.disabled: 0
fastcgi.backend.localhost.2.load: 0
fastcgi.backend.localhost.2.overloaded: 0
fastcgi.backend.localhost.3.connected: 69502
fastcgi.backend.localhost.3.died: 0
fastcgi.backend.localhost.3.disabled: 0
fastcgi.backend.localhost.3.load: 0
fastcgi.backend.localhost.3.overloaded: 0
fastcgi.backend.localhost.load: 0
fastcgi.requests: 86451

lighttpd.error.log shows NOTHING more than the usual "Connection reset by peer" (crazy error that I just cant get rid :-( ):

2009-08-24 15:33:20: (connections.c.132) (warning) close: 26 Connection reset by peer
2009-08-24 16:38:39: (connections.c.132) (warning) close: 23 Connection reset by peer
2009-08-24 16:47:54: (connections.c.132) (warning) close: 20 Connection reset by peer

At 19:30 whet I realized it was locked down, I tried KILL cmd.
But kill does not worked, so at 20:31 I did a "reboot" command on ssh.

2009-08-24 20:32:30: (log.c.172) server started
2009-08-24 20:52:53: (server.c.1495) server stopped by UID = 0 PID = 0
2009-08-24 20:53:16: (log.c.172) server started
2009-08-24 21:36:35: (log.c.172) server started
2009-08-24 21:37:50: (connections.c.132) (warning) close: 25 Connection reset by peer
2009-08-25 10:58:26: (connections.c.132) (warning) close: 38 Connection reset by peer

/var/log/php-error.log: I know I have this syntax error somewhere, but I dont believe it be the cause of 100% utilisation of CPU.

[24-Aug-2009 18:33:05] PHP Parse error:  syntax error, unexpected T_DOUBLE_ARROW, expecting ')' in /usr/dangelina/inetsrv/include/hylibnet.mdl(1437) : eval()
[24-Aug-2009 *19:27:58*] PHP Parse error:  syntax error, unexpected T_DOUBLE_ARROW, expecting ')' in /usr/dangelina/inetsrv/include/hylibnet.mdl(1437) : eval()
[24-Aug-2009 *20:55:56*] PHP Parse error:  syntax error, unexpected T_DOUBLE_ARROW, expecting ')' in /usr/dangelina/inetsrv/include/hylibnet.mdl(1437) : eval()

Nothing usefull on /var/log/messages either.

/usr/local/etc/lighttpd.conf

## My trying Tweaks 
server.max-keep-alive-requests = 4
server.max-keep-alive-idle = 4
server.max-fds = 6144
server.max-connections = 2048
server.max-read-idle = 20
server.max-write-idle = 180

fastcgi.server             = ( ".php" =>
                               ( "localhost" =>
                                 (
                                   "socket" => "/var/run/lighttpd/php-fastcgi.socket",
                                   "bin-path" => "/usr/local/bin/php-cgi",
                                   "max-procs"         => 4,
                                   "min-procs"         => 1,
                                   "max-load-per-proc" => 10,
                                   "idle-timeout"      => 20,
                                   "bin-environment" => (
                                                "PHP_FCGI_CHILDREN" => "2",
                                                "PHP_FCGI_MAX_REQUESTS" => "500" 
                                                        ),
                                   "bin-copy-environment" => (""),
                                   "broken-scriptfilename" => "enable" 
                                 )
                               )
                            )

Thank you in advance for any directions, any help...
Francisco

www-crash-2009-08-29-19-30.jpg (351 KB) www-crash-2009-08-29-19-30.jpg CACTI graphs at instance of last crash

Replies (19)

RE: FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!! - Added by stbuehler almost 8 years ago

If kill -9 doesn't work there is something wrong with the kernel; a process can't block that signal.

RE: FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!! - Added by icy almost 8 years ago

Can you provide a strace/truss log of lighty when it goes to 100% cpu usage?

RE: FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!! - Added by francisco almost 8 years ago

Hi, looking into logs, one more time, I think I found something: kern.maxfiles...

So I tried:
ee /etc/sysctl.conf
--------------------------
sysctl kern.maxfiles=20000
kern.ipc.somaxconn=4096
--------------------------

Aug 24 14:38:53 www kernel: kern.maxfiles limit exceeded by uid 80, please see tuning(7).
Aug 24 14:38:54 www kernel: kern.maxfiles limit exceeded by uid 80, please see tuning(7).
Aug 24 14:38:57 www kernel: pid 73688 (php-cgi), uid 80: exited on signal 11 (core dumped)
Aug 24 14:38:58 www kernel: pid 73684 (php-cgi), uid 80: exited on signal 11 (core dumped)
Aug 24 15:00:42 www kernel: kern.maxfiles limit exceeded by uid 80, please see tuning(7).
Aug 24 15:00:42 www kernel: pid 73780 (php-cgi), uid 80: exited on signal 11 (core dumped)
Aug 24 15:00:44 www kernel: pid 73682 (php-cgi), uid 80: exited on signal 11 (core dumped)
Aug 24 15:00:46 www kernel: pid 73781 (php-cgi), uid 80: exited on signal 11 (core dumped)

Aug 24 20:29:34 www su: francisco to root on /dev/ttyp1
*Aug 24 20:30:03 www reboot: rebooted by francisco*

Aug 24 20:30:03 www syslogd: exiting on signal 15
Aug 24 20:31:54 www syslogd: kernel boot file is /boot/kernel/kernel
Aug 25 23:27:56 www kernel: pid 25289 (php-cgi), uid 80: exited on signal 11 (core dumped)
Aug 25 23:28:57 www su: francisco to root on /dev/ttyp2
Aug 25 23:30:07 www kernel: kern.maxfiles limit exceeded by uid 80, please see tuning(7).
Aug 25 23:30:10 www kernel: pid 25290 (php-cgi), uid 80: exited on signal 11 (core dumped)
Aug 25 23:30:11 www kernel: pid 25206 (php-cgi), uid 80: exited on signal 11 (core dumped)
Aug 25 23:30:15 www kernel: pid 25205 (php-cgi), uid 80: exited on signal 11 (core dumped)
Aug 25 23:30:16 www kernel: pid 25333 (php-cgi), uid 80: exited on signal 11 (core dumped)
Aug 25 23:30:22 www kernel: pid 25350 (php-cgi), uid 80: exited on signal 11 (core dumped)
Aug 25 23:30:29 www kernel: pid 25353 (php-cgi), uid 80: exited on signal 11 (core dumped)
Aug 25 23:30:31 www kernel: pid 25351 (php-cgi), uid 80: exited on signal 11 (core dumped)
Aug 25 23:30:33 www kernel: kern.maxfiles limit exceeded by uid 80, please see tuning(7).
Aug 25 23:30:35 www kernel: pid 25355 (php-cgi), uid 80: exited on signal 11 (core dumped)
Aug 25 23:30:36 www kernel: pid 25352 (php-cgi), uid 80: exited on signal 11 (core dumped)
Aug 25 23:30:38 www kernel: pid 25357 (php-cgi), uid 80: exited on signal 11 (core dumped)
Aug 25 23:30:43 www kernel: pid 25360 (php-cgi), uid 80: exited on signal 11 (core dumped)
Aug 25 23:30:44 www kernel: pid 25359 (php-cgi), uid 80: exited on signal 11 (core dumped)
Aug 25 23:31:36 www kernel: pid 25362 (php-cgi), uid 80: exited on signal 11 (core dumped)
Aug 25 23:31:54 www kernel: kern.maxfiles limit exceeded by uid 80, please see tuning(7).
Aug 25 23:31:56 www last message repeated 2 times
Aug 25 23:31:57 www kernel: pid 25370 (php-cgi), uid 80: exited on signal 11 (core dumped)
Aug 25 23:31:57 www kernel: kern.maxfiles limit exceeded by uid 80, please see tuning(7).
Aug 25 23:31:58 www kernel: pid 25364 (php-cgi), uid 80: exited on signal 11 (core dumped)

Aug 26 16:43:22 www su: francisco to root on /dev/ttyp0
*Aug 26 16:45:35 www reboot: rebooted by francisco*

Aug 26 16:45:35 www syslogd: exiting on signal 15
Aug 26 16:47:27 www syslogd: kernel boot file is /boot/kernel/kernel
Aug 26 16:47:27 www savecore: no dumps found
Aug 26 16:48:12 www kernel: mfi0: 1055 (304620493s/0x0008/info) - Battery started charging
Aug 26 17:03:22 www kernel: mfi0: 1056 (304621403s/0x0008/info) - Battery charge complete

Aug 26 18:10:51 www su: francisco to root on /dev/ttyp0
*Aug 26 18:12:02 www reboot: rebooted by francisco*

Aug 26 18:12:02 www syslogd: exiting on signal 15
Aug 26 18:13:52 www syslogd: kernel boot file is /boot/kernel/kernel
  • no warnings about kern.maxfiles on crash at 18:12:02 :-(

RE: FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!! - Added by JamesKay almost 8 years ago

server.event-handler = "poll"
server.network-backend = "write"

try that in lighttpd .. seems to have helped us.

RE: FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!! - Added by stbuehler almost 8 years ago

And it looks like you should fix your php installation...

RE: FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!! - Added by spaam almost 8 years ago

francisco wrote:

Hi, looking into logs, one more time, I think I found something: kern.maxfiles...

So I tried:
ee /etc/sysctl.conf
--------------------------
sysctl kern.maxfiles=20000
kern.ipc.somaxconn=4096
--------------------------

[...]

  • no warnings about kern.maxfiles on crash at 18:12:02 :-(

looks like your php-cgi is broken.. maybe one module or so..

RE: FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!! - Added by francisco almost 8 years ago

I tried:
server.event-handler = "poll"
server.network-backend = "write"

No lucky yet ... my lighttpd continues to freeze without any reason - no usefull messages on logs.

I noticed that in worst cases one cpu goes to 100% utilisation and lighttpd stops to respond.
In that cases, "stop" does not work. And Kill does not work too.

Few minutes ago, my website just stopped to respond, just dont open on webbrowsers.
I did:
/usr/local/etc/rc.d/lighttpd stop
and for my lucky, it stopped
so, I did:
/usr/local/etc/rc.d/lighttpd start
and it started and web browsers now can access again

My actual configs are:

1) lighttpd.conf

server.max-keep-alive-requests = 4
server.max-keep-alive-idle = 4
server.max-fds = 2048
server.max-connections = 412
server.max-read-idle = 20
server.max-write-idle = 180

server.event-handler = "select" 
server.network-backend = "freebsd-sendfile" 
net.inet.tcp.recvspace = 4096

...

fastcgi.server             = ( ".php" =>
( "localhost" =>
(
"socket" => "/var/run/lighttpd/php-fastcgi.socket",
"bin-path" => "/usr/local/bin/php-cgi",
"max-procs"         => 4,
"min-procs"         => 1,
"max-load-per-proc" => 10,
"idle-timeout"      => 20,
"bin-environment" => (
"PHP_FCGI_CHILDREN" => "2",
"PHP_FCGI_MAX_REQUESTS" => "500" 
),
"bin-copy-environment" => (""),
"broken-scriptfilename" => "enable" 
)
)
)

2) cat /etc/sysctl.conf

net.inet.icmp.icmplim=150
net.inet.icmp.maskrepl=0
net.inet.icmp.drop_redirect=1
net.inet.icmp.bmcastecho=0
net.inet.tcp.icmp_may_rst=0

net.inet.tcp.drop_synfin=1
kern.ipc.somaxconn=4096
net.inet.ip.fw.one_pass=1
net.inet.tcp.msl=7500
net.inet.ip.stealth=0

net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1

net.inet.ip.rtexpire=2
net.inet.ip.rtminexpire=2
net.inet.ip.rtmaxcache=256

net.inet.ip.accept_sourceroute=0
net.inet.ip.sourceroute=0

kern.ipc.shmmax=134217728
kern.ipc.shmall=32768
kern.ipc.semmap=256

kern.maxfiles=65536

kern.ipc.maxsockets="131072" 
vm.pmap.pg_ps_enabled=1
vm.kmem_size=1G
kern.ipc.nmbclusters=32768
security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
security.bsd.unprivileged_read_msgbuf=0
kern.maxfilesperproc=52428
kern.threads.max_threads_per_proc=4096
net.inet.tcp.keepinit=1000
net.inet.ip.portrange.last=65535
net.inet.ip.portrange.first=10000@

3) some packages with php I have installed:

# pkg_info | grep -i php
php5-5.2.10         PHP Scripting Language
php5-ctype-5.2.10   The ctype shared extension for php
php5-dbase-5.2.10   The dbase shared extension for php
php5-dom-5.2.10     The dom shared extension for php
php5-exif-5.2.10    The exif shared extension for php
php5-extensions-1.3 A "meta-port" to install PHP extensions
php5-filter-5.2.10  The filter shared extension for php
php5-gd-5.2.10      The gd shared extension for php
php5-hash-5.2.10    The hash shared extension for php
php5-iconv-5.2.10   The iconv shared extension for php
php5-json-5.2.10    The json shared extension for php
php5-mysql-5.2.10   The mysql shared extension for php
php5-mysqli-5.2.10  The mysqli shared extension for php
php5-openssl-5.2.10 The openssl shared extension for php
php5-pcre-5.2.10    The pcre shared extension for php
php5-pdo-5.2.10     The pdo shared extension for php
php5-pdo_sqlite-5.2.10 The pdo_sqlite shared extension for php
php5-posix-5.2.10   The posix shared extension for php
php5-session-5.2.10 The session shared extension for php
php5-simplexml-5.2.10 The simplexml shared extension for php
php5-snmp-5.2.10    The snmp shared extension for php
php5-sockets-5.2.10 The sockets shared extension for php
php5-spl-5.2.10     The spl shared extension for php
php5-sqlite-5.2.10  The sqlite shared extension for php
php5-tokenizer-5.2.10 The tokenizer shared extension for php
php5-xml-5.2.10     The xml shared extension for php
php5-xmlreader-5.2.10 The xmlreader shared extension for php
php5-xmlwriter-5.2.10 The xmlwriter shared extension for php
php5-zip-5.2.10     The zip shared extension for php
php5-zlib-5.2.10    The zlib shared extension for php
xcache-1.2.2        A fast and stable php opcode cacher@

Could you point me some tip on why "it looks like my php-cgi is broken" and where do I check to confirm that?

Thank you for any help

RE: FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!! - Added by spaam almost 8 years ago

Try this:
/path/to/php-cgi -m; echo $?

Does it return 0?

RE: FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!! - Added by francisco almost 8 years ago

yes... last info is zero, see:

www# /usr/local/bin/php -m ; echo $?

[PHP Modules]
ctype
date
dbase
dom
exif
fileinfo
filter
gd
hash
iconv
json
libxml
mysql
mysqli
openssl
pcre
PDO
pdo_sqlite
posix
Reflection
session
SimpleXML
snmp
sockets
SPL
SQLite
standard
tokenizer
XCache
xml
xmlreader
xmlwriter
zip
zlib

[Zend Modules]
XCache

0
www#

RE: FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!! - Added by mr_bond almost 8 years ago

Hi.

Please provide output of "ps axl", from when lighttpd is actually hanging. please provide this output of PS, before AND after "killall -9".
Cacti graphs give me, no useful information about this issue.

Have you built a custom kernel? What kind of CFLAGS do you have in /etc/make.conf

For the record, I'm running a lot of lighttpds under various FreeBSD versions.
FreeBSD-sendfile has been very buggy in lighttpd for me, so I use "writev" as network-backend. I
use kqueue as event-handler.

RE: FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!! - Added by francisco almost 8 years ago

Hi.

www# cat /etc/make.conf

PERL_VERSION=5.8.9

To make my kernel, I did this:
1) created www file from GENERIC

cd /usr/src/sys/amd64/conf
cp GENERIC www

2) added this at end of www:

options         IPFIREWALL        
options         IPFIREWALL_VERBOSE
options         IPFIREWALL_FORWARD
options         IPFIREWALL_VERBOSE_LIMIT=1000
options         IPDIVERT
options         IPFILTER
options         IPFILTER_LOG
options         ACCEPT_FILTER_HTTP
options         DUMMYNET

3) Compiled kernel www:

config www
cd ../compile/www
make cleandepend && make depend
make && make install

Output of PS command right now:

www# ps axl

  UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN STAT  TT       TIME COMMAND
    0     0     0   0 -16  0     0     0 sched  DLs   ??    0:00.38 [swapper]
    0     1     0   0   8  0  2180   528 wait   SLs   ??    0:00.04 /sbin/init --
    0     2     0   0  -8  0     0    16 -      DL    ??    0:02.83 [g_event]
    0     3     0   0  -8  0     0    16 -      DL    ??    0:16.40 [g_up]
    0     4     0   0  -8  0     0    16 -      DL    ??    0:11.68 [g_down]
    0     5     0   0   8  0     0    16 -      DL    ??    0:00.00 [acpi_task_0]
    0     6     0   0   8  0     0    16 -      DL    ??    0:00.00 [acpi_task_1]
    0     7     0   0   8  0     0    16 -      DL    ??    0:00.00 [acpi_task_2]
    0     8     0   0   8  0     0    16 -      DL    ??    0:00.00 [thread taskq]
    0     9     0   0  -8  0     0    16 ccb_sc DL    ??    0:00.00 [xpt_thrd]
    0    10     0   0 -16  0     0    16 audit_ DL    ??    0:00.00 [audit]
    0    11     0   0 171  0     0    16 -      RL    ??  1328:41.36 [idle: cpu7]
    0    12     0   0 171  0     0    16 -      RL    ??  1327:23.72 [idle: cpu6]
    0    13     0   0 171  0     0    16 -      RL    ??  1326:59.74 [idle: cpu5]
    0    14     0   0 171  0     0    16 -      RL    ??  1323:58.93 [idle: cpu4]
    0    15     0   0 171  0     0    16 -      RL    ??  1313:13.24 [idle: cpu3]
    0    16     0   0 171  0     0    16 -      RL    ??  1304:02.69 [idle: cpu2]
    0    17     0   0 171  0     0    16 -      RL    ??  1284:36.69 [idle: cpu1]
    0    18     0   0 171  0     0    16 -      RL    ??  1283:59.94 [idle: cpu0]
    0    19     0   0 -44  0     0    16 -      WL    ??    0:00.28 [swi1: net]
    0    20     0   0 -32  0     0    16 -      WL    ??    4:09.87 [swi4: clock sio]
    0    21     0   0 -36  0     0    16 -      WL    ??    0:00.00 [swi3: vm]
    0    22     0   0 -16  0     0    16 -      DL    ??    0:06.41 [yarrow]
    0    23     0   0 -28  0     0    16 -      WL    ??    0:00.00 [swi5: +]
    0    24     0   0 -24  0     0    16 -      WL    ??    0:00.00 [swi6: Giant taskq]
    0    25     0   0 -24  0     0    16 -      WL    ??    0:00.00 [swi6: task queue]
    0    26     0   0 -40  0     0    16 -      WL    ??    0:00.00 [swi2: cambio]
    0    27     0   0   8  0     0    16 -      DL    ??    0:00.00 [kqueue taskq]
    0    28     0   0 -52  0     0    16 -      WL    ??    0:00.00 [irq9: acpi0]
    0    29     0   0 -68  0     0    16 -      WL    ??    0:00.22 [irq256: bce0]
    0    30     0   0 -64  0     0    16 -      WL    ??    0:08.36 [irq16: mfi0]
    0    31     0   0 -68  0     0    16 -      WL    ??    1:11.09 [irq257: bce1]
    0    32     0   0 -64  0     0    16 -      WL    ??    0:00.00 [irq21: uhci0 uhci+]
    0    33     0   0   8  0     0    16 usbevt DL    ??    0:00.01 [usb0]
    0    34     0   0   8  0     0    16 usbtsk DL    ??    0:00.00 [usbtask-hc]
    0    35     0   0   8  0     0    16 usbtsk DL    ??    0:00.00 [usbtask-dr]
    0    36     0   0 -64  0     0    16 -      WL    ??    0:00.00 [irq20: uhci1 uhci3]
    0    37     0   0   8  0     0    16 usbevt DL    ??    0:00.01 [usb1]
    0    38     0   0   8  0     0    16 usbevt DL    ??    0:00.01 [usb2]
    0    39     0   0   8  0     0    16 usbevt DL    ??    0:00.00 [usb3]
    0    40     0   0   8  0     0    16 usbevt DL    ??    0:00.01 [usb4]
    0    41     0   0 -48  0     0    16 -      WL    ??    0:00.00 [swi0: sio]
    0    42     0   0 -60  0     0    16 -      WL    ??    0:00.00 [irq1: atkbd0]
    0    43     0   0 -16  0     0    24 waitin DL    ??    0:00.00 [sctp_iterator]
    0    44     0   0 -16  0     0    16 psleep DL    ??    0:00.07 [pagedaemon]
    0    45     0   0  20  0     0    16 psleep DL    ??    0:00.00 [vmdaemon]
    0    46     0   0 171  0     0    16 pgzero DL    ??    0:00.00 [pagezero]
    0    47     0   0 -16  0     0    16 psleep DL    ??    0:00.43 [bufdaemon]
    0    48     0   0  20  0     0    16 syncer DL    ??    1:57.18 [syncer]
    0    49     0   0  -4  0     0    16 vlruwt DL    ??    0:00.40 [vnlru]
    0    50     0   0 -16  0     0    16 sdflus DL    ??    0:06.42 [softdepflush]
    0   136     1   0  20  0  2604   940 pause  Is    ??    0:00.00 adjkerntz -i
    0   487     1   0  59  0  2180   640 select Is    ??    0:00.00 /sbin/devd
    0   594     1   0  44  0  5712  1408 select Is    ??    0:00.12 /sbin/routed -q
    0   644     1   0  44  0  5728  1396 select Ss    ??    0:00.27 /usr/sbin/syslogd -l /var/run/log -l /var/named/var/run/log -s
    0   732     1   0  44  0  6784  1508 select Ss    ??    0:00.06 /usr/sbin/rpcbind
    0   748     1   0  44  0  4644  2860 select Is    ??    0:00.01 /usr/sbin/amd -p -a /.amd_mnt -l syslog /host /etc/amd.map /net /etc/amd.map
    0   795     1   0  44  0 28412  8080 select S     ??    3:01.69 /usr/local/sbin/snmpd -p /var/run/snmpd.pid
    0   811     1   0  44  0 15656  3236 select Ss    ??    0:00.71 /usr/local/sbin/nmbd -D -s /usr/local/etc/smb.conf
    0   815     1   0  44  0 20036  4964 select Is    ??    0:00.01 /usr/local/sbin/smbd -D -s /usr/local/etc/smb.conf
    0   861   815   0  44  0 20036  4884 select I     ??    0:00.00 /usr/local/sbin/smbd -D -s /usr/local/etc/smb.conf
    0   873     1   0  44  0 17064  4360 select Ss    ??    0:00.34 proftpd: (accepting connections) (proftpd)
    0   893     1   0  44  0 12884  4948 select Ss    ??    0:07.89 /usr/local/sbin/openvpn --cd /usr/local/etc/openvpn --daemon openvpn --config /usr/local
    0  1134     1   0  44  0 22916  3920 select Is    ??    0:00.00 /usr/sbin/sshd
    0  1140     1   0  44  0 10864  4228 select Ss    ??    0:01.38 sendmail: accepting connections (sendmail)
   25  1144     1   0  20  0 10864  3960 pause  Is    ??    0:00.02 sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail)
    0  1150     1   0   8  0  6784  1504 nanslp Is    ??    0:00.14 /usr/sbin/cron -s
    0  1216     1   0  96  0  7836  1572 select Is    ??    0:00.00 /usr/sbin/inetd -wW -C 60
    0  5496  1134   0   4  0 33772  4728 sbwait Is    ??    0:00.03 sshd: adminuser [priv] (sshd)
 1001  5499  5496   0  44  0 33772  4852 select S     ??    0:00.22 sshd: adminuser@ttyp0 (sshd)
   80  5527     1   0  44  0 33704  9740 select S     ??    0:46.55 /usr/local/sbin/lighttpd -f /usr/local/etc/lighttpd.conf
   80  5528  5527   0   8  0 338828 14812 wait   Is    ??    0:00.06 /usr/local/bin/php-cgi
   80  5529  5527   0   8  0 338828 14812 wait   Is    ??    0:00.06 /usr/local/bin/php-cgi
   80  5534  5527   0   8  0 338828 14812 wait   Is    ??    0:00.10 /usr/local/bin/php-cgi
   80  5537  5527   0   8  0 338828 14812 wait   Is    ??    0:00.05 /usr/local/bin/php-cgi
   80  5539  5537   0   4  0 343052 35808 sbwait I     ??    0:05.79 /usr/local/bin/php-cgi
   80  5540  5527   0  44  0     0     0 -      Z     ??    0:00.01 <defunct>
   80  5541  5527   0   8  0  7100  1760 wait   I     ??    0:00.01 sh -c /usr/local/sbin/cronolog /var/log/light/mysite.com.au/access-%Y%m%d.log
   80  5542  5527   0   8  0  7100  1760 wait   I     ??    0:00.01 sh -c /usr/local/sbin/cronolog /var/log/light/mysite.com/access-%Y%m%d.log
   80  5543  5527   0   8  0  7100  1760 wait   I     ??    0:00.01 sh -c /usr/local/sbin/cronolog /var/log/light/tv.mysite.com/access-%Y%m%d.log
   80  5544  5527   0   8  0  7100  1760 wait   I     ??    0:00.01 sh -c /usr/local/sbin/cronolog /var/log/light/mysite2.com.au/access-%Y%m%d.log
   80  5545  5527   0   8  0  7100  1760 wait   I     ??    0:00.01 sh -c /usr/local/sbin/cronolog /var/log/light/mysite4.com/access-%Y%m%d.log
   80  5546  5527   0   8  0  7100  1760 wait   I     ??    0:00.01 sh -c /usr/local/sbin/cronolog /var/log/light/mysite4.com.au/access-%Y%m%d.log
   80  5547  5527   0   8  0  7100  1760 wait   I     ??    0:00.01 sh -c /usr/local/sbin/cronolog /var/log/light/mysite5.com.au/access-%Y%m%d.log
   80  5548  5527   0   8  0  7100  1760 wait   I     ??    0:00.01 sh -c /usr/local/sbin/cronolog /var/log/light/mysite7.net/access-%Y%m%d.log
   80  5549  5541   0  -8  0  2752   968 piperd S     ??    0:01.86 /usr/local/sbin/cronolog /var/log/light/mysite.com.au/access-%Y%m%d.log
   80  5551  5527   0   8  0  7100  1760 wait   I     ??    0:00.01 sh -c /usr/local/sbin/cronolog /var/log/light/mysite8.com/access-%Y%m%d.log
   80  5552  5527   0   8  0  7100  1760 wait   I     ??    0:00.01 sh -c /usr/local/sbin/cronolog /var/log/light/mysite3.com/access-%Y%m%d.log
   80  5553  5542   0  -8  0  2752   964 piperd S     ??    0:00.17 /usr/local/sbin/cronolog /var/log/light/mysite.com/access-%Y%m%d.log
   80  5554  5527   0   8  0  7100  1760 wait   I     ??    0:00.01 sh -c /usr/local/sbin/cronolog /var/log/light/mysite3.com.au/access-%Y%m%d.log
   80  5555  5543   0  -8  0  2752   964 piperd S     ??    0:00.06 /usr/local/sbin/cronolog /var/log/light/tv.mysite.com/access-%Y%m%d.log
   80  5556  5527   0   8  0  7100  1760 wait   I     ??    0:00.01 sh -c /usr/local/sbin/cronolog /var/log/light/mysite6.com/access-%Y%m%d.log
   80  5557  5544   0  -8  0  2752   964 piperd I     ??    0:00.01 /usr/local/sbin/cronolog /var/log/light/mysite2.com.au/access-%Y%m%d.log
   80  5558  5527   0   8  0  7100  1760 wait   I     ??    0:00.01 sh -c /usr/local/sbin/cronolog /var/log/light/mysite6.com.au/access-%Y%m%d.log
   80  5559  5546   0  -8  0  2752   964 piperd I     ??    0:00.07 /usr/local/sbin/cronolog /var/log/light/mysite4.com.au/access-%Y%m%d.log
   80  5560  5545   0  -8  0  2752   964 piperd I     ??    0:00.02 /usr/local/sbin/cronolog /var/log/light/mysite4.com/access-%Y%m%d.log
   80  5561  5547   0  -8  0  2752   964 piperd I     ??    0:00.02 /usr/local/sbin/cronolog /var/log/light/mysite5.com.au/access-%Y%m%d.log
   80  5562  5548   0  -8  0  2752   964 piperd I     ??    0:00.01 /usr/local/sbin/cronolog /var/log/light/mysite7.net/access-%Y%m%d.log
   80  5563  5551   0  -8  0  2752   964 piperd I     ??    0:00.01 /usr/local/sbin/cronolog /var/log/light/mysite8.com/access-%Y%m%d.log
   80  5564  5552   0  -8  0  2752   964 piperd I     ??    0:00.01 /usr/local/sbin/cronolog /var/log/light/mysite3.com/access-%Y%m%d.log
   80  5565  5556   0  -8  0  2752   964 piperd I     ??    0:00.01 /usr/local/sbin/cronolog /var/log/light/mysite6.com/access-%Y%m%d.log
   80  5566  5558   0  -8  0  2752   964 piperd I     ??    0:00.00 /usr/local/sbin/cronolog /var/log/light/mysite6.com.au/access-%Y%m%d.log
   80  5567  5554   0  -8  0  2752   968 piperd I     ??    0:00.09 /usr/local/sbin/cronolog /var/log/light/mysite3.com.au/access-%Y%m%d.log
   80 38778  5529   0   4  0 470028 154632 sbwait I     ??    0:37.51 /usr/local/bin/php-cgi
   80 38779  5529   0   4  0 446476 133292 accept I     ??    0:34.59 /usr/local/bin/php-cgi
   80 38909  5537   0   4  0 343052 34648 accept S     ??    0:08.86 /usr/local/bin/php-cgi
    0 39091   815   0  44  0 20360  5924 select S     ??    0:00.09 /usr/local/sbin/smbd -D -s /usr/local/etc/smb.conf
   80 39114  5534   0   4  0 346124 34808 kqread S     ??    0:10.77 /usr/local/bin/php-cgi
   80 39115  5534   0   4  0 346124 33448 accept S     ??    0:10.16 /usr/local/bin/php-cgi
   80 39116  5528   0   4  0 343052 28964 accept S     ??    0:02.06 /usr/local/bin/php-cgi
   80 39117  5528   0   4  0 343052 28776 accept S     ??    0:02.04 /usr/local/bin/php-cgi
    0  1235     1   0   5  0  5724  1304 ttyin  Is+   v0    0:00.01 /usr/libexec/getty Pc ttyv0
    0  1236     1   0   5  0  5724  1304 ttyin  Is+   v1    0:00.01 /usr/libexec/getty Pc ttyv1
    0  1237     1   0   5  0  5724  1304 ttyin  Is+   v2    0:00.01 /usr/libexec/getty Pc ttyv2
    0  1238     1   0   5  0  5724  1304 ttyin  Is+   v3    0:00.00 /usr/libexec/getty Pc ttyv3
    0  1239     1   0   5  0  5724  1304 ttyin  Is+   v4    0:00.01 /usr/libexec/getty Pc ttyv4
    0  1240     1   0   5  0  5724  1304 ttyin  Is+   v5    0:00.01 /usr/libexec/getty Pc ttyv5
    0  1241     1   0   5  0  5724  1304 ttyin  Is+   v6    0:00.01 /usr/libexec/getty Pc ttyv6
    0  1242     1   0   5  0  5724  1304 ttyin  Is+   v7    0:00.01 /usr/libexec/getty Pc ttyv7
   88   911     1   0   8  0  7100  1808 wait   I    con-   0:00.02 /bin/sh /usr/local/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --user=mys
   88  1063   911   0  46  0 495016 294400 ucond  S    con- 168:00.95 [mysqld]
 1001  5501  5499   0   8  0  9056  2788 wait   Is    p0    0:00.01 -bash (bash)
    0  5502  5501   0   8  0 20440  1884 wait   I     p0    0:00.00 su
    0  5503  5502   0  20  0 10144  3032 pause  S     p0    0:00.06 _su (csh)
    0 39133  5503   0  44  0  6828  1440 -      R+    p0    0:00.01 ps axl

  • So, I modified lighttpd.conf to use kqueue and writev and restarted lighttpd... lets see if it will hang again...:*
    server.event-handler = "freebsd-kqueue" 
    server.network-backend = "writev" 
    

RE: FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!! - Added by mr_bond almost 8 years ago

Good, no special make flags and a very standard system.

It looks like some process is not calling wait(); properly, since PID 5540 is a zombie - but that could be anything, not necessarily related to lighty or PHP.
Lets see how it goes, and hope for another deadlock soon - so we can try to find out of it.

RE: FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!! - Added by stbuehler almost 8 years ago

The zombie is a child of lighttpd; i guess some include_shell process - this got fixed in 1.4.20, so please update your version (current and recommended is 1.4.23)

And i recommend you should rethink your php setting; you are using max-procs 4 and php-children 2 (=> 4 master php processes, 8 worker) (min-procs and max-load-per-proc are not used).

RE: FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!! - Added by francisco almost 8 years ago

Hi and thanks a lot for the tips... now I'm waiting to see if it will hang again.

hey stbuehler, I'm already using 1.4.23 (I think) look:

www# pkg_info | grep -i light

lighttpd-1.4.23     A secure, fast, compliant, and very flexible Web Server

about my php setting:
I have a machine with 2 QuadCore processors on it and 4GRAM, so, is this correct? 4 max-procs and 8 fcgi-children?

fastcgi.server = ( ".php" =>
   ( "localhost" =>
      (
         "socket" => "/var/run/lighttpd/php-fastcgi.socket",
         "bin-path" => "/usr/local/bin/php-cgi",
         "max-procs"         => 4,
         "idle-timeout"      => 20,
         "bin-environment" => (
            "PHP_FCGI_CHILDREN" => "8",
            "PHP_FCGI_MAX_REQUESTS" => "500" 
          ),
         "bin-copy-environment" => (""),
         "broken-scriptfilename" => "enable" 
     )
  )
)

www.mysite.com/server-status shows this:

Server-Status
Hostname www.mysite.com () 
Uptime 1 hours 23 min 21 s 
Started at 2009-09-09 13:41:01 
absolute (since start) 
Requests 45 kreq 
Traffic 529.49 Mbyte 
average (since start) 
Requests 9 req/s 
Traffic 108.42 kbyte/s 
average (5s sliding average) 
Requests 11 req/s 
Traffic 169.38 kbyte/s 

RE: FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!! - Added by nitrox almost 8 years ago

fastcgi.server = ( ".php" =>
   ( "localhost" =>
      (
         "socket" => "/var/run/lighttpd/php-fastcgi.socket",
         "bin-path" => "/usr/local/bin/php-cgi",
         "max-procs"         => 1,
         "bin-environment" => (
            "PHP_FCGI_CHILDREN" => "8",
            "PHP_FCGI_MAX_REQUESTS" => "5000" 
          ),
         "bin-copy-environment" => (""),
         "broken-scriptfilename" => "enable" 
     )
  )
)

RE: FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!! - Added by francisco almost 8 years ago

ok... 1 max-procs, 8 fcgi_children and 5000 max_requests from now...

just for the records: at beggining, my default settings were: 1 max-procs, 4 children and 10000 max_requests...

  1. netstat -an | grep -c "\.80"
    // it shows 285 connections

lets see now how it works...

RE: FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!! - Added by francisco almost 8 years ago

hi!
just for the records: the server did not stopped yet (five days up).

Hostname www.mysite.com () 
Uptime 5 days 2 hours 10 min 40 s 
Started at 2009-09-09 15:12:05 
absolute (since start) 
Requests 1 Mreq 
Traffic 26.43 Gbyte 
average (since start) 
Requests 4 req/s 
Traffic 63.01 kbyte/s 
average (5s sliding average) 
Requests 19 req/s 
Traffic 146.72 kbyte/s 

RE: FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!! - Added by nitrox almost 8 years ago

Thx for testing and reporting back!

So concluding we can say writev/kqueue and "sane" settings for php seems to be stable on fbsd.

RE: FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!! - Added by pkirk over 7 years ago

I had a similar problem with Debian.
Resolved putting

server.network-backend = "writev" 

    (1-19/19)