FreeBSD 7.2 Freezing, Locking on Lighttpd+php fast-cgi!!
Hi All.
I need some help here.
I used to use Apache 1.3, but now, that I have a new server,
I migrate my website to Lighty, BUT IT IS FREEZING !!
One, two, three times a day it locks down on lighty rising CPU usage to 100%!
It seems to be something related to lighttpd + php fast-cgi.
On 2009-08-24, it stopped at 19:30 according to CACTI graphs that shows 100% CPU Utilization
- it jumped from normal 2-5 cpu (at 19:25) to 100% (at 19:30).
When the server locks, I try to KILL -9 lighttpd process, but it just dont work!!!
"Waiting for PIDS: 7748 7748 7748 7748 7748 7748 7748 7748 7748 7748 7748..."
(and never stops)
-------------------------------------------------
Some info about my scenario:
-------------------------------------------------
FreeBSD 7.2-RELEASE #0: Tue Jul 28 19:36:22 BRT 2009 root@server.com :/usr/src/sys/amd64/compile/www Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5410 @ 2.33GHz (2327.51-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x1067a Stepping = 10 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8 ,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36 ,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x40ce3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST ,TM2,SSSE3,CX16,xTPR,PDCM,DCA,<b19>,XSAVE> AMD Features=0x20100800<SYSCALL,NX,LM> AMD Features2=0x1<LAHF> Cores per package: 4 usable memory = 4276776960 (4078 MB) avail memory = 4104232960 (3914 MB) ACPI APIC Table: <DELL PE_SC3 > FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 ------------------------------------------------- mysql-server-5.1.36 xcache-1.2.2 php5-5.2.10 lighttpd-1.4.23 -------------------------------------------------
When I restarted lighttpd, FREE cmd shows:
mem_wire: 507826176 ( 484MB) [ 12%] mem_active: + 308924416 ( 294MB) [ 7%] mem_inactive:+ 360247296 ( 343MB) [ 8%] mem_cache: + 5332992 ( 5MB) [ 0%] mem_free: + 2943963136 ( 2807MB) [ 71%] mem_gap_vm: + 487424 ( 0MB) [ 0%] 17 hours after last LOCK DOWN, free cmd shows: mem_wire: 671326208 ( 640MB) [ 16%] mem_active: + 472195072 ( 450MB) [ 11%] mem_inactive:+ 1835315200 ( 1750MB) [ 44%] mem_cache: + 2838528 ( 2MB) [ 0%] mem_free: + 1144553472 ( 1091MB) [ 27%] mem_gap_vm: + 552960 ( 0MB) [ 0%]
Server-status and Server-statistics reports:
Uptime 16 hours 53 s Started at 2009-08-24 21:36:35 absolute (since start) Requests 245 kreq Traffic 3.46 Gbyte average (since start) Requests 4 req/s Traffic 62.99 kbyte/s average (5s sliding average) Requests 18 req/s Traffic 161.91 kbyte/s -------------------------------- fastcgi.active-requests: 0 fastcgi.backend.localhost.0.connected: 676 fastcgi.backend.localhost.0.died: 0 fastcgi.backend.localhost.0.disabled: 0 fastcgi.backend.localhost.0.load: 0 fastcgi.backend.localhost.0.overloaded: 0 fastcgi.backend.localhost.1.connected: 2921 fastcgi.backend.localhost.1.died: 0 fastcgi.backend.localhost.1.disabled: 0 fastcgi.backend.localhost.1.load: 0 fastcgi.backend.localhost.1.overloaded: 0 fastcgi.backend.localhost.2.connected: 13352 fastcgi.backend.localhost.2.died: 0 fastcgi.backend.localhost.2.disabled: 0 fastcgi.backend.localhost.2.load: 0 fastcgi.backend.localhost.2.overloaded: 0 fastcgi.backend.localhost.3.connected: 69502 fastcgi.backend.localhost.3.died: 0 fastcgi.backend.localhost.3.disabled: 0 fastcgi.backend.localhost.3.load: 0 fastcgi.backend.localhost.3.overloaded: 0 fastcgi.backend.localhost.load: 0 fastcgi.requests: 86451
lighttpd.error.log shows NOTHING more than the usual "Connection reset by peer" (crazy error that I just cant get rid :-( ):
2009-08-24 15:33:20: (connections.c.132) (warning) close: 26 Connection reset by peer 2009-08-24 16:38:39: (connections.c.132) (warning) close: 23 Connection reset by peer 2009-08-24 16:47:54: (connections.c.132) (warning) close: 20 Connection reset by peer At 19:30 whet I realized it was locked down, I tried KILL cmd. But kill does not worked, so at 20:31 I did a "reboot" command on ssh. 2009-08-24 20:32:30: (log.c.172) server started 2009-08-24 20:52:53: (server.c.1495) server stopped by UID = 0 PID = 0 2009-08-24 20:53:16: (log.c.172) server started 2009-08-24 21:36:35: (log.c.172) server started 2009-08-24 21:37:50: (connections.c.132) (warning) close: 25 Connection reset by peer 2009-08-25 10:58:26: (connections.c.132) (warning) close: 38 Connection reset by peer
/var/log/php-error.log: I know I have this syntax error somewhere, but I dont believe it be the cause of 100% utilisation of CPU.
[24-Aug-2009 18:33:05] PHP Parse error: syntax error, unexpected T_DOUBLE_ARROW, expecting ')' in /usr/dangelina/inetsrv/include/hylibnet.mdl(1437) : eval() [24-Aug-2009 *19:27:58*] PHP Parse error: syntax error, unexpected T_DOUBLE_ARROW, expecting ')' in /usr/dangelina/inetsrv/include/hylibnet.mdl(1437) : eval() [24-Aug-2009 *20:55:56*] PHP Parse error: syntax error, unexpected T_DOUBLE_ARROW, expecting ')' in /usr/dangelina/inetsrv/include/hylibnet.mdl(1437) : eval()
Nothing usefull on /var/log/messages either.
/usr/local/etc/lighttpd.conf
## My trying Tweaks
server.max-keep-alive-requests = 4
server.max-keep-alive-idle = 4
server.max-fds = 6144
server.max-connections = 2048
server.max-read-idle = 20
server.max-write-idle = 180
fastcgi.server = ( ".php" =>
( "localhost" =>
(
"socket" => "/var/run/lighttpd/php-fastcgi.socket",
"bin-path" => "/usr/local/bin/php-cgi",
"max-procs" => 4,
"min-procs" => 1,
"max-load-per-proc" => 10,
"idle-timeout" => 20,
"bin-environment" => (
"PHP_FCGI_CHILDREN" => "2",
"PHP_FCGI_MAX_REQUESTS" => "500"
),
"bin-copy-environment" => (""),
"broken-scriptfilename" => "enable"
)
)
)
Thank you in advance for any directions, any help...
Francisco
| www-crash-2009-08-29-19-30.jpg (351 KB) www-crash-2009-08-29-19-30.jpg | CACTI graphs at instance of last crash |
