Lighttpd

sbwait state means there is some socket event that php is waiting for but never happens. Can you check what they are actually waiting for?
Attach gdb to a php process in sbwait state and then type in "bt" to get a backtrace. Or attach with strace/ktrace and see if that turns out anything usefull.

Without knowing what php is waiting for, it's impossible to tell what the problem is.

I can reliably reproduce this. Here's the config of the system:

FreeBSD 7.0-RELEASE amd64 using an SMP kernel with SCHED_ULE. PHP 5.2.6 and lighttpd-1.4.19_3 from ports running as a FastCGI. I can reliably reproduce this by running Gallery2 (2.2 or 2.3) and doing an upload with Gallery Remote. After a few images are uploaded the applet hangs with "Upload complete: server processing" and one of the copies of php-cgi are hang as sbwait.

Here's what I was able to get so far. Since I can reproduce this at will I'll gladly reproduce it as needed to gather more info. Just let me know what you need.

As seen in top:

81073 c0nsumer        1   4    0   116M 20988K sbwait 3   0:01  0.00% php-cgi

Some basic info from gdb:

c0nsumer@banstyle:/> sudo gdb /usr/local/bin/php-cgi 81073

(gdb) bt
#0  0x00000008010a476a in read () from /lib/libc.so.7
#1  0x000000000057d5fc in fcgi_read ()
#2  0x000000000057e306 in sapi_cgi_read_post ()
#3  0x00000000004c84a4 in fill_buffer ()
#4  0x00000000004c88b5 in multipart_buffer_read ()
#5  0x00000000004c9c08 in rfc1867_post_handler ()
#6  0x00000000004c6ee5 in sapi_handle_post ()
#7  0x00000000004cc30c in php_default_treat_data ()
#8  0x00000000004cc7eb in php_hash_environment ()
#9  0x00000000004bff47 in php_request_startup ()
#10 0x000000000057f8c6 in main ()

(gdb) f 0
#0  0x00000008010a476a in read () from /lib/libc.so.7
(gdb) info frame
Stack level 0, frame at 0x7fffffff9c50:
 rip = 0x8010a476a in read; saved rip 0x57d5fc
 called by frame at 0x7fffffff9db0
 Arglist at 0x7fffffff9c40, args: 
 Locals at 0x7fffffff9c40, Previous frame's sp is 0x7fffffff9c50
 Saved registers:
  rip at 0x7fffffff9c48

I noticed that 1.4.20 was in ports, so I upgraded to that to give it a try. That didn't resolve the problem and I was able to reproduce it after uploading just two ~5MB images.

ok. we just had an user with a similar problem and for him
server.network-backend = "writev"
server.event-handler = "freebsd-kqueue"

fixed the issue. can you please test that and report back?

server.network-backend = "writev" seems to have done it. Doing the same test as before, 36 photos have uploaded without a problem. A friend with the same issue on a different server (similar config) tried this as well, and it seems to have sorted things out for him as well.

server.event-handler = "freebsd-kqueue" was already set, so that likely didn't have an impact. From what I can tell, the default server.network-backend setting is "write". Is that correct?

yes the default is write and seems horribly broken on fbsd 7.
i got reports on 6.3 it was working properly with
server.network-backend = "write"
server.event-handler = "freebsd-kqueue"

so what ever happened it broke something.

I actually had it working great on 6.3 myself with write (because it was the default and not set), and with freebsd-kqueue manually specified. So, add me to that list as well.

Thanks for your help on this, I'm glad to know there is a solution. Hopefully a more permanent fix can be put in place eventually, wherever it is needed.

By the way, I'd read that server.network-backend = "freebsd-sendfile" breaks the OS itself pretty horribly. Any idea if this is still the case? I'd try it, but the server I'm currently working with is in production and I don't have a handy spare.

IIRC there where some problems if you tried freebsd-sendfile with unix sockets, it guess it will work for tcp connections. (the network-backend is used for the connections to fastcgi/... too).

The main problem is that there is no "fallback" if the OS tells us that it doesn't support sendfile on a specific socket or file.

Thanks!

I will try

#server.network-backend = "freebsd-sendfile"
server.network-backend = "writev"

3 days later, no more frozen php-cgi.

Thanks a lot.

freebsd-sendfile worked fine for me in 6.3-RELEASE, but when I upgraded to 7.0-RELEASE I noticed same problems as described, easily reproducable, switching to "writev" also solved the problem..

However.. I was going to get a stack-trace of the problem for darix, but after upgrading the server in question to 7.1-RELEASE (using freebsd-update (this also patches the KERNEL to use the new SCHED_ULE scheduler)), I could no longer reproduce the problem with php-uploads/fcgi via socket.

I've tried to reproduce it all day, with everything from ubuntu 700MB ISO-file, to SLES10.1 4.3GB DVD ISO-file - without successfully leaving process hanging.

I suggest others having this issue tries to upgrade to 7.1-RELEASE if possible, and report back if there is any issues with it.

PS: this is also an SMP system

Cheers!

Really good news!

I will try before the Chinese New Year.