[SOLVED] Internal server error 500 after there is over ~400+ simultaneous connections open to the server
Added by Devilicious over 12 years ago
My server info can be found from the end of the post.
Problem:
I'm doing a stress test to my server with some windows stress test tool. It's set to simulate ~2 000 page requests per minute. Those requests go to my PHP api which uses mysql and facebook API to deliver the asked info to the user. API response time is from around 2 seconds to 40 seconds (due to the slowness of Facebook graph api mainly) so when doing 2000 page requests per minute, some of them will load about 40 seconds and therefor there are many open connections.
The problem itself is, when about 400 (the number isn't exactly 400, but should be around it) users are sitting with open connection and waiting for the response from the API new users get Internal Server Error 500.
When doing the same stresstest against a static php file (phpinfo() in it), no errors. But again, as the stresstester program doesn't open all the 1000 connections at the same time, there weren't so many open connections this time (compared with the API stresstest)
Question:
How can i resolve this? What should i change to make the server accept more concurrent requests so there could be more open connections.
Also, the request go slower and slower as the open connection count rises (logical, i know), but i have the feeling that my server can do better than this (linode.com graphs show that during the stresstest server CPU useage is around 14%, i couldn't find mem useage graph tho)
Side question:
I can't find a log file that logs those errors. Is it just me or the system doesn't actually log them? I'm just thinking that maybe there's more info in the log file about why the users get error 500.
- I'm all new in this server managing stuff. Please describe every step as good as possible (i want to know what and why i'm doing it, so i can make stuff clearer to myself and come out of situations like this in the future), if you want to know something more about the server setup, feel free to ask (and possibly provide a way how to find out the answer).
- I'm using Debian 6 (squeezy)
- I'm using lighttpd/1.4.28 (ssl) (Build-Date: Nov 16 2010 14:53:43)
- I'm using PHP 5.3.3-7+squeeze3 with Suhosin-Patch (cli) (built: Jun 28 2011 13:13:26)
- I'm using mysql version: 5.1.49-3 (Debian)
- My lighttpd.conf can be found from here: http://paste.lighttpd.net/2 (i edited this a little bit
- My (only conf file in that folder) lighttpd/conf-enabled/10-fastcgi.conf can be found from here: http://paste.lighttpd.net/1
(even if this problem is not related to the lighttpd i'd appreciate if you help me to solve it)
Replies (3)
RE: Internal server error 500 after there is over ~400+ simultaneous connections open to the server - Added by nitrox over 12 years ago
That error is probably produced because lighty can´t connect to php anymore as all php processes are busy or error out themselves. You can enable php´s error_log at php.ini and make it write to a separate logfile. You have set max-procs to 2 and "PHP_FCGI_CHILDREN" => "4", you can easily raise them from 4 to 8 or 16, max-procs should stay low and 2 is ok.
But basically you can´t do anything against it, if all workers are busy to get data via an external source you´re doomed. 2-40s is way too much, maybe you can cache repeating requests.
Benchmarking isn´t reliable either, you should watch and adapt it when you go live.
RE: Internal server error 500 after there is over ~400+ simultaneous connections open to the server - Added by icy over 12 years ago
In your fastcgi config, you set PHP to use 2 main processes with 4 children each. That means you have 8 PHP backend processes that can each process 1 request at a time.
In your scenario, eventually all 8 of them will be hanging around handling a request that takes 40 seconds. All other requests are waiting and piling up.
Evetually you'll reach a point where the server or PHP backend say "ok this is too much, go away".
Since you can't speed up the Facebook API the only way to deal with this is spawn more PHP backends. You always need as many PHP backends as you want to process concurrent requests.
Try setting max-procs to 4 and children to 16. That would allow you to handle 64 concurrent requests. Depending on the load of your website that might be enough or not.
Unfortunately handling such a situation with PHP is not easy but you might want to look into PHP-FPM which can spawn more processes as they are needed provided you have enough RAM.
RE: Internal server error 500 after there is over ~400+ simultaneous connections open to the server - Added by Devilicious over 12 years ago
Really appreciated guys!
I changed the processes count and it went better. Tho i still wasn't able to make sure what was the mem usage under load, but i guess i'll figure it out somehow (to see how many PHP processes it is possible to run concurrently (do they have like fixed memory limit they can take so i could calculate it out somehow?))
Many thanks for the described answers.