Bug #25
closedlighttpd crashing under load on Linux
Description
I have had Lighttpd versions 1.3.6 upto current 1.3.11 crashing unpredictably. The server may run for 10 minutes or for several days before crashing, but usually it crashes 2-3 times a day.
The system running Lighttpd is a Debian woody and it serves about 700,000 page views per day, using PHP with FastCGI.
One crash I witnessed today gave me the following backtrace under gdb:
(gdb) r -D -f /etc/lighttpd/lighttpd.conf Starting program: /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf (no debugging symbols found)...(no debugging symbols found)... (no debugging symbols found)...(no debugging symbols found)... (no debugging symbols found)...(no debugging symbols found)... (no debugging symbols found)...(no debugging symbols found)... (no debugging symbols found)...(no debugging symbols found)... (no debugging symbols found)...(no debugging symbols found)... Program received signal SIGSEGV, Segmentation fault. 0x4009c1b7 in memcpy () from /lib/libc.so.6 (gdb) bt !#0 0x4009c1b7 in memcpy () from /lib/libc.so.6 !#1 0x0804e024 in buffer_append_string_len () !#2 0x0804e067 in buffer_append_string_buffer () !#3 0x0804f594 in log_error_write () !#4 0x08052eda in http_request_parse () !#5 0x08055f92 in connection_state_machine () !#6 0x0804da10 in main () !#7 0x4004014f in __libc_start_main () from /lib/libc.so.6 (gdb)
-- Martijn Grendelman
Updated by jan almost 20 years ago
- Status changed from New to Assigned
Is it possible to use a non-stripped binary together with gdb ? We would get linenumbers and would see where it crashes.
Updated by Anonymous almost 20 years ago
I will try. I use the Debian package that results from typing dpkg-buildpackage
in the source directory. I'll try and find out how to prevent stripping.
In the mean time, I have had another crash, with a slightly different backtrace:
Program received signal SIGSEGV, Segmentation fault. 0x4009c1b7 in memcpy () from /lib/libc.so.6 (gdb) bt #0 0x4009c1b7 in memcpy () from /lib/libc.so.6 #1 0x0804ddec in buffer_copy_string_len () #2 0x0804de34 in buffer_copy_string_buffer () #3 0x08051668 in http_response_prepare () #4 0x08055ff2 in connection_state_machine () #5 0x0804da10 in main () #6 0x4004014f in __libc_start_main () from /lib/libc.so.6
-- Martijn Grendelman
Updated by Anonymous almost 20 years ago
In the source-directory, edit {{{debian/rules}}} and comment out the line with {{{dh@dh_strip@ and rebuild the package.
-- Mark van Eijk
Updated by Anonymous almost 20 years ago
Yeah, I found that. However, I just ran the binary that was already in src/. The server had been running for almost three hours when it crashed, just now. Backtrace:
Program received signal SIGSEGV, Segmentation fault. 0x4009c1b7 in memcpy () from /lib/libc.so.6 (gdb) bt #0 0x4009c1b7 in memcpy () from /lib/libc.so.6 #1 0x0804e024 in buffer_append_string_len (b=0x8064260, s=0x4013c6b8 "", s_len=1075037879) at buffer.c:252 #2 0x0804e067 in buffer_append_string_buffer (b=0x8064260, src=0xa147370) at buffer.c:263 #3 0x0804f594 in log_error_write (srv=0x8063f78, filename=0x805fa32 "request.c", line=868, fmt=0x805fc08 "bs") at log.c:180 #4 0x08052eda in http_request_parse (srv=0x8063f78, con=0x840cb10) at request.c:590 #5 0x08055f92 in connection_state_machine (srv=0x8063f78, con=0x840cb10) at connections.c:1218 #6 0x0804da10 in main (argc=4, argv=0xbffffc14) at server.c:984
-- Martijn Grendelman
Updated by jan almost 20 years ago
does
if (con->request.http_host &&
0 != request_check_hostname(srv, con, con->request.http_host)) {
in request.c, line 867 help ?
Updated by Anonymous almost 20 years ago
I modified request.c as you suggested and recompiled, but that didn't fix the problem:
Program received signal SIGSEGV, Segmentation fault. 0x4009c1b7 in memcpy () from /lib/libc.so.6 (gdb) bt #0 0x4009c1b7 in memcpy () from /lib/libc.so.6 #1 0x0804e024 in buffer_append_string_len (b=0x8064260, s=0x929dfa0 "È\004", s_len=1075037879) at buffer.c:252 #2 0x0804e067 in buffer_append_string_buffer (b=0x8064260, src=0x8f0b178) at buffer.c:263 #3 0x0804f594 in log_error_write (srv=0x8063f78, filename=0x805fa32 "request.c", line=869, fmt=0x805fc08 "bs") at log.c:180 #4 0x08052eda in http_request_parse (srv=0x8063f78, con=0x88144f0) at request.c:590 #5 0x08055fa2 in connection_state_machine (srv=0x8063f78, con=0x88144f0) at connections.c:1218 #6 0x0804da10 in main (argc=4, argv=0xbffffc14) at server.c:984 (gdb)
A check for a non-empty string doesn't seem to cut it. "È\004" also looks like a strange hostname to me ;-)
-- Martijn Grendelman
Updated by Anonymous almost 20 years ago
I don't get it. How should I read this backtrace? Do I get this right?
#
44 says that in request.c on line 590, log_error_write was called (fmt="s"), but the arguments that log_error_write got according to #
33, were ''line=869_ and _fmt="bs"''
Isn't the problem in either request.c:590 or in log_error_write() ?
-- Martijn Grendelman
Updated by Anonymous almost 20 years ago
Any news on this issue?
In the mean time, I changed request.c from line 868 onward to read:
log_error_write(srv, __FILE__, __LINE__, "s", "Invalid Hostname: (not logged) -> 400");
where it first said:
log_error_write(srv, __FILE__, __LINE__, "sbs", "Invalid Hostname:", con->request.http_host, "-> 400");
and the crashes have gone away. My conclusion is, that there is either something in con->request.http_host that shouldn't be there or buffer_append_string_len() doesn't handle things the way it should.
Martijn.
-- Martijn Grendelman
Updated by jan almost 20 years ago
- Status changed from Assigned to Fixed
- Resolution set to fixed
This has been fixed ...no, let's better say circumvented by changeset r48.
I still have no idea why it happends. Perhaps we get some more info with this debug-code.
Please enable
debug.log-request-headers-on-error = "enable"
and watch out for the reported headers.
Updated by Anonymous almost 20 years ago
Alright, I have all my servers (7 in total) running a patched server with the debug-option turned on.
The problem is, that not "Invalid Hostname" error crashes the server. I have now compiled Lighttpd with the new debug-code, but also with the "malicious" log statement, right after the debug code. Hopefully, this way the headers of the request that causes the crash get logged.
I'll let you know. Thanks for your support.
-- Martijn Grendelman
Also available in: Atom