Project

General

Profile

Actions
Copy link

Bug #2382

closed

Lighttpd Segfault

Added by molliver almost 13 years ago. Updated 11 months ago.

Status:
Fixed
Priority:
High
Category:
core
Target version:
1.4.31
ASK QUESTIONS IN Forums:
No

Description

I have noticed we are getting a large number of segfault on lighttpd, below is the information I have.
This seems to have been happening not only with out current version but also the previous version as well.

lighttpd/1.4.30-devel-b3fdfbf (ssl) - a light and fast webserver
Build-Date: Dec 28 2011 11:22:22

Dec  2 06:55:05 web kernel: [3993015.316728] lighttpd[20229]: segfault at ad5a000 ip b7475b39 sp bfefcfec error 4 in libz.so.1.2.3.3[b7474000+13000]
Dec  7 07:23:52 web kernel: [170687.846561] lighttpd[1148]: segfault at b3c3000 ip b75effe1 sp bf86e67c error 4 in libc-2.11.1.so[b757b000+153000]
Dec 23 08:49:28 web kernel: [1558224.520738] lighttpd[23746]: segfault at c38f000 ip b74b0b39 sp bff7cdec error 4 in libz.so.1.2.3.3[b74af000+13000]
Dec 23 12:02:04 web kernel: [1569780.056483] lighttpd[27789]: segfault at aefe000 ip b7478fc6 sp bf9c2038 error 4 in libc-2.11.1.so[b7404000+153000]
Dec 23 13:25:18 web kernel: [1574774.206902] lighttpd[16213]: segfault at b094000 ip b73f5af8 sp bf85dd4c error 4 in libz.so.1.2.3.3[b73f4000+13000]
Dec 23 13:29:02 web kernel: [1574998.203875] lighttpd[25049]: segfault at 9726000 ip b7571fe1 sp bfb3a42c error 4 in libc-2.11.1.so[b74fd000+153000]
Dec 28 16:55:58 web kernel: [2019414.457601] lighttpd[713]: segfault at 9b76000 ip b74d8fe1 sp bfa2952c error 4 in libc-2.11.1.so[b7464000+153000]
Jan  2 11:44:37 web kernel: [402632.423190] lighttpd[1137]: segfault at c659000 ip b7573fe1 sp bfbada4c error 4 in libc-2.11.1.so[b74ff000+153000]
Jan  2 16:42:56 web kernel: [420531.110163] lighttpd[22909]: segfault at 9485000 ip b75defe1 sp bfcc652c error 4 in libc-2.11.1.so[b756a000+153000]
Jan  2 20:52:44 web kernel: [435519.759361] lighttpd[16360]: segfault at 9f7d000 ip b746efe1 sp bfd4199c error 4 in libc-2.11.1.so[b73fa000+153000]
Jan  3 15:56:59 web kernel: [504174.841949] lighttpd[4659]: segfault at b779000 ip b75ecfe1 sp bfc246ac error 4 in libc-2.11.1.so[b7578000+153000]
Jan  3 20:04:08 web kernel: [ 8256.077964] lighttpd[1145]: segfault at f651000 ip b741bfe1 sp bffa2e9c error 4 in libc-2.11.1.so[b73a7000+153000]
Jan  4 19:31:18 web kernel: [92686.361097] lighttpd[5046]: segfault at ca09000 ip b751cfe1 sp bfa3f6bc error 4 in libc-2.11.1.so[b74a8000+153000]
Jan  5 12:56:18 web kernel: [155386.153243] lighttpd[1965]: segfault at 9970000 ip b7497fe1 sp bfbc58bc error 4 in libc-2.11.1.so[b7423000+153000]
Jan  5 14:40:12 web kernel: [161619.909660] lighttpd[15063]: segfault at b5a9000 ip b74e8fe1 sp bf96bdbc error 4 in libc-2.11.1.so[b7474000+153000]
Jan  6 10:56:37 web kernel: [234605.428297] lighttpd[8283]: segfault at abad000 ip b7497fe1 sp bfa3050c error 4 in libc-2.11.1.so[b7423000+153000]
Jan  7 17:48:56 web kernel: [345743.677970] lighttpd[11973]: segfault at a5c3000 ip b754dfe1 sp bff5c6fc error 4 in libc-2.11.1.so[b74d9000+153000]
Jan  8 16:33:27 web kernel: [427615.081242] lighttpd[7448]: segfault at b717000 ip b7557fe1 sp bf9f3fec error 4 in libc-2.11.1.so[b74e3000+153000]
Nov  8 09:53:39 web kernel: [1930128.806360] lighttpd[26909]: segfault at b475000 ip b75a2fe1 sp bf88c05c error 4 in libc-2.11.1.so[b752e000+153000]
Nov 16 17:15:27 web kernel: [2647836.764856] lighttpd[25889]: segfault at a229000 ip b7485fe1 sp bf972c3c error 4 in libc-2.11.1.so[b7411000+153000]
Nov 19 19:53:41 web kernel: [2916530.957676] lighttpd[25884]: segfault at acfb000 ip b7536b00 sp bfbe240c error 4 in libz.so.1.2.3.3[b7535000+13000]
Nov 23 08:48:53 web kernel: [3222242.865007] lighttpd[10841]: segfault at bdec000 ip b742ffe1 sp bfe4573c error 4 in libc-2.11.1.so[b73bb000+153000]
Oct 14 13:06:20 web kernel: [4803599.686706] lighttpd[27294]: segfault at fb23000 ip b741bfe1 sp bf84d20c error 4 in libc-2.11.1.so[b73a7000+153000]
Oct 17 10:25:35 web kernel: [31245.637606] lighttpd[7597]: segfault at a3fe000 ip b753bfe1 sp bf95d61c error 4 in libc-2.11.1.so[b74c7000+153000]
Oct 20 10:15:00 web kernel: [289810.487955] lighttpd[6956]: segfault at ba05000 ip b7512af8 sp bf82538c error 4 in libz.so.1.2.3.3[b7511000+13000]

Thanks

Mark

Actions
Copy link
 #1

Updated by spaam almost 13 years ago

  • Status changed from New to Need Feedback

Can you get a stack trace? you need to install a version with debug symbols for that. kinda hard to guess what it is if we don't have it..

Actions
Copy link
 #2

Updated by spaam almost 13 years ago

are you using mod_compress?

Actions
Copy link
 #3

Updated by molliver almost 13 years ago

Hi,

No we are not using mod_compress and this is a live production server so we cant really switch to a debug version.

Sorry

Mark

Actions
Copy link
 #4

Updated by stbuehler almost 13 years ago

  • Description updated (diff)
  • Priority changed from Urgent to High
  • You had segfaults from Dec 2 to Oct 20 (in 2011???) - looks like 10 months to me. can't be that urgent.
  • Oct 20 is before Dec 28, so no segfaults in the current version
  • segfaults in libz, but no mod_compress loaded: could be ssl - or mod_deflate (which isn't in upstream 1.4 and therefore not supported).
  • where does this version come from: -devel-b3fdfbf ? custom patches?

debug symbols != debug version. you can compile with full optimization and debug symbols (-g), and even strip the debug symbols into separate files if the memory usage of these would be a problem.
then you can enable core dumps, and after the next crash you can load the core dump with gdb and get a useful backtrace.

i'm afraid we can't do anything without details.

Actions
Copy link
 #5

Updated by marek almost 13 years ago

I got the same:

lighttpd[28625]: segfault at 0000000000000358 rip 000000000040e9d7 rsp 00007fff26fde8f0 error 6
lighttpd[11052]: segfault at 0000000000000358 rip 000000000040e9d7 rsp 00007fff26fde8f0 error 6
lighttpd[11187]: segfault at 0000000000000358 rip 000000000040e9d7 rsp 00007fff26fde8f0 error 6
lighttpd[11210]: segfault at 0000000000000358 rip 000000000040e9d7 rsp 00007fff26fde8f0 error 6
lighttpd[2897]: segfault at 0000000000000358 rip 000000000040e9d7 rsp 00007fff26fde8f0 error 6
lighttpd[11248]: segfault at 0000000000000358 rip 000000000040e9d7 rsp 00007fff26fde8f0 error 6
lighttpd[11250]: segfault at 0000000000000358 rip 000000000040e9d7 rsp 00007fff26fde8f0 error 6
lighttpd[11235]: segfault at 0000000000000358 rip 000000000040e9d7 rsp 00007fff26fde8f0 error 6
lighttpd[11276]: segfault at 0000000000000358 rip 000000000040e9d7 rsp 00007fff26fde8f0 error 6
lighttpd[11202]: segfault at 0000000000000358 rip 000000000040e9d7 rsp 00007fff26fde8f0 error 6
lighttpd[11318]: segfault at 0000000000000358 rip 000000000040e9d7 rsp 00007fff26fde8f0 error 6
lighttpd[2855]: segfault at 0000000000000358 rip 000000000040e9d7 rsp 00007fff26fde8f0 error 6
lighttpd[11340]: segfault at 0000000000000358 rip 000000000040e9d7 rsp 00007fff26fde8f0 error 6
Actions
Copy link
 #6

Updated by marek almost 13 years ago

Also php-cgi is segfaulting too:

php-cgi[11351]: segfault at 00000000059d5950 rip 00002b06f7c3ef50 rsp 00007fff44a34230 error 4
php-cgi[12109]: segfault at 0000000004420950 rip 00002b3baaee4f50 rsp 00007fff3713bcf0 error 4
php-cgi[13511]: segfault at 0000000014ca0860 rip 00002ae8f75a6d74 rsp 00007fffad5afb10 error 4
php-cgi[16595]: segfault at 000000000e0fb860 rip 00002addb06c5f50 rsp 00007fffa2daad00 error 4
php-cgi[16905]: segfault at 000000000ab5c860 rip 00002b91fff2ad74 rsp 00007fff35a49f20 error 4
php-cgi[24509]: segfault at 0000000017655950 rip 00002ad6d0e9cf50 rsp 00007fff6ac439e0 error 4
php-cgi[1387]: segfault at 00002ac97bdbd439 rip 00002ac97bdbd439 rsp 00000000449f90a0 error 14
php-cgi[1382]: segfault at 00002ac97bdbd5d2 rip 00002ac97bdbd5d2 rsp 0000000040ef60d8 error 14
php-cgi[1383]: segfault at 00002ac97bdbd5c8 rip 00002ac97bdbd5c8 rsp 00000000421f50d8 error 14
php-cgi[1384]: segfault at 00002ac97bdbd5d0 rip 00002ac97bdbd5d0 rsp 0000000042bf60d8 error 14
php-cgi[1385]: segfault at 00002ac97bdbd5c8 rip 00002ac97bdbd5c8 rsp 00000000435f70d8 error 14
php-cgi[1388]: segfault at 00002ac97bdbd5c8 rip 00002ac97bdbd5c8 rsp 00000000453fa0d8 error 14
php-cgi[1386]: segfault at 00002ac97bdbd5d6 rip 00002ac97bdbd5d6 rsp 0000000043ff80d8 error 14
php-cgi[7326]: segfault at 0000000020169860 rip 00002aabfcfc8f50 rsp 00007fff212e04e0 error 4
php-cgi[17901]: segfault at 000000001ae7e860 rip 00002ad8a462bd74 rsp 00007fffbb0d5e00 error 4
php-cgi[18380]: segfault at 000000001ae7e860 rip 00002b393b360d74 rsp 00007fff286c06b0 error 4
php-cgi[20190]: segfault at 00002ae8954fc5d6 rip 00002ae8954fc5d6 rsp 0000000041fef0d8 error 14
php-cgi[20196]: segfault at 00002ae8954fc5d6 rip 00002ae8954fc5d6 rsp 00000000451f40d8 error 14
php-cgi[20193]: segfault at 00002ae8954fc5d6 rip 00002ae8954fc5d6 rsp 00000000433f10d8 error 14
php-cgi[20195]: segfault at 00002ae8954fc5d6 rip 00002ae8954fc5d6 rsp 00000000447f30d8 error 14
php-cgi[20191]: segfault at 00002ae8954fc5d2 rip 00002ae8954fc5d2 rsp 0000000040ecb0d8 error 14
php-cgi[20192]: segfault at 00002ae8954fc5d2 rip 00002ae8954fc5d2 rsp 00000000429f00d8 error 14
php-cgi[20194]: segfault at 00002ae8954fc5cc rip 00002ae8954fc5cc rsp 0000000043df20d8 error 14
php-cgi[29854]: segfault at 0000000011e8f950 rip 00002ad1d32b1f50 rsp 00007fffc4c3fea0 error 4
php-cgi[2441]: segfault at 000000000a69e950 rip 00002b13cb0e3f50 rsp 00007fffb7995750 error 4
php-cgi[3677]: segfault at 000000000f26c950 rip 00002ba9e46a3f50 rsp 00007fff4fc5dc90 error 4
php-cgi[5459]: segfault at 00002ab77d92c5d2 rip 00002ab77d92c5d2 rsp 00000000434560d8 error 14
php-cgi[5456]: segfault at 00002ab77d92c5c8 rip 00002ab77d92c5c8 rsp 00000000420540d8 error 14
php-cgi[5458]: segfault at 00002ab77d92c5d6 rip 00002ab77d92c5d6 rsp 0000000042a550d8 error 14
php-cgi[7803]: segfault at 00002b6f2c5db5d0 rip 00002b6f2c5db5d0 rsp 00000000444af0d8 error 14
php-cgi[7799]: segfault at 00002b6f2c5db5d2 rip 00002b6f2c5db5d2 rsp 0000000041cab0d8 error 14
php-cgi[7805]: segfault at 00002b6f2c5db5d6 rip 00002b6f2c5db5d6 rsp 00000000458b10d8 error 14
php-cgi[7802]: segfault at 00002b6f2c5db5d2 rip 00002b6f2c5db5d2 rsp 0000000043aae0d8 error 14
php-cgi[7801]: segfault at 00002b6f2c5db5d0 rip 00002b6f2c5db5d0 rsp 00000000430ad0d8 error 14
php-cgi[16808]: segfault at 00000000123bb950 rip 00002b8fe88d0f50 rsp 00007fff45ffb5c0 error 4
php-cgi[16828]: segfault at 000000000e2ee950 rip 00002af501c3fd74 rsp 00007ffff47e7000 error 4
php-cgi[23617]: segfault at 000000000a32e950 rip 00002ae40e3fff50 rsp 00007fff7c2bdb90 error 4
php-cgi[7657]: segfault at 000000001adb3950 rip 00002b2699179d74 rsp 00007ffff320ac40 error 4
php-cgi[1659]: segfault at 0000000015119930 rip 00002b3eb386ff50 rsp 00007fff168309b0 error 4
php-cgi[1672]: segfault at 0000000004ce69f0 rip 00002b67e6b50f50 rsp 00007fffcdb0ead0 error 4
php-cgi[10391]: segfault at 000000001f879950 rip 00002b68b2f89f50 rsp 00007fffe94aa750 error 4
php-cgi[20404]: segfault at 0000000019412950 rip 00002b8a39feff50 rsp 00007fffa7cd96c0 error 4
php-cgi[25993]: segfault at 000000000cd3b860 rip 00002ba2a4b03f50 rsp 00007fff977e7c60 error 4
php-cgi[25892]: segfault at 000000000cd3b860 rip 00002ac821f85d74 rsp 00007fff6c05ab40 error 4
Actions
Copy link
 #7

Updated by stbuehler almost 13 years ago

"same"... you have a 64-bit build, the original report is 32-bit (afaics), and different error numbers (can be the same problem ofc)

and again: no details, no fixing possible.

btw: could it be that your memory is failing? and you could at least provide some environment details, like distribution and so on.

Actions
Copy link
 #8

Updated by marek almost 13 years ago

I'm using CentOS 5.7 64-bit on dedicated server. My memory is fine.

Lighttpd 1.4.30 compiled with openssl 64-bit.

lighttpd/1.4.30 (ssl) - a light and fast webserver
Build-Date: Dec 27 2011 16:15:51
Actions
Copy link
 #9

Updated by marek almost 13 years ago

Error logs:

2012-01-12 19:09:32: (mod_cgi.c.1313) cleaning up CGI: process died with signal 11
2012-01-12 19:14:08: (mod_cgi.c.601) cgi died, pid: 14840
2012-01-12 19:14:08: (mod_cgi.c.1313) cleaning up CGI: process died with signal 11
2012-01-12 19:15:11: (mod_cgi.c.601) cgi died, pid: 18889
2012-01-12 19:16:49: (mod_cgi.c.601) cgi died, pid: 25413
2012-01-12 19:19:25: (mod_cgi.c.601) cgi died, pid: 2781
2012-01-12 19:19:25: (mod_cgi.c.601) cgi died, pid: 2783
2012-01-12 19:19:27: (mod_cgi.c.1313) cleaning up CGI: process died with signal 11
2012-01-12 19:22:25: (mod_cgi.c.601) cgi died, pid: 28934
2012-01-12 19:22:26: (mod_cgi.c.601) cgi died, pid: 28211
Actions
Copy link
 #10

Updated by marek almost 13 years ago

My server specs are:

Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 16 GB RAM DDR3, 2x3 TB HDD S/W RAID1 + 120 GB SSD

PHP 5.3.8 (cli) (built: Sep 30 2011 20:07:13) 
Copyright (c) 1997-2011 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2011 Zend Technologies
    with XCache v1.3.2, Copyright (c) 2005-2011, by mOo
    with the ionCube PHP Loader v4.0.11, Copyright (c) 2002-2011, by ionCube Ltd., and
    with TrueBug PHP Loader v1.2.0, Copyright (c) 2006-2010, by TrueBug Software
    with SourceGuardian v8.2, Copyright (c) 2000-2010, by Inovica Ltd.
    with Zend Guard Loader v3.3, Copyright (c) 1998-2010, by Zend Technologies
Actions
Copy link
 #11

Updated by icy almost 13 years ago

Please enable core-dumps and provide a stacktrace via gdb.
Or if it crashes very frequently or is producable just attach gdb to the running process.

Without that, we can't figure out at which part it's crashing.

BTW: how did you make sure you RAM is fine?

Actions
Copy link
 #12

Updated by redsolar almost 13 years ago

After getting the same error as marek, under the same OS (CentOS 5.7 x86_64), came across this issue, and did a backtrace in gdb with the following result:

This happens across multiple servers, and is not happening if i downgrade back to lighttpd 1.4.29

(gdb) backtrace
#0  ssl_info_callback (ssl=<value optimized out>, where=<value optimized out>,
    ret=<value optimized out>) at network.c:36
#1  0x00002b6bb0d547a2 in ssl23_accept () from /lib64/libssl.so.6
#2  0x00002b6bb0d55786 in ssl23_read () from /lib64/libssl.so.6
#3  0x000000000040ab22 in connection_handle_read (srv=0x14ac25e0,
    con=0x14b7ea40) at connections.c:224
#4  connection_handle_read_state (srv=0x14ac25e0, con=0x14b7ea40)
    at connections.c:907
#5  0x000000000040c6cf in connection_state_machine (srv=0x14ac25e0,
    con=0x14b7ea40) at connections.c:1655
#6  0x000000000040d59b in network_server_handle_fdevent (srv=0x14ac25e0,
    context=0x14b22fc0, revents=<value optimized out>) at network.c:64
#7  0x0000000000408548 in main (argc=<value optimized out>,
    argv=0x7fffed6ed828) at server.c:1470
Actions
Copy link
 #13

Updated by darix almost 13 years ago

redsolar: your bug should be fixed with applying http://redmine.lighttpd.net/projects/lighttpd/repository/revisions/2822 on top of 1.4.30

Actions
Copy link
 #14

Updated by marek almost 13 years ago

So this will be fixed in 1.4.31 ? I've tested my memory with memtest.

Actions
Copy link
 #15

Updated by redsolar almost 13 years ago

Thanks for the link to the patch, darix, everything works for me now

Actions
Copy link
 #16

Updated by marek almost 13 years ago

1.4.31-devel works fine for me too.

lighttpd/1.4.31-devel-2826 (ssl) - a light and fast webserver
Build-Date: Jan 14 2012 13:16:59

Actions
Copy link
 #17

Updated by stbuehler almost 13 years ago

  • Status changed from Need Feedback to Fixed

Fixed in r2822

Actions
Copy link
 #18

Updated by stbuehler almost 13 years ago

  • Status changed from Fixed to Missing Feedback

ah, i didn't have the initial report in mind.

  • the crashes in ssl are fixed in r2822.
  • the initial report (segfaults in libc/zlib): Missing Feedback.
Actions
Copy link
 #19

Updated by gstrauss 11 months ago

  • Status changed from Missing Feedback to Fixed
  • ASK QUESTIONS IN Forums set to No

mod_compress was replaced with mod_deflate (backported to lighttpd in lighttpd 1.4.42) and there are no known outstanding crashes.

Actions
Copy link

Also available in: Atom