https://redmine.lighttpd.net/https://redmine.lighttpd.net/favicon.ico?13667327412012-04-06T14:43:54Zlighty labsLighttpd - Bug #2405: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***https://redmine.lighttpd.net/issues/2405?journal_id=78212012-04-06T14:43:54Znitrox
<ul><li><strong>Priority</strong> changed from <i>Urgent</i> to <i>Normal</i></li></ul><p>"Urgent" and up is reserved for critical bugs.</p> Lighttpd - Bug #2405: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***https://redmine.lighttpd.net/issues/2405?journal_id=78232012-04-06T14:52:32ZDavidAnderson
<ul></ul><p>OK, thanks for the info. For future ref, What counts as critical? lighttpd not being able to parse a valid config file and hence dying before running was pretty critical for me!</p> Lighttpd - Bug #2405: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***https://redmine.lighttpd.net/issues/2405?journal_id=78242012-04-06T15:39:03ZDavidAnderson
<ul></ul><p>I've done some further testing.</p>
<p>It's not the size of the file directly - I managed to cut 20k (305kb -> 284kb) off by factoring some common elements. No difference in the outcome, except now sometimes I just get a segfault instead of glibc detecting the problem and aborting.</p>
<p>Even an empty fragment triggers the problem, e.g.<br />$HTTP["host"] =~ "^(<a class="external" href="http://www.)?mumia.example.org">www.)?mumia.example.org</a>$" {<br />}<br />i.e. With the empty fragment, lighttpd blows up. Without it, it doesn't.</p>
<p>It seems to be related to the number of fragments; if I remove one then all is OK. I seem to have approx. 750 $HTTP["host"] =~ sections in the file. Not all fragments seem equal however - some I can remove without ending the segfaulting - but some when removed stop the segfaulting. There is no clear pattern I can see; they are all regexes of the same kind of format.</p> Lighttpd - Bug #2405: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***https://redmine.lighttpd.net/issues/2405?journal_id=78252012-04-06T16:39:55ZDavidAnderson
<ul></ul><p>I think I've done as much testing as I can now - I also discovered that:</p>
<ul>
<li>I could remove about 40% of the $HTTP["host"] sections and still get the crash. But I cannot work out what the pattern is in which ones. As said before, I can remove just 1 section and avoid the crash, if I remove the right 1.</li>
</ul>
<ul>
<li>I copied the config over to another machine to test it there; same results.</li>
</ul> Lighttpd - Bug #2405: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***https://redmine.lighttpd.net/issues/2405?journal_id=78262012-04-06T17:05:33ZDavidAnderson
<ul></ul><p>I found the difference between crashing and not crashing....</p>
<p>I can take away 200 sections and it still crashes; but if I remove or even rewrite just <strong>one</strong> section that matches a certain pattern then the crashing stops.</p>
<p>If I add in ONE more section like:</p>
<p>$HTTP["host"] =~ "^(<a class="external" href="http://www.)?.example.org">www.)?.example.org</a>$" {</p>
<p>... then it crashes.</p>
<p>However if I add in:</p>
<p>$HTTP["host"] =~ "^www.example.org$" {<br />or even this one, which of course is equivalent to the one that crashes (so that's my work-around for now):<br />$HTTP["host"] =~ "^www.example.org$|^example.org$" {</p>
<p>... then no crash.</p>
<p>In an effort to get a simpler reproducible test case, I wrote a shell script that outputs 5000 sections like this:</p>
<p>$HTTP["host"] =~ "^(<a class="external" href="http://www.)?1.example.com$|^frubble1.zuggy.example.com">www.)?1.example.com$|^frubble1.zuggy.example.com</a>$" {
# Nothing to see here.<br />}</p>
<p>$HTTP["host"] =~ "^(<a class="external" href="http://www.)?2.example.com$|^frubble2.zuggy.example.com">www.)?2.example.com$|^frubble2.zuggy.example.com</a>$" {
# Nothing to see here.<br />}</p>
<p>etc.</p>
<p>However, that didn't produce a crash, even when added to the end of my base config (i.e. my config before the config for any individual website - before all my $HTTP["host"] =~ sections). So, the heap corruption is more subtle than just the number of such sections. Too subtle for someone like me who knows no C.</p>
<p>Anything else I can do to help, or does that give you enough data to work on?</p> Lighttpd - Bug #2405: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***https://redmine.lighttpd.net/issues/2405?journal_id=78272012-04-06T17:16:55ZDavidAnderson
<ul></ul><p>Nope, it's more temperamental still.</p>
<p>Changing <strong>one</strong> such section fixed things.</p>
<p>But when I changed my code to output <strong>all</strong> sections in that new style, it was back to crashing.</p>
<p>Now in the new style, I find that changing just one section from:</p>
<p>$HTTP["host"] =~ "^www.example.org" {</p>
<p>to:</p>
<p>$HTTP["host"] =~ "www.example.org" {</p>
<p>is the difference between crashing and not crashing. (the difference is the ^ , in case you didn't spot it).</p> Lighttpd - Bug #2405: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***https://redmine.lighttpd.net/issues/2405?journal_id=78282012-04-06T21:29:09Zstbuehler
<ul></ul><p>You could run it with valgrind; that might show the source of the memory corruption.</p> Lighttpd - Bug #2405: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***https://redmine.lighttpd.net/issues/2405?journal_id=78292012-04-07T08:54:31ZDavidAnderson
<ul><li><strong>File</strong> <a href="/attachments/1356">valgrind.txt</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/1356/valgrind.txt">valgrind.txt</a> added</li></ul><p>Thank you... I've attached the output of testing a known-bad config file (that produces the glibc abort):</p>
<p>valgrind -v --leak-check=yes lighttpd -t -f lighttpd.conf.faulty</p>
<p>Finishes with:<br />27998 ERROR SUMMARY: 642 errors from 67 contexts (suppressed: 43 from 8)</p> Lighttpd - Bug #2405: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***https://redmine.lighttpd.net/issues/2405?journal_id=78302012-04-07T08:56:31ZDavidAnderson
<ul></ul><p>For comparison, the presently running config file (one that doesn't cause a crash) ends with:</p>
28573 ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 43 from 8) Lighttpd - Bug #2405: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***https://redmine.lighttpd.net/issues/2405?journal_id=78312012-04-07T14:12:31Zstbuehler
<ul></ul><p>Hm, strange.</p>
<ul>
<li>It would be helpful if you can get debug symbols for lighttpd (compile with -g, perhaps there is a separate package for them).</li>
<li>Do you/CentOS apply patches we don't have upstream?</li>
<li>Perhaps you could share a "faulty" config; i guess it is triggered by the conditionals, so you probably can remove all the options. (If you don't want it public you can mail it to <a class="email" href="mailto:security@lighttpd.net">security@lighttpd.net</a> or <a class="email" href="mailto:stbuehler@lighttpd.net">stbuehler@lighttpd.net</a>)</li>
<li>Are you using "global { ... }" blocks?</li>
</ul> Lighttpd - Bug #2405: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***https://redmine.lighttpd.net/issues/2405?journal_id=78322012-04-07T17:06:26ZDavidAnderson
<ul><li><strong>File</strong> <a href="/attachments/1357">valgrind-withsymbols.txt</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/1357/valgrind-withsymbols.txt">valgrind-withsymbols.txt</a> added</li></ul><p>Thank you...</p>
<p>1) I installed the debuginfo package. A new valgrind output is attached.</p>
<p>This is for 1.4.28 from EPEL current. (EPEL = Extra Packages for Enterprise Linux - packages from the Fedora community for RHEL 5 and all clones, e.g. CentOS). As I say, I got the same problem in 1.4.30 from EPEL testing (but I had other problems with that one, which I reported to bugzilla.redhat.com, and then down-graded back). So I assume the issue is present the same in both.</p>
<p>2) I downloaded the EPEL source package. There was only one patch, which was to enable mod_geoip.</p>
<p>3) I'll email you the config file that produced this output. Please let me know if you don't get it immediately after I post this. It's 300kb, though I got the same problems on ones less than 200kb (it used to be over 500k until I did some refactoring a few months ago - with more refactoring yesterday whilst testing I got it below 200kb).</p>
<p>4) No, I wasn't aware of the existence of global { } until I was googling yesterday to look into this problem and found it mentioned in a different bug.</p> Lighttpd - Bug #2405: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***https://redmine.lighttpd.net/issues/2405?journal_id=78332012-04-07T17:15:01ZDavidAnderson
<ul></ul><p>One other piece of info - the bug seems widely reproducible. With the same config file you get the same error on Cent OS 5 32 bit, Cent OS 6 64 bit and Fedora 16 64 bit every time.</p> Lighttpd - Bug #2405: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***https://redmine.lighttpd.net/issues/2405?journal_id=78342012-04-08T09:32:01Zstbuehler
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li></ul><p>ok, i found the problem: buffer_caseless_compare isn't transitive:</p>
<pre>
(gdb) p buffer_caseless_compare("B", 1, "_", 1)
$1 = -29
(gdb) p buffer_caseless_compare("_", 1, "a", 1)
$2 = -2
(gdb) p buffer_caseless_compare("B", 1, "a", 1)
$3 = 1
</pre>
<p>So "B" < "_" and "_" < "a", but "B" > "a". buffer_caseless_compare is used to build binary trees to store the conditional blocks (the keys are the "paths" of conditions to a block).</p>
<p>In some cases the parser won't detect if you used a condition twice, and tries to insert it a second time; in the first tree this doesn't matter much, it just inserts a second entry with the same key (it didn't find the already existing entry!); but a second array detects the duplicate entry and frees the first, leading to dangling references and memory corruptions. (the funny part: the second tree isn't actually used)</p>
<p>These trees (called "array" in the code...) are used for other things too (like http headers). As one entry is usually only used in one array there shouldn't be any dangling references, but i didn't the other usages yet.</p> Lighttpd - Bug #2405: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***https://redmine.lighttpd.net/issues/2405?journal_id=78352012-04-08T10:05:10Zstbuehler
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Fixed</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>Applied in changeset r2828.</p> Lighttpd - Bug #2405: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***https://redmine.lighttpd.net/issues/2405?journal_id=78362012-04-09T08:47:51ZDavidAnderson
<ul></ul><p>Thanks very much. Works for me now (I recompiled 1.4.28 current from EPEL with that extra patch).</p> Lighttpd - Bug #2405: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***https://redmine.lighttpd.net/issues/2405?journal_id=78372012-04-09T10:24:57Zstbuehler
<ul></ul><p>Nice!</p>
<p>Thank you very much for the bug report and all the details :)</p> Lighttpd - Bug #2405: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***https://redmine.lighttpd.net/issues/2405?journal_id=105672016-10-18T16:15:12Zgstrauss
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-10 priority-4 priority-default closed" href="/issues/2404">Bug #2404</a>: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***</i> added</li></ul> Lighttpd - Bug #2405: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***https://redmine.lighttpd.net/issues/2405?journal_id=105692016-10-18T16:15:37Zgstrauss
<ul><li><strong>Related to</strong> deleted (<i><a class="issue tracker-1 status-10 priority-4 priority-default closed" href="/issues/2404">Bug #2404</a>: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***</i>)</li></ul> Lighttpd - Bug #2405: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***https://redmine.lighttpd.net/issues/2405?journal_id=105712016-10-18T16:15:49Zgstrauss
<ul><li><strong>Has duplicate</strong> <i><a class="issue tracker-1 status-10 priority-4 priority-default closed" href="/issues/2404">Bug #2404</a>: Crash parsing configuration: *** glibc detected *** /usr/sbin/lighttpd: malloc(): memory corruption (fast): 0x0a0354d8 ***</i> added</li></ul>