Bug #1879
closedLighttpd 1.4.20 Crash (SIGBUS in mod_compress)
Description
Lighttpd 1.4.20
Denial of Service Crash (low risk)
I inadvertently stumbled onto a Lighttpd crash while using a CSS fuzzer that I wrote. I was using lighttpd to serve up the fuzzed CSS files when I realized it kept crashing for no apparent reason. It kept crashing with a SIGBUS error and I tracked it down to a memcpy in zlib. I did some quick debugging work and here is what I came up with. If you find something inaccurate or need more details let me know.
On line 437 of mod_compress.c the file descriptor 'ifd' is assigned to the file requested and it is opened.
The file handle is sent to stat_cache_lstat() so the 'st' struct can be filled in. On line 458 of mod_compress.c lighttpd mmaps the file being requested using the file descriptor 'ifd', the pointer 'start' points to it and the
length argument to mmap is that of sce->st.size (the file size at the time of lstat()).
This can be proven by opening lighttpd in GDB and breaking on deflate_file_to_buffer_gzip(). Notice the st_size argument. On the first one its 3325952 and on the second its 9965568. This indicates the entire file has not been
written to disk yet. (It is about 12mb when complete dumped to disk).
A memcpy in the zlib function fill_window() is responsible for the SIGBUS error, most likely due to trying to copy memory from an address that is not currently mapped.
Responsible call chain:
mod_compress_physical -> deflate_file_to_file -> deflate_file_to_buffer_gzip -> (zlib) deflate -> (zlib) fill_window -> memcpy
GDB Output
(gdb) break deflate_file_to_buffer_gzip
Breakpoint 1 at 0xb7df13b4: file mod_compress.c, line 198.
(gdb) cont
Breakpoint 1, deflate_file_to_buffer_gzip (srv=0x89f6008, con=0x8a35718,
p=0x8a312f0,
start=0xb7a53000 "absolute { absolute: ", '9' <repeats 15 times>, ";
}\nabsolute:absolute{ !0?;=aaa}\nabsolute {
absolute:%0c;%0c\\t#??px{{{{{{}}}}}}(*%ffpx%0a%0000:;pt~::;<>:;%0c%0d[]&%000a?%000a)(@@===::><*@@end%ff!-1==%0a^(?@css#@>"...,
st_size=3325952, mtime=1229811418) at mod_compress.c:198
warning: Source file is more recent than executable.
198 static int deflate_file_to_buffer_gzip(server *srv, connection *con,
plugin_data *p, char *start, off_t st_size, time_t mtime) {
(gdb) cont
Continuing.
Breakpoint 1, deflate_file_to_buffer_gzip (srv=0x89f6008, con=0x8a35718,
p=0x8a312f0,
start=0xb6d54000 "absolute { absolute: ", '9' <repeats 15 times>, ";
}\nabsolute:absolute{ ;:;<>~%000aaaa}\nabsolute {
absolute:em%0a!#0000;~^@css%?::}{.%n-+:{{#;|[]&%;%0c;:;<>%;%ff%0a><%000a)()_%%ff;^~`}{.%n)_em%ff#-1><0%0000==@end0"...,
st_size=9965568, mtime=1229811438) at mod_compress.c:198
198 static int deflate_file_to_buffer_gzip(server *srv, connection *con,
plugin_data *p, char *start, off_t st_size, time_t mtime) {
(gdb) cont
Continuing.
Program received signal SIGBUS, Bus error.
0xb7e76b56 in memcpy () from /lib/tls/i686/cmov/libc.so.6
(gdb) bt
#0 0xb7e76b56 in memcpy () from /lib/tls/i686/cmov/libc.so.6
#1 0x08a69900 in ?? ()
#2 0xb7dda87d in deflate_slow (s=0xb73a0000, flush=4) at deflate.c:1601
#3 0xb7dd8f32 in deflate (strm=0xbfaae1a0, flush=4) at deflate.c:822
#4 0xb7df14fb in deflate_file_to_buffer_gzip (srv=<value optimized out>,
con=<value optimized out>, p=0x8a312f0,
start=0xb6d54000 "absolute { absolute: ", '9' <repeats 15 times>, ";
}\nabsolute:absolute{ %00}{.%n$0%aaa}\nabsolute {
absolute:-+:{{^pt)_-1^%00ff[]&%0000==!%;:%00-+:{{@end-
+:{{~~%00ff)_@end[]&^#0000%00ff*em%0cpx-+:{{-+:{{pt@css;-+:{"...,
st_size=<value optimized out>, mtime=<value optimized out>) at
mod_compress.c:245
#5 0xb7df2455 in mod_compress_physical (srv=0x89f6008, con=0x8a35718,
p_d=0x8a312f0) at mod_compress.c:468
#6 0x08060e16 in plugins_call_handle_subrequest_start (srv=0x89f6008,
con=0x8a35718) at plugin.c:268
#7 0x0804fe40 in http_response_prepare (srv=0x89f6008, con=0x8a35718) at
response.c:645
#8 0x0805314a in connection_state_machine (srv=0x89f6008, con=0x8a35718) at
connections.c:1426
#9 0x0804e7e0 in main (argc=4, argv=0xbfaaeb44) at server.c:1432
-- reported by Chris Rohlf
Updated by icy almost 16 years ago
Thank you for your mail Chris.
We are currently investigating the issue and so far have come to the conclusion that using mmap() can indeed result in a SIGBUS when the data behind the mmap()ed address cannot be read.
Usually this should only happen when the file in question is truncated after the stat() and we try to read from an address that is behind the end of the file.
But your backtrace suggests that the file is growing and has not been shrunken. The read should succeed.
Could you perform the following additional steps:
- provide more information about the OS in question (32bits?)
- install glibc debug symbols
- create a valgrind report
Knowing the address that memcpy() tries to read would be helpful.
Please also provide exact information (config e.g.) on how to reproduce the bug.
This might help us investigate the issue further.
Updated by stbuehler almost 16 years ago
Thanks for the reply. Here is all the info I can provide on short notice, I realize this is not the best bug report, but then again I never intended to find this :)
Ubuntu Server x86 32bit (I compiled lighttpd from source, I did not use the Ubuntu package)
The only configuration change was enabling mod_compress obviously. Nothing else.
I will have to get back to you on enabling libc-dbg and retesting this as I have to move to another machine to do that.
Heres the valgrind output:
valgrind -v ./lighttpd -D -f /etc/lighttpd/lighttpd.conf ==12667== Memcheck, a memory error detector. ==12667== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward et al. ==12667== Using LibVEX rev 1854, a library for dynamic binary translation. ==12667== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP. ==12667== Using valgrind-3.3.1-Debian, a dynamic binary instrumentation framework. ==12667== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward et al. ==12667== --12667-- Command line --12667-- ./lighttpd --12667-- -D --12667-- -f --12667-- /etc/lighttpd/lighttpd.conf --12667-- Startup, with flags: --12667-- --suppressions=/usr/lib/valgrind/debian-libc6-dbg.supp --12667-- -v --12667-- Contents of /proc/version: --12667-- Linux version 2.6.27-9-server (buildd@rothera) (gcc version 4.3.2 (Ubuntu 4.3.2-1ubuntu11) ) #1 SMP Thu Nov 20 22:53:41 UTC 2008 --12667-- Arch and hwcaps: X86, x86-sse1-sse2 --12667-- Page sizes: currently 4096, max supported 4096 --12667-- Valgrind library directory: /usr/lib/valgrind --12667-- Reading syms from /lib/ld-2.8.90.so (0x4000000) --12667-- Reading debug info from /lib/ld-2.8.90.so... --12667-- ... CRC mismatch (computed 371b8ee6 wanted cc0a418a) --12667-- Reading debug info from /usr/lib/debug/lib/ld-2.8.90.so... --12667-- Reading syms from /usr/src/lighttpd-1.4.20-testing/src/lighttpd (0x8048000) --12667-- Reading syms from /usr/lib/valgrind/x86-linux/memcheck (0x38000000) --12667-- object doesn't have a dynamic symbol table --12667-- Reading suppressions file: /usr/lib/valgrind/debian-libc6-dbg.supp --12667-- Reading suppressions file: /usr/lib/valgrind/default.supp --12667-- REDIR: 0x40155d0 (index) redirected to 0x3802cf63 (vgPlain_x86_linux_REDIR_FOR_index) --12667-- Reading syms from /usr/lib/valgrind/x86-linux/vgpreload_core.so (0x401F000) --12667-- Reading syms from /usr/lib/valgrind/x86-linux/vgpreload_memcheck.so (0x4022000) ==12667== WARNING: new redirection conflicts with existing -- ignoring it --12667-- new: 0x040155d0 (index ) R-> 0x040261a0 index --12667-- REDIR: 0x40157c0 (strlen) redirected to 0x4026450 (strlen) --12667-- Reading syms from /lib/libpcre.so.3.12.1 (0x4031000) --12667-- Reading debug info from /lib/libpcre.so.3.12.1... --12667-- ... CRC mismatch (computed 9f2da4fa wanted 2ee8967a) --12667-- object doesn't have a symbol table --12667-- Reading syms from /usr/lib/debug/libdl-2.8.90.so (0x405B000) --12667-- Reading syms from /usr/lib/debug/libc-2.8.90.so (0x405F000) --12667-- REDIR: 0x40d22f0 (rindex) redirected to 0x4026080 (rindex) --12667-- REDIR: 0x40d1950 (strcmp) redirected to 0x4026730 (strcmp) --12667-- REDIR: 0x40cec50 (calloc) redirected to 0x4023d20 (calloc) --12667-- REDIR: 0x40cefb0 (malloc) redirected to 0x4025c70 (malloc) --12667-- REDIR: 0x40d1f00 (strlen) redirected to 0x4026430 (strlen) --12667-- REDIR: 0x40d3490 (memcpy) redirected to 0x40268a0 (memcpy) --12667-- REDIR: 0x40d17e0 (index) redirected to 0x4026170 (index) --12667-- REDIR: 0x40d2ec0 (memmove) redirected to 0x40273a0 (memmove) --12667-- REDIR: 0x40cc730 (free) redirected to 0x4024a90 (free) --12667-- REDIR: 0x40cf470 (realloc) redirected to 0x4025d80 (realloc) --12667-- REDIR: 0x40d2110 (strncmp) redirected to 0x40266a0 (strncmp) --12667-- Reading syms from /usr/local/lib/mod_indexfile.so (0x402A000) --12667-- Reading syms from /usr/local/lib/mod_access.so (0x402D000) --12667-- Reading syms from /usr/local/lib/mod_alias.so (0x45A3000) --12667-- Reading syms from /usr/local/lib/mod_accesslog.so (0x45A6000) --12667-- Reading syms from /usr/local/lib/mod_compress.so (0x45AB000) --12667-- Reading syms from /usr/lib/libz.so.1.2.3.3 (0x45B7000) --12667-- Reading debug info from /usr/lib/libz.so.1.2.3.3... --12667-- ... CRC mismatch (computed f60911fc wanted bd516366) --12667-- Reading debug info from /usr/lib/debug/usr/lib/libz.so.1.2.3.3... --12667-- Reading syms from /lib/libbz2.so.1.0.4 (0x45CD000) --12667-- object doesn't have a symbol table --12667-- Reading syms from /usr/local/lib/mod_dirlisting.so (0x45B0000) --12667-- Reading syms from /usr/local/lib/mod_staticfile.so (0x45DE000) --12667-- REDIR: 0x40d2d00 (memchr) redirected to 0x4026850 (memchr) --12667-- REDIR: 0x40d4010 (strchrnul) redirected to 0x4027410 (strchrnul) --12667-- REDIR: 0x40d2fd0 (mempcpy) redirected to 0x4027470 (mempcpy) --12667-- REDIR: 0x40d3180 (stpcpy) redirected to 0x40270d0 (stpcpy) --12667-- Reading syms from /usr/lib/debug/libnss_compat-2.8.90.so(0x45E3000) --12667-- Reading syms from /usr/lib/debug/libnsl-2.8.90.so (0x45EC000) --12667-- REDIR: 0x40d19e0 (strcpy) redirected to 0x4026490 (strcpy) --12667-- Reading syms from /usr/lib/debug/libnss_nis-2.8.90.so (0x4603000) --12667-- Reading syms from /usr/lib/debug/libnss_files-2.8.90.so(0x460D000) --12667-- REDIR: 0x40d3f40 (rawmemchr) redirected to 0x4027450 (rawmemchr) --12667-- REDIR: 0x40d2220 (strncpy) redirected to 0x4026560 (strncpy) --12667-- REDIR: 0x40d2f70 (memset) redirected to 0x4027340 (memset) ==12667== ==12667== Process terminating with default action of signal 7 (SIGBUS) ==12667== Non-existent physical address at address 0x4768000 ==12667== at 0x40269F0: memcpy (mc_replace_strmem.c:402) ==12667== by 0x45BBB0D: fill_window (string3.h:52) ==12667== by 0x45BD87C: deflate_slow (deflate.c:1601) ==12667== by 0x45BBF31: deflate (deflate.c:822) ==12667== by 0x45AC4FA: deflate_file_to_buffer_gzip (mod_compress.c:245) ==12667== by 0x45AD454: mod_compress_physical (mod_compress.c:468) ==12667== by 0x8060E15: plugins_call_handle_subrequest_start (plugin.c:268) ==12667== by 0x804FE3F: http_response_prepare (response.c:645) ==12667== by 0x8053149: connection_state_machine (connections.c:1426) ==12667== by 0x80541EB: network_server_handle_fdevent (network.c:51) ==12667== by 0x804EA9E: main (server.c:1407) --12667-- Discarding syms at 0x45E3000-0x45EC000 in /usr/lib/debug/ libnss_compat-2.8.90.so due to munmap() --12667-- Discarding syms at 0x4603000-0x460D000 in /usr/lib/debug/ libnss_nis-2.8.90.so due to munmap() --12667-- Discarding syms at 0x45EC000-0x4603000 in /usr/lib/debug/ libnsl-2.8.90.so due to munmap() --12667-- Discarding syms at 0x460D000-0x4618000 in /usr/lib/debug/ libnss_files-2.8.90.so due to munmap() ==12667== ==12667== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 68 from 1) --12667-- --12667-- supp: 68 dl-hack3-cond-1 ==12667== malloc/free: in use at exit: 11,584,374 bytes in 9,122 blocks. ==12667== malloc/free: 16,050 allocs, 6,928 frees, 12,187,238 bytes allocated. ==12667== ==12667== searching for pointers to 9,122 not-freed blocks. ==12667== checked 885,176 bytes. ==12667== ==12667== LEAK SUMMARY: ==12667== definitely lost: 444 bytes in 31 blocks. ==12667== possibly lost: 0 bytes in 0 blocks. ==12667== still reachable: 11,583,930 bytes in 9,091 blocks. ==12667== suppressed: 0 bytes in 0 blocks. ==12667== Rerun with --leak-check=full to see details of leaked memory. --12667-- memcheck: sanity checks: 285 cheap, 11 expensive --12667-- memcheck: auxmaps: 0 auxmap entries (0k, 0M) in use --12667-- memcheck: auxmaps_L1: 0 searches, 0 cmps, ratio 0:10 --12667-- memcheck: auxmaps_L2: 0 searches, 0 nodes --12667-- memcheck: SMs: n_issued = 55 (880k, 0M) --12667-- memcheck: SMs: n_deissued = 0 (0k, 0M) --12667-- memcheck: SMs: max_noaccess = 65535 (1048560k, 1023M) --12667-- memcheck: SMs: max_undefined = 166 (2656k, 2M) --12667-- memcheck: SMs: max_defined = 174 (2784k, 2M) --12667-- memcheck: SMs: max_non_DSM = 55 (880k, 0M) --12667-- memcheck: max sec V bit nodes: 17 (0k, 0M) --12667-- memcheck: set_sec_vbits8 calls: 61 (new: 17, updates: 44) --12667-- memcheck: max shadow mem size: 1184k, 1M --12667-- translate: fast SP updates identified: 7,440 ( 87.5%) --12667-- translate: generic_known SP updates identified: 782 ( 9.2%) --12667-- translate: generic_unknown SP updates identified: 274 ( 3.2%) --12667-- tt/tc: 15,606 tt lookups requiring 16,282 probes --12667-- tt/tc: 15,606 fast-cache updates, 7 flushes --12667-- transtab: new 6,872 (156,868 -> 2,194,828; ratio 139:10) [0 scs] --12667-- transtab: dumped 0 (0 -> ??) --12667-- transtab: discarded 268 (5,057 -> ??) --12667-- scheduler: 28,532,439 jumps (bb entries). --12667-- scheduler: 285/32,263 major/minor sched events. --12667-- sanity: 286 cheap, 11 expensive checks. --12667-- exectx: 769 lists, 711 contexts (avg 0 per list) --12667-- exectx: 23,046 searches, 22,713 full compares (985 per 1000) --12667-- exectx: 0 cmp2, 290 cmp4, 0 cmpAll --12667-- errormgr: 11 supplist searches, 96 comparisons during search --12667-- errormgr: 68 errlist searches, 290 comparisons during search Bus error
Oh and as far as reproducing the bug heres an easy how-to (it basically repros what I was using).
1. Create an HTML page that loads a CSS file using a LINK tag and a meta
refresh tag
<meta http-equiv="refresh" content="1">
<LINK href="BADCSS.css" rel="stylesheet" type="text/css">
2. Load this page up in a browser.
3. Write a script that continuously dumps into the CSS file. Mine looks like
this
for i in `seq 1 5000`; do
ruby cssfuzz.rb > BADCSS.css
sleep 1
done
Sorry but I cant share the ruby file. It should be trivial to recreate this scenario though.
-- Chris Rohlf
Updated by gstrauss over 8 years ago
ruby cssfuzz.rb > BADCSS.css
Doing that in a loop, the shell will truncate BADCSS.css to 0-bytes, and then will execute ruby which will send its output to BADCSS.css. Note that this writes into the same inode on disk. Anything in the middle of reading a prior, non-zero length, version of the file via an mmap() mapping is at risk of suddenly having the mapped pages disappear and receiving a SIGBUS on the next attempt to read (now) invalid mapped pages.
Updated by gstrauss over 8 years ago
- Category set to mod_compress
- Status changed from New to Fixed
- Target version set to 1.4.x
This was fixed in https://redmine.lighttpd.net/issues/2391 because changes were made in mod_compress.c to wrap the mmap() code in #if defined(USE_MMAP). Since mod_compress.c does not #include "network_backends.h", USE_MMAP was never defined in mod_compress.c, and consequently, mmap() is never used in mod_compress. (Note: given the performance implications, I'll probably re-enable mmap() in mod_compress.c along with trapping SIGBUS, but both only when USE_MMAP is enabled.)
commit f4c3a99eea1edbcb6b6d084b30f568db64419620
Author: Stefan Bühler <stbuehler@web.de>
Date: Fri Feb 24 18:34:20 2012 +0000
Disable mmap by default (fixes #2391)
* if a user truncates a file we are mmapping, reading the truncated
area leads to SIGBUS
* mod_cgi and mod_webdav still use mmap for reading the tmp files
created for large request bodies.
as no other user should have write access for those this isn't
a (security) problem.
git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@2827 152afb58-edef-0310-8abb-c4023f1b3aa9
Updated by gstrauss over 8 years ago
submitted pull request https://github.com/lighttpd/lighttpd1.4/pull/56 to enhance mod_compress to use mmap() and trap SIGBUS, if lighttpd built with --enable-mmap
Updated by stbuehler over 8 years ago
- Target version changed from 1.4.x to 1.4.40
- Private changed from Yes to No
Also available in: Atom