Project

General

Profile

Actions

Bug #2147

closed

slow request dos/oom attack

Added by stbuehler about 14 years ago. Updated over 13 years ago.

Status:
Fixed
Priority:
Urgent
Category:
-
Target version:
-
ASK QUESTIONS IN Forums:

Description

Hi, all

I found a serious problem.
I wrote a C program which sends a HTTP request to lighttpd very slowly, one byte each second (write 1 bytes, then sleep 1 second).

And I also wrote a shell script as below:

##slow_test.sh
for ((j=0;j<1000;j++)) do
  for ((i=0; i<50; i++)) do
  ## slow_client is a C program which sends a HTTP request very slowly
    ./slow_client http://xxx.xxx.xxx.xxx:8080/>/dev/null 2>/dev/null &
  done&
  sleep 3
done

I started slow_test.sh. After a few minutes, lighttpd had occupied much memory, and finally all of 4G memory was exhausted, the server machine was unable to serve any more.

I read the source code, and found lighttpd will allocate a new chunk(16384 bytes) every time for the new request data which triggered by the event FDEVENT_IN. But the problem is, once FDEVENT_IN arrives, the new data is always stored in a new chunk, no matter how many bytes have been received at the monent. So it will be a waste of memory.

I suggest stroring request bytes in a compact way. If last chunk is not full, new data should be put into the last chunk. I submit my patch(based on lighttpd 1.5.0 svn r2700) in attachment.

Thanks

Ming, Li


Files

network_openssl.patch (1.33 KB) network_openssl.patch stbuehler, 2010-01-06 14:12
network_write.patch (1.41 KB) network_write.patch stbuehler, 2010-01-06 14:12
fix-2147-in-1.4.x.patch (6.93 KB) fix-2147-in-1.4.x.patch Fix for 1.4.x stbuehler, 2010-01-21 23:32
fix-2147-in-1.4.x.patch (6.97 KB) fix-2147-in-1.4.x.patch Fix for 1.4.x (version 2 - fix read_offset calculation) stbuehler, 2010-01-22 17:47
fix-2147-in-1.5.patch (5.76 KB) fix-2147-in-1.5.patch Fix for 1.5 stbuehler, 2010-01-22 17:47
fix-2147-in-1.5.patch (5.82 KB) fix-2147-in-1.5.patch Fix for 1.5 (s/buffer_copy_string_len/buffer_append_string_len/) stbuehler, 2010-01-29 18:31
Actions #1

Updated by stbuehler about 14 years ago

So, i think i have a patch for 1.4.x:

  • Remove ssl_error_want_reuse_buffer for SSL_read:
    Although the manual states we have to use the same arguments in the next call after SSL_ERROR_WANT_*, it has been running without this in 1.5 for a long time now.
  • As POST-data chunks get copied to the next queue, we reuse chunks there as well.
Actions #2

Updated by stbuehler about 14 years ago

  • File slowclient.sh added

I wrote a small shell script which does "slow" upload (testing with 60000 bytes post data, small header).
On 1.4.x the effect is not that bad ("only" a small factor of wasted memory), as the buffers were allocated with the ioctl FIONREAD size. On system which don't support this it is probably worse (4k pages).
The above patch seems to have fixed that problem for 1.4.x. Didn't test ssl yet.

Updated by stbuehler about 14 years ago

To test ssl I replaced nc 127.0.0.1 8081 with openssl s_client -connect 127.0.0.1:8443 in the shell script.
1.5 already has some improved request handling and will throw away POST-data for static files immediately.

Attached new fix for 1.4.x and fix for 1.5.

Actions #4

Updated by stbuehler about 14 years ago

Fix a bug in the patch for 1.5 (forgot to replace buffer_copy_string_len with buffer_append_string_len).

Actions #5

Updated by stbuehler about 14 years ago

  • File deleted (slowclient.sh)
Actions #6

Updated by stbuehler about 14 years ago

  • Project changed from 2 to Lighttpd
Actions #7

Updated by stbuehler about 14 years ago

  • Status changed from New to Fixed
  • % Done changed from 0 to 100

Applied in changeset r2710.

Actions

Also available in: Atom