Bug #2147

slow request dos/oom attack

Added by stbuehler over 4 years ago. Updated almost 4 years ago.

Status:FixedStart date:2010-01-06
Priority:UrgentDue date:
Assignee:-% Done:

100%

Category:-
Target version:-
Missing in 1.5.x:

Description

Hi, all

I found a serious problem.
I wrote a C program which sends a HTTP request to lighttpd very slowly, one byte each second (write 1 bytes, then sleep 1 second).

And I also wrote a shell script as below:

##slow_test.sh
for ((j=0;j<1000;j++)) do
  for ((i=0; i<50; i++)) do
  ## slow_client is a C program which sends a HTTP request very slowly
    ./slow_client http://xxx.xxx.xxx.xxx:8080/>/dev/null 2>/dev/null &
  done&
  sleep 3
done

I started slow_test.sh. After a few minutes, lighttpd had occupied much memory, and finally all of 4G memory was exhausted, the server machine was unable to serve any more.

I read the source code, and found lighttpd will allocate a new chunk(16384 bytes) every time for the new request data which triggered by the event FDEVENT_IN. But the problem is, once FDEVENT_IN arrives, the new data is always stored in a new chunk, no matter how many bytes have been received at the monent. So it will be a waste of memory.

I suggest stroring request bytes in a compact way. If last chunk is not full, new data should be put into the last chunk. I submit my patch(based on lighttpd 1.5.0 svn r2700) in attachment.

Thanks

Ming, Li

network_openssl.patch Magnifier (1.33 KB) stbuehler, 2010-01-06 14:12

network_write.patch Magnifier (1.41 KB) stbuehler, 2010-01-06 14:12

fix-2147-in-1.4.x.patch Magnifier - Fix for 1.4.x (6.93 KB) stbuehler, 2010-01-21 23:32

fix-2147-in-1.4.x.patch Magnifier - Fix for 1.4.x (version 2 - fix read_offset calculation) (6.97 KB) stbuehler, 2010-01-22 17:47

fix-2147-in-1.5.patch Magnifier - Fix for 1.5 (5.76 KB) stbuehler, 2010-01-22 17:47

fix-2147-in-1.5.patch Magnifier - Fix for 1.5 (s/buffer_copy_string_len/buffer_append_string_len/) (5.82 KB) stbuehler, 2010-01-29 18:31

Associated revisions

Revision 2710
Added by stbuehler over 4 years ago

Append to previous buffer in con read (fixes #2147, found by liming, CVE-2010-0295)

  • Remove ssl_error_want_reuse_buffer for SSL_read:
    Although the manual states we have to use the same arguments in the
    next call after SSL_ERROR_WANT_*, it has been running without this
    in 1.5 for a long time now.
  • As POST-data chunks get copied to the next queue, we reuse chunks
    there as well.

Revision 2711
Added by stbuehler over 4 years ago

Append to previous buffer in con read (fixes #2147, found by liming, CVE-2010-0295)

History

#1 Updated by stbuehler over 4 years ago

So, i think i have a patch for 1.4.x:

  • Remove ssl_error_want_reuse_buffer for SSL_read:
    Although the manual states we have to use the same arguments in the next call after SSL_ERROR_WANT_*, it has been running without this in 1.5 for a long time now.
  • As POST-data chunks get copied to the next queue, we reuse chunks there as well.

#2 Updated by stbuehler over 4 years ago

  • File slowclient.sh added

I wrote a small shell script which does "slow" upload (testing with 60000 bytes post data, small header).
On 1.4.x the effect is not that bad ("only" a small factor of wasted memory), as the buffers were allocated with the ioctl FIONREAD size. On system which don't support this it is probably worse (4k pages).
The above patch seems to have fixed that problem for 1.4.x. Didn't test ssl yet.

#3 Updated by stbuehler over 4 years ago

To test ssl I replaced nc 127.0.0.1 8081 with openssl s_client -connect 127.0.0.1:8443 in the shell script.
1.5 already has some improved request handling and will throw away POST-data for static files immediately.

Attached new fix for 1.4.x and fix for 1.5.

#4 Updated by stbuehler over 4 years ago

Fix a bug in the patch for 1.5 (forgot to replace buffer_copy_string_len with buffer_append_string_len).

#5 Updated by stbuehler over 4 years ago

  • File deleted (slowclient.sh)

#6 Updated by stbuehler over 4 years ago

  • Project changed from internal to Lighttpd

#7 Updated by stbuehler over 4 years ago

  • Status changed from New to Fixed
  • % Done changed from 0 to 100

Applied in changeset r2710.

Also available in: Atom