Project

General

Profile

Bug #1283

Memory usage increases when proxy+ssl+large file

Added by DeSnajpa about 9 years ago. Updated 3 months ago.

Status:
Fixed
Priority:
Normal
Assignee:
-
Category:
mod_proxy
Target version:
Start date:
Due date:
% Done:

100%

Missing in 1.5.x:
No

Description

Requirements:
  • machine 1 with 128M-ram and lighttpd set up, and machine 2
  • both machines have debian testing/lenny 4.0 (upgraded 31-07-2007), compiled lighttpd 1.4.16 (not from repository), on machine 1 with ssl support, on both without ipv6, with bzip2; configured and runs and works
  • network setup: internet<-eth0->machine 1<-eth1->machine 2
  • machine 1 replies to *example.org requests; forwards *test.example.org requests to machine 2
  • machine 1 has 2 ports bound, ssl and http, both act for whole example.org; machine 2 only http
  • machine 1 hosts big (434M) file at http://ftp.example.org/filez/big.zip, machine2 hosts the fame file at machine 1 hosts big (434M) file http://ftp.test.example.org/filez/big.zip
Action:

I have found similar ticket, but it did not fit as I did not use cgi, but proxy, so posted this ticket.

Attaching lighttpd.conf's for both machines.

lighttpd.conf View - config of machine 1 (13 KB) DeSnajpa, 2007-08-01 20:02

lighttpd.conf.2 - config of machine 2 (4.94 KB) DeSnajpa, 2007-08-01 20:05


Related issues

Related to Bug #949: fastcgi, cgi, flush, php5 problem. Fixed

Associated revisions

Revision 5a91fd4b (diff)
Added by gstrauss 4 months ago

[core] buffer large responses to tempfiles (fixes #758, fixes #760, fixes #933, fixes #1387, #1283, fixes #2083)

This replaces buffering entire response in memory which might lead to
huge memory footprint and possibly to memory exhaustion.

use tempfiles of fixed size so disk space is freed as each file sent

update callers of http_chunk_append_mem() and http_chunk_append_buffer()
to handle failures when writing to tempfile.

x-ref:
"memory fragmentation leads to high memory usage after peaks"
https://redmine.lighttpd.net/issues/758
"Random crashing on FreeBSD 6.1"
https://redmine.lighttpd.net/issues/760
"lighty should buffer responses (after it grows above certain size) on disk"
https://redmine.lighttpd.net/issues/933
"Memory usage increases when proxy+ssl+large file"
https://redmine.lighttpd.net/issues/1283
"lighttpd+fastcgi memory problem"
https://redmine.lighttpd.net/issues/1387
"Excessive Memory usage with streamed files from PHP"
https://redmine.lighttpd.net/issues/2083

Revision 5ab7944d (diff)
Added by gstrauss 3 months ago

[TLS] release openssl buffers as used (fixes #1265, fixes #1283, #881)

use SSL_MODE_RELEASE_BUFFERS (OpenSSL >= 1.0.0) to free buffers
as they are used, to potentially reduce memory footprint of
idle SSL connections

x-ref:
"memory usage when ssl.engine used and large data uploaded through CGI"
https://redmine.lighttpd.net/issues/881
"SSL + file upload = lots of memory"
https://redmine.lighttpd.net/issues/1265
"Memory usage increases when proxy+ssl+large file"
https://redmine.lighttpd.net/issues/1283

Revision 18a7b2be (diff)
Added by gstrauss 3 months ago

[core] option to stream response body to client (fixes #949, #760, #1283, #1387)

Set server.stream-response-body = 1 or server.stream-response-body = 2
to have lighttpd stream response body to client as it arrives from the
backend (CGI, FastCGI, SCGI, proxy).

default: buffer entire response body before sending response to client.
(This preserves existing behavior for now, but may in the future be
changed to stream response to client, which is the behavior more
commonly expected.)

x-ref:
"fastcgi, cgi, flush, php5 problem."
https://redmine.lighttpd.net/issues/949
"Random crashing on FreeBSD 6.1"
https://redmine.lighttpd.net/issues/760
"Memory usage increases when proxy+ssl+large file"
https://redmine.lighttpd.net/issues/1283
"lighttpd+fastcgi memory problem"
https://redmine.lighttpd.net/issues/1387

History

#1 Updated by Rubidium over 7 years ago

I've got a similar problem with (lighttpd + mod-proxy) + (apache2 + dav_svn).

With 1.4.22 checking out a subversion repository over http makes lighttpd consume about 1 MB per 7 MB of checked out data. When the checkout succesfully completes this memory will be freed. However, when the checkout gets cancelled lighttpd does not free the memory; it even finishes the checkout it is proxing to apache2. This means that a simple "request checkout" can cause apache2 to be streaming data to lighttpd and lighttpd buffers this without anyone ever reading it. Given a big enough repository once checked out (4 GB; lots of tags) and a "small" enough amount of memory (512 MB for lighttpd alone) you can cause OOM fairly easily. This can be replicated by having a large repository, checking it completely out and killing the subversion client that does the checkout.
The significant changes over a default Debian (Squeeze) lighttpd + apache2 + libapache2-svn installation, besides a large subversion repository) are changing the port for apache2 to e.g. 8081 and adding the following to lighttpd's configuration:
$HTTP["host"] == "localhost" {
proxy.server = ( "" => (("host" => "127.0.0.1", "port" => 8081)) )
}

With 1.5 r2529 some of the problems of 1.4.22 are aleviated; if you cancel a checkout the stream from apache2 to lighttpd is cut. However, if the client checking out is slow, the memory consumption of lighttpd will go up equally rapidly, again causing OOMs.
The significant changes over a default Debian (Squeeze) lighttpd + apache2 + libapache2-svn + manually compiled lighttpd1.5 installation, besides a large subversion repository) are changing the port for apache2 to e.g. 8081 and adding the following to lighttpd's configuration:
$HTTP["host"] == "localhost" {
proxy-core.protocol = "http"
proxy-core.backends = ( "127.0.0.1:8081" )
}

Finally something closely related to this: if one checkout (proxy) is running one cannot connect to the repository over http anymore. Once the streaming from apache2 to lighttpd has completed you can use the connection again, however with this you can do the checkout with slow client trick and increase the memory usage again. Given the 1 MB of extra server memory usage per 7 MB of checked out data this would mean that with a repository checkout of 1 GB and 7 subsequent runs you can fill 1 GB of the server's memory. With very minimal bandwidth on the client side and only a few lines in the logs that look like (harmless) checkouts. To me this looks like a denial of service!

What I would like is that:
1) the absurd amount of used memory is reclaimed when a client disconnects
2) lighttpd does not use an extreme amount of memory for this scenario, or at least make it configurable so you can limit the proxy 'cache' to e.g. 5 MB. If that limit is reached it should just stall the backend from sending data.
3) the checkout process is canceled when a client disconnects (this already works in 1.5 r2529.
4) the ability to allow multiple simultanious checkouts for a single proxy.

#2 Updated by mezza9 almost 7 years ago

Bump. This is also a significant issue for me. RAM usage starts below 1% and easily gets up to >50% on a 1.5GB RAM box after a few large SVN updates or commits.

Running 1.4.24 on Debian Etch.

#3 Updated by stbuehler almost 7 years ago

  • Status changed from New to Wontfix
  • Assignee deleted (jan)
  • Target version deleted (1.5.0)
  • Missing in 1.5.x set to No

Known issue, many reports, won't fix in 1.x.
Don't send big files via proxy, cgi, scgi, ...

#4 Updated by jbsnyder about 6 years ago

The solution here sounds roughly equivalent to the old joke:
Patient: It hurts when I do this.
Doctor: Well, then don't do it!

Unfortunately, I'm not sure what a good workaround to this might be if one had existing, rather large, repositories running on apache but migrated the front-end server on a machine to lighttpd. Should one use redirects to the alternate-port apache server? Or is there some other way to pass connections through that avoids this problem while allowing people with previously checked-out repositories to continue accessing the svn repo from the same urls.

Suggestions?

#5 Updated by gstrauss 4 months ago

  • Related to Bug #949: fastcgi, cgi, flush, php5 problem. added

#6 Updated by gstrauss 4 months ago

  • Status changed from Wontfix to Patch Pending
  • Target version set to 1.4.40

New: asynchronous, bidirectional streaming support for request and response
Submitted pull request: https://github.com/lighttpd/lighttpd1.4/pull/66

included in the pull request are flags to openssl (SSL_MODE_RELEASE_BUFFERS) to release memory buffers when finished with them, instead of holding onto the buffers.

#7 Updated by gstrauss 3 months ago

  • Status changed from Patch Pending to Fixed
  • % Done changed from 0 to 100

Also available in: Atom