Bug #1283

Memory usage increases when proxy+ssl+large file

Added by DeSnajpa about 7 years ago. Updated about 4 years ago.

Status:WontfixStart date:
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:mod_proxy
Target version:-
Missing in 1.5.x:No

Description

Requirements:
  • machine 1 with 128M-ram and lighttpd set up, and machine 2
  • both machines have debian testing/lenny 4.0 (upgraded 31-07-2007), compiled lighttpd 1.4.16 (not from repository), on machine 1 with ssl support, on both without ipv6, with bzip2; configured and runs and works
  • network setup: internet<-eth0->machine 1<-eth1->machine 2
  • machine 1 replies to *example.org requests; forwards *test.example.org requests to machine 2
  • machine 1 has 2 ports bound, ssl and http, both act for whole example.org; machine 2 only http
  • machine 1 hosts big (434M) file at http://ftp.example.org/filez/big.zip, machine2 hosts the fame file at machine 1 hosts big (434M) file http://ftp.test.example.org/filez/big.zip
Action:

I have found similar ticket, but it did not fit as I did not use cgi, but proxy, so posted this ticket.

Attaching lighttpd.conf's for both machines.

lighttpd.conf Magnifier - config of machine 1 (13 KB) DeSnajpa, 2007-08-01 20:02

lighttpd.conf.2 - config of machine 2 (4.94 KB) DeSnajpa, 2007-08-01 20:05

History

#1 Updated by Rubidium over 5 years ago

I've got a similar problem with (lighttpd + mod-proxy) + (apache2 + dav_svn).

With 1.4.22 checking out a subversion repository over http makes lighttpd consume about 1 MB per 7 MB of checked out data. When the checkout succesfully completes this memory will be freed. However, when the checkout gets cancelled lighttpd does not free the memory; it even finishes the checkout it is proxing to apache2. This means that a simple "request checkout" can cause apache2 to be streaming data to lighttpd and lighttpd buffers this without anyone ever reading it. Given a big enough repository once checked out (4 GB; lots of tags) and a "small" enough amount of memory (512 MB for lighttpd alone) you can cause OOM fairly easily. This can be replicated by having a large repository, checking it completely out and killing the subversion client that does the checkout.
The significant changes over a default Debian (Squeeze) lighttpd + apache2 + libapache2-svn installation, besides a large subversion repository) are changing the port for apache2 to e.g. 8081 and adding the following to lighttpd's configuration:
$HTTP["host"] == "localhost" {
proxy.server = ( "" => (("host" => "127.0.0.1", "port" => 8081)) )
}

With 1.5 r2529 some of the problems of 1.4.22 are aleviated; if you cancel a checkout the stream from apache2 to lighttpd is cut. However, if the client checking out is slow, the memory consumption of lighttpd will go up equally rapidly, again causing OOMs.
The significant changes over a default Debian (Squeeze) lighttpd + apache2 + libapache2-svn + manually compiled lighttpd1.5 installation, besides a large subversion repository) are changing the port for apache2 to e.g. 8081 and adding the following to lighttpd's configuration:
$HTTP["host"] == "localhost" {
proxy-core.protocol = "http"
proxy-core.backends = ( "127.0.0.1:8081" )
}

Finally something closely related to this: if one checkout (proxy) is running one cannot connect to the repository over http anymore. Once the streaming from apache2 to lighttpd has completed you can use the connection again, however with this you can do the checkout with slow client trick and increase the memory usage again. Given the 1 MB of extra server memory usage per 7 MB of checked out data this would mean that with a repository checkout of 1 GB and 7 subsequent runs you can fill 1 GB of the server's memory. With very minimal bandwidth on the client side and only a few lines in the logs that look like (harmless) checkouts. To me this looks like a denial of service!

What I would like is that:
1) the absurd amount of used memory is reclaimed when a client disconnects
2) lighttpd does not use an extreme amount of memory for this scenario, or at least make it configurable so you can limit the proxy 'cache' to e.g. 5 MB. If that limit is reached it should just stall the backend from sending data.
3) the checkout process is canceled when a client disconnects (this already works in 1.5 r2529.
4) the ability to allow multiple simultanious checkouts for a single proxy.

#2 Updated by mezza9 almost 5 years ago

Bump. This is also a significant issue for me. RAM usage starts below 1% and easily gets up to >50% on a 1.5GB RAM box after a few large SVN updates or commits.

Running 1.4.24 on Debian Etch.

#3 Updated by stbuehler almost 5 years ago

  • Status changed from New to Wontfix
  • Assignee deleted (jan)
  • Target version deleted (1.5.0)
  • Missing in 1.5.x set to No

Known issue, many reports, won't fix in 1.x.
Don't send big files via proxy, cgi, scgi, ...

#4 Updated by jbsnyder about 4 years ago

The solution here sounds roughly equivalent to the old joke:
Patient: It hurts when I do this.
Doctor: Well, then don't do it!

Unfortunately, I'm not sure what a good workaround to this might be if one had existing, rather large, repositories running on apache but migrated the front-end server on a machine to lighttpd. Should one use redirects to the alternate-port apache server? Or is there some other way to pass connections through that avoids this problem while allowing people with previously checked-out repositories to continue accessing the svn repo from the same urls.

Suggestions?

Also available in: Atom