Bug #1283
closedMemory usage increases when proxy+ssl+large file
Description
- machine 1 with 128M-ram and lighttpd set up, and machine 2
- both machines have debian testing/lenny 4.0 (upgraded 31-07-2007), compiled lighttpd 1.4.16 (not from repository), on machine 1 with ssl support, on both without ipv6, with bzip2; configured and runs and works
- network setup: internet<-eth0->machine 1<-eth1->machine 2
- machine 1 replies to *example.org requests; forwards *test.example.org requests to machine 2
- machine 1 has 2 ports bound, ssl and http, both act for whole example.org; machine 2 only http
- machine 1 hosts big (434M) file at http://ftp.example.org/filez/big.zip, machine2 hosts the fame file at machine 1 hosts big (434M) file http://ftp.test.example.org/filez/big.zip
- if remote host requests file https://ftp.example.org/filez/big.zip (internet<->machine 1), headers and file will be sent
- if remote host requests either http://ftp.example.org/filez/big.zip or http://ftp.test.example.org/filez/big.zip , he will receive headers and file
- if remote host requests file https://ftp.test.example.org/filez/big.zip (internet<->machine 1<->machine 2), lighttpd will use as much RAM as the file size and never release it (seen in
top
, memory usage gets bigger and bigger). Because server has much less memory, will occur out-of-memory-error and neither headers nor file will be sent.
I have found similar ticket, but it did not fit as I did not use cgi, but proxy, so posted this ticket.
Attaching lighttpd.conf's for both machines.
Files
Updated by Rubidium over 15 years ago
I've got a similar problem with (lighttpd + mod-proxy) + (apache2 + dav_svn).
With 1.4.22 checking out a subversion repository over http makes lighttpd consume about 1 MB per 7 MB of checked out data. When the checkout succesfully completes this memory will be freed. However, when the checkout gets cancelled lighttpd does not free the memory; it even finishes the checkout it is proxing to apache2. This means that a simple "request checkout" can cause apache2 to be streaming data to lighttpd and lighttpd buffers this without anyone ever reading it. Given a big enough repository once checked out (4 GB; lots of tags) and a "small" enough amount of memory (512 MB for lighttpd alone) you can cause OOM fairly easily. This can be replicated by having a large repository, checking it completely out and killing the subversion client that does the checkout.
The significant changes over a default Debian (Squeeze) lighttpd + apache2 + libapache2-svn installation, besides a large subversion repository) are changing the port for apache2 to e.g. 8081 and adding the following to lighttpd's configuration:
$HTTP["host"] == "localhost" {
proxy.server = ( "" => (("host" => "127.0.0.1", "port" => 8081)) )
}
With 1.5 r2529 some of the problems of 1.4.22 are aleviated; if you cancel a checkout the stream from apache2 to lighttpd is cut. However, if the client checking out is slow, the memory consumption of lighttpd will go up equally rapidly, again causing OOMs.
The significant changes over a default Debian (Squeeze) lighttpd + apache2 + libapache2-svn + manually compiled lighttpd1.5 installation, besides a large subversion repository) are changing the port for apache2 to e.g. 8081 and adding the following to lighttpd's configuration:
$HTTP["host"] == "localhost" {
proxy-core.protocol = "http"
proxy-core.backends = ( "127.0.0.1:8081" )
}
Finally something closely related to this: if one checkout (proxy) is running one cannot connect to the repository over http anymore. Once the streaming from apache2 to lighttpd has completed you can use the connection again, however with this you can do the checkout with slow client trick and increase the memory usage again. Given the 1 MB of extra server memory usage per 7 MB of checked out data this would mean that with a repository checkout of 1 GB and 7 subsequent runs you can fill 1 GB of the server's memory. With very minimal bandwidth on the client side and only a few lines in the logs that look like (harmless) checkouts. To me this looks like a denial of service!
What I would like is that:
1) the absurd amount of used memory is reclaimed when a client disconnects
2) lighttpd does not use an extreme amount of memory for this scenario, or at least make it configurable so you can limit the proxy 'cache' to e.g. 5 MB. If that limit is reached it should just stall the backend from sending data.
3) the checkout process is canceled when a client disconnects (this already works in 1.5 r2529.
4) the ability to allow multiple simultanious checkouts for a single proxy.
Updated by mezza9 about 15 years ago
Bump. This is also a significant issue for me. RAM usage starts below 1% and easily gets up to >50% on a 1.5GB RAM box after a few large SVN updates or commits.
Running 1.4.24 on Debian Etch.
Updated by stbuehler about 15 years ago
- Status changed from New to Wontfix
- Assignee deleted (
jan) - Target version deleted (
1.5.0) - Missing in 1.5.x set to No
Known issue, many reports, won't fix in 1.x.
Don't send big files via proxy, cgi, scgi, ...
Updated by jbsnyder over 14 years ago
The solution here sounds roughly equivalent to the old joke:
Patient: It hurts when I do this.
Doctor: Well, then don't do it!
Unfortunately, I'm not sure what a good workaround to this might be if one had existing, rather large, repositories running on apache but migrated the front-end server on a machine to lighttpd. Should one use redirects to the alternate-port apache server? Or is there some other way to pass connections through that avoids this problem while allowing people with previously checked-out repositories to continue accessing the svn repo from the same urls.
Suggestions?
Updated by gstrauss over 8 years ago
- Related to Bug #949: fastcgi, cgi, flush, php5 problem. added
Updated by gstrauss over 8 years ago
- Status changed from Wontfix to Patch Pending
- Target version set to 1.4.40
New: asynchronous, bidirectional streaming support for request and response
Submitted pull request: https://github.com/lighttpd/lighttpd1.4/pull/66
included in the pull request are flags to openssl (SSL_MODE_RELEASE_BUFFERS) to release memory buffers when finished with them, instead of holding onto the buffers.
Updated by gstrauss over 8 years ago
- Status changed from Patch Pending to Fixed
- % Done changed from 0 to 100
Applied in changeset 5a91fd4b9032e65ee4f6ebe1ee51e82db6b90a15.
Also available in: Atom