Project

General

Profile

Bug #2083

Excessive Memory usage with streamed files from PHP

Added by eqisow over 7 years ago. Updated 10 months ago.

Status:
Fixed
Priority:
Normal
Assignee:
-
Category:
core
Target version:
Start date:
2009-10-13
Due date:
% Done:

100%

Missing in 1.5.x:

Description

When downloading a tar archive generated on the fly with the below PHP page lighttpd's memory usage increases dramatically (it gains almost 1MB for every MB downloaded) and the memory isn't released when the download completes. I can confirm this issue in 1.4.19 in Debian Lenny as well as 1.4.23 from Squeeze. I thought it might be an issue with the code, but Apache does not seem to suffer the same problem.

The offending code:

http://codepad.org/JfvSWxcm

download.php View (14.5 KB) eqisow, 2009-10-13 06:28


Related issues

Duplicates Bug #949: fastcgi, cgi, flush, php5 problem. Fixed

Associated revisions

Revision 5a91fd4b (diff)
Added by gstrauss 11 months ago

[core] buffer large responses to tempfiles (fixes #758, fixes #760, fixes #933, fixes #1387, #1283, fixes #2083)

This replaces buffering entire response in memory which might lead to
huge memory footprint and possibly to memory exhaustion.

use tempfiles of fixed size so disk space is freed as each file sent

update callers of http_chunk_append_mem() and http_chunk_append_buffer()
to handle failures when writing to tempfile.

x-ref:
"memory fragmentation leads to high memory usage after peaks"
https://redmine.lighttpd.net/issues/758
"Random crashing on FreeBSD 6.1"
https://redmine.lighttpd.net/issues/760
"lighty should buffer responses (after it grows above certain size) on disk"
https://redmine.lighttpd.net/issues/933
"Memory usage increases when proxy+ssl+large file"
https://redmine.lighttpd.net/issues/1283
"lighttpd+fastcgi memory problem"
https://redmine.lighttpd.net/issues/1387
"Excessive Memory usage with streamed files from PHP"
https://redmine.lighttpd.net/issues/2083

History

#1 Updated by eqisow over 7 years ago

It may also be worth noting that the same issue occurs with torrentflux(-b4rt) which uses a similar method to tar directories for download.

#2 Updated by nitrox over 7 years ago

  • Priority changed from High to Normal

Try x-sendfile for this if possible, its basically one more header for your php code and an option at the fastcgi.server config within lighttpd.conf. That way php just generates the file but lighty is used directly for download. Should also speed up your php at that place.

#3 Updated by eqisow over 7 years ago

Well, that doesn't doesn't seem to have any effect. Granted, I could be doing it wrong. It seems to me that the problem is probably with the handling of the passthru function. As you can see, I'm using passthru to download straight from the tar pipe, so there is no actual file. I'm not sure x-sendfile will work like this, though (again) I could be wrong. Here's the actual download snippet I have at the moment:

function download_file($path) {
header("Content-type: application/x-tar");
header("Content-Disposition: attachment; filename=archive.tar");
header("X-Sendfile: archive.tar");
$dir = dirname($path);
$file = basename($path);
passthru("tar -cf - -C '$dir' '$file'");
exit;
}

Edit: Indeed, x-sendfile seems to not work like this.

error.log: 2009-10-13 02:39:45: (mod_fastcgi.c.2574) send-file error: couldn't get stat_cache entry for: archive.tar

If there's a different way I should be doing it, let me know.

#4 Updated by icy over 7 years ago

  • Status changed from New to Wontfix

X-Sendfile expects a path to an actual file.
In your case, I would let tar write to a temporary file and use X-LIGHTTPD-send-tempfile mytmpfile.tar to make lighty send it to the browser. Lighty will automatically delete the file afterwards.
This way, you don't pass the whole file through buffers in ram.
Lighty 2.0 will not buffer everything in ram but I don't see any change coming for 1.x.
Closing as wontfix because this is the 1.x bugtracker.

#5 Updated by eqisow over 7 years ago

  • Status changed from Wontfix to Reopened

Are you serious? Writing multiple huge archives to the disk all day is not a solution.

Even if the buffering behaviour won't be changed in 1.x, shouldn't it at least let go of the memory after the transfer completes? Before I realized what was going on lighttpd was eating up half of a 2GB page file. How is this a wontfix for a "lightweight" web server?

I would much rather work with lighttpd than apache, but with stuff like this I'm not left much choice. I hope you'll reconsider closing this.

Edit: I suppose I would write the temp file to a ramdisk, but I still think it's an ugly hack and should be resolved in lighttpd before 2.0, whenever that may be.

#6 Updated by stbuehler over 7 years ago

  • Status changed from Reopened to Wontfix
  • Target version deleted (1.4.24)

If we say "wontfix" why do you think you should reopen it? Isn't it clear that we understand what the problem is?
And btw: there is no mem leak.

#7 Updated by eqisow over 7 years ago

Ohhh, I see! So >1GB reserved for a low traffic server is what you're going for? Genius. Really.

If you don't want me to be able to reopen it perhaps the default permissions should be different, eh?

#8 Updated by icy over 7 years ago

Maybe you should stay a bit more friendly. Last time I checked, you didn't pay for lighty.
If it does not fit your usecase, use something else. But attacking us with your sarcasm does not help in any way.

#9 Updated by simmel over 7 years ago

eqisow wrote:

Are you serious? Writing multiple huge archives to the disk all day is not a solution.

Even if the buffering behaviour won't be changed in 1.x, shouldn't it at least let go of the memory after the transfer completes? Before I realized what was going on lighttpd was eating up half of a 2GB page file. How is this a wontfix for a "lightweight" web server?

I would much rather work with lighttpd than apache, but with stuff like this I'm not left much choice. I hope you'll reconsider closing this.

Edit: I suppose I would write the temp file to a ramdisk, but I still think it's an ugly hack and should be resolved in lighttpd before 2.0, whenever that may be.

IMO (and alot of other peoples opinion) your solution is wrong. But I still think you deserve an explaination.

This conversation is an excerpt taken from IRC:


1220.04 simmel, icy: Do you know what the reason is why it isn't returing the memory?
1222.06 spaam, memorypools
1224.01 simmel, spaam: Mkay? Care to elaborate?
1224.08  * simmel is a simple scripter
1225.04 simmel, C is too hard for me! (pun intended)
1225.20 spaam, http://en.wikipedia.org/wiki/Memory_pool like IIRC icy did
1225.22 spaam, do
1225.35 spaam, when i asked
1226.42 simmel, = )
1226.45 simmel, Thanks spaam 
1229.33 simmel, So, the memory is allocated but not in use?
1230.11 icy, it is allocated and then reused later
1231.00 icy, in 2.0, we have a mechanism to limit the buffer usage (and return unused buffers)

So, it's going to be solved in 2.0 and it sort of isn't leaked memory (since it will be reused by lighttpd).

I hope this doesnt ignite the whole discussion again, but I still thought that you might have wanted an explaination eqisow.

#10 Updated by eqisow over 7 years ago

Thank you simmel. I still think it's more major than you guys do. Lighty may re-use it, but the rest of the system can't and huge amounts of memory still end up paged. It's kind of frustrating because I really prefer lighty and would rather use it. Anyway, you guys understand lighty's structure better than myself so there's no need to argue it further. The reason I reopened it is because I've found that with many projects one person will shut something down without it really getting discussed or ever seeing the light of day again, where if somebody else takes a look at it they may have a different opinion. If that's not proper I won't do it again.

I apologize for the sarcasm as well. I felt like stbuehler was talking down to me and that's just one of those things that gets under my skin. I look forward to being able to use 2.0.

#11 Updated by darix over 7 years ago

you can already use 1.5. with the x-lighttpd-send-temp-file, with very minimal changes to your script.

#12 Updated by darix over 7 years ago

  • Subject changed from Memory leak from PHP download to Excessive Memory usage with streamed files from PHP

#13 Updated by eqisow over 7 years ago

Does that only work in 1.5? Icy mentioned it and I just assumed he meant 1.4. Running the development branch is a bit scary...

Edit: Much better title.

#14 Updated by darix over 7 years ago

we run 1.5 in production since years now. on lighttpd.net and also on my work sites. it is production ready but we just dont feel like we want to release it as is. and the development work went more into 2.0 than in 1.5

#15 Updated by stbuehler over 7 years ago

eqisow:

I still think you got that wrong. We know it is a major issue, which is why we fixed it in our new design (btw: when i discovered that flaw, i was a little bit upset too, see #949 :). 1.4.x doesn't have it, and we will not change it (although that would be possible), as it is the stable branch and it would require too many core changes. (That is imho the point of a development branch: that you are allowed to break things, so you can fix the design in the long run)

And we of course want our user here to be allowed to reopen bugs; but we don't want such wontfix<->reopen fights - as long as you cannot provide new evidence, you should respect the decision of the developer team.

lighttpd.net is running 2.0 btw now :)

#16 Updated by eqisow over 7 years ago

That's an interesting read stbuehler. Since it appears to be the same issue I'm curious why this was labelled wontfix when the other shows a target of 1.4.x, even if it was filed two years ago. Then again "duplicate" doesn't seem to be an option on this bug tracker. shrug

The development branch is where you break things, of course, but imo major issues like this need to be fixed in the stable branch as well. Sometimes the bug you know is better than the one you don't, but in this case I think what you're getting by not fixing it is "stable but broken." How does that help anybody? So maybe 1.4.24 stays in testing for longer, so what? That seems acceptable to me.

By the way, the comments on #949 seem to suggest that this issue is fixed in 1.5. Is this correct? At the risk of getting off-topic, what's the hold-up on 1.5? I can see that it's been in development for at least two years based just on the comments in that bug report. Even one of the comments on this bug calls it production ready, yet calling it prerelease keeps it from being packaged in Fedora, Debian, etc.

I thought about tackling 2.0, but the documentation is rather sparse (which is understandable for where it's at, of course) and the layout/syntax of the config file seems to have changed a bit. These two things together would make for a long day.

(wasn't going for a fight, btw, and certainly wouldn't have opened it again anyway)

#17 Updated by stbuehler over 7 years ago

I think i left it open so people find the issue (most people don't look at closed ones), and there is a Duplicate status and i just added a reference :)

And no, the issue is not fixed in 1.5, it just may handle the buffer reusing a little bit better and has more features to "workaround" the problem.

We don't have the man power to support an 1.5.x stable branch (and we can't just drop 1.4 if we make 1.5 stable).

#18 Updated by gstrauss 11 months ago

  • Category set to core
  • Status changed from Wontfix to Patch Pending
  • Target version set to 1.4.40

New: asynchronous, bidirectional streaming support for request and response
Submitted pull request: https://github.com/lighttpd/lighttpd1.4/pull/66

included in the pull request is buffering large responses to temporary files instead of keeping it all in memory

#19 Updated by gstrauss 10 months ago

  • Status changed from Patch Pending to Fixed
  • % Done changed from 0 to 100

Also available in: Atom