Lighttpd

It may also be worth noting that the same issue occurs with torrentflux(-b4rt) which uses a similar method to tar directories for download.

Well, that doesn't doesn't seem to have any effect. Granted, I could be doing it wrong. It seems to me that the problem is probably with the handling of the passthru function. As you can see, I'm using passthru to download straight from the tar pipe, so there is no actual file. I'm not sure x-sendfile will work like this, though (again) I could be wrong. Here's the actual download snippet I have at the moment:

function download_file($path) {
header("Content-type: application/x-tar");
header("Content-Disposition: attachment; filename=archive.tar");
header("X-Sendfile: archive.tar");
$dir = dirname($path);
$file = basename($path);
passthru("tar -cf - -C '$dir' '$file'");
exit;
}

Edit: Indeed, x-sendfile seems to not work like this.

error.log: 2009-10-13 02:39:45: (mod_fastcgi.c.2574) send-file error: couldn't get stat_cache entry for: archive.tar

If there's a different way I should be doing it, let me know.

Ohhh, I see! So >1GB reserved for a low traffic server is what you're going for? Genius. Really.

If you don't want me to be able to reopen it perhaps the default permissions should be different, eh?

Maybe you should stay a bit more friendly. Last time I checked, you didn't pay for lighty.
If it does not fit your usecase, use something else. But attacking us with your sarcasm does not help in any way.

eqisow wrote:

Are you serious? Writing multiple huge archives to the disk all day is not a solution.

Even if the buffering behaviour won't be changed in 1.x, shouldn't it at least let go of the memory after the transfer completes? Before I realized what was going on lighttpd was eating up half of a 2GB page file. How is this a wontfix for a "lightweight" web server?

I would much rather work with lighttpd than apache, but with stuff like this I'm not left much choice. I hope you'll reconsider closing this.

Edit: I suppose I would write the temp file to a ramdisk, but I still think it's an ugly hack and should be resolved in lighttpd before 2.0, whenever that may be.

IMO (and alot of other peoples opinion) your solution is wrong. But I still think you deserve an explaination.

This conversation is an excerpt taken from IRC:

1220.04 simmel, icy: Do you know what the reason is why it isn't returing the memory?
1222.06 spaam, memorypools
1224.01 simmel, spaam: Mkay? Care to elaborate?
1224.08  * simmel is a simple scripter
1225.04 simmel, C is too hard for me! (pun intended)
1225.20 spaam, http://en.wikipedia.org/wiki/Memory_pool like IIRC icy did
1225.22 spaam, do
1225.35 spaam, when i asked
1226.42 simmel, = )
1226.45 simmel, Thanks spaam 
1229.33 simmel, So, the memory is allocated but not in use?
1230.11 icy, it is allocated and then reused later
1231.00 icy, in 2.0, we have a mechanism to limit the buffer usage (and return unused buffers)

So, it's going to be solved in 2.0 and it sort of isn't leaked memory (since it will be reused by lighttpd).

I hope this doesnt ignite the whole discussion again, but I still thought that you might have wanted an explaination eqisow.

Thank you simmel. I still think it's more major than you guys do. Lighty may re-use it, but the rest of the system can't and huge amounts of memory still end up paged. It's kind of frustrating because I really prefer lighty and would rather use it. Anyway, you guys understand lighty's structure better than myself so there's no need to argue it further. The reason I reopened it is because I've found that with many projects one person will shut something down without it really getting discussed or ever seeing the light of day again, where if somebody else takes a look at it they may have a different opinion. If that's not proper I won't do it again.

I apologize for the sarcasm as well. I felt like stbuehler was talking down to me and that's just one of those things that gets under my skin. I look forward to being able to use 2.0.

you can already use 1.5. with the x-lighttpd-send-temp-file, with very minimal changes to your script.

Does that only work in 1.5? Icy mentioned it and I just assumed he meant 1.4. Running the development branch is a bit scary...

Edit: Much better title.

we run 1.5 in production since years now. on lighttpd.net and also on my work sites. it is production ready but we just dont feel like we want to release it as is. and the development work went more into 2.0 than in 1.5

eqisow:

I still think you got that wrong. We know it is a major issue, which is why we fixed it in our new design (btw: when i discovered that flaw, i was a little bit upset too, see #949 :). 1.4.x doesn't have it, and we will not change it (although that would be possible), as it is the stable branch and it would require too many core changes. (That is imho the point of a development branch: that you are allowed to break things, so you can fix the design in the long run)

And we of course want our user here to be allowed to reopen bugs; but we don't want such wontfix<->reopen fights - as long as you cannot provide new evidence, you should respect the decision of the developer team.

lighttpd.net is running 2.0 btw now :)

That's an interesting read stbuehler. Since it appears to be the same issue I'm curious why this was labelled wontfix when the other shows a target of 1.4.x, even if it was filed two years ago. Then again "duplicate" doesn't seem to be an option on this bug tracker. shrug

The development branch is where you break things, of course, but imo major issues like this need to be fixed in the stable branch as well. Sometimes the bug you know is better than the one you don't, but in this case I think what you're getting by not fixing it is "stable but broken." How does that help anybody? So maybe 1.4.24 stays in testing for longer, so what? That seems acceptable to me.

By the way, the comments on #949 seem to suggest that this issue is fixed in 1.5. Is this correct? At the risk of getting off-topic, what's the hold-up on 1.5? I can see that it's been in development for at least two years based just on the comments in that bug report. Even one of the comments on this bug calls it production ready, yet calling it prerelease keeps it from being packaged in Fedora, Debian, etc.

I thought about tackling 2.0, but the documentation is rather sparse (which is understandable for where it's at, of course) and the layout/syntax of the config file seems to have changed a bit. These two things together would make for a long day.

(wasn't going for a fight, btw, and certainly wouldn't have opened it again anyway)

I think i left it open so people find the issue (most people don't look at closed ones), and there is a Duplicate status and i just added a reference :)

And no, the issue is not fixed in 1.5, it just may handle the buffer reusing a little bit better and has more features to "workaround" the problem.

We don't have the man power to support an 1.5.x stable branch (and we can't just drop 1.4 if we make 1.5 stable).