Lighttpd

I don't think it has something todo with the size of one temp file. And you certainly do not want to buffer the post-body in memory.

"I don't think it has something todo with the size of one temp file."

I think it does. When I post a 750 meg file /var/tmp fills with 750 1 meg files which have to be re-assembled. The CPU utilization climbs when the PHP is being called. On the same computer running Apache using the same php config there is no delay or increase in CPU utilization. The file is simply uploaded and without delay the next file begins uploading.

Thanks.

Anyway, is there a way I can adjust how Litghttpd stores post requests?

Just wondering, what is the advantage of splitting the file into 1 mb chunks anyway?

spinnerdog wrote:

Is there a way to increase the size of the post/upload buffer size?

you have to set a sane limit on POST-DATA, so 1MB is ok.

I use a Java applet to post/upload large files to lighttpd-1.4.18. lighttpd spools the incoming files into 1 meg tmp files in /var/tmp.

try /tmp or any other tmpfs if you have enough memory.
/var/tmp/ is disk and thus might be slow.

Olaf-van-der-Spek wrote:

Just wondering, what is the advantage of splitting the file into 1 mb chunks anyway?

nitrox wrote:

you have to set a sane limit on POST-DATA, so 1MB is ok.

I'm not so sure. Apache spools the data directly into the cgi script. I can tell because with lighttpd the tmp files are created and then lighttpd calls the php script and php then creates its tmp files based on settings in php.ini. Apache+php produces the php tmp files as the data is spooling in.

nitrox wrote:
try /tmp or any other tmpfs if you have enough memory.

Memory is the problem. My device only has 128 meg of ram.

Well, lightty itself isn't actually reassmbling the file; it's reading the individual temp files and spitting them over to the CGI. However, many CGI/FCGI programs are going to take that and write it directly to disk for the application to use (after splitting MIME data apart), which has the same effect.

I'm not sure if this is why files are being split into small chunks, but a theory: with small chunks, lightty can delete individual chunks as they're being sent to the CGI program. It writes the first 1MB, then deletes it before sending the next one. This way, if the application is writing the data back to disk (as most will), it doesn't double memory usage if they're both going to a tmpfs. (I havn't confirmed that this is actually what happens.)

The basic problem vs. Apache is that lightty won't stream uploaded files directly to the application. That's the most efficient thing to do. It's not always the best: some unfortunate systems aren't threadsafe, so FCGI backends may be expensive, and Apache's way locks up the whole backend while the upload happens. But not all systems are so broken, and Lightty really should be able to stream uploads directly if FCGI server threads are cheap. This way, the backend can stream data in however it wants, and Lightty doesn't impose any costs or request latency.

For example, if a site is receiving large pictures and generating thumbnails, a cleverly-written FCGI server could start to process the thumbnail immediately, as the picture streams in, instead of waiting for the whole file to be uploaded and then doing all the work. Lightty makes this impossible.