Project

General

Profile

[Solved] Different location for chunked "lighttpd-upload" files and final post upload file

Added by jens-maus almost 3 years ago

For one of my application using lighttpd I am supporting to upload files via POST requests.

There, I am using server.stream-request-body = 1 as well as server.upload-dirs = ( "/usr/local/tmp" ) to let lighttpd stream to temp files on disk (because of limited memory constraints). However, looking at the upload process, I do see the following files being created in /usr/local/tmp:

-rw------- 1 root root 17025248 Jul 13 12:11 lighttpd-upload-dDWIDa
-rw------- 1 root root 16778792 Jul 13 12:11 lighttpd-upload-orl9Tc
-rw------- 1 root root 1294336 Jul 13 12:11 lighttpd-upload-pkMkgb
-rw------- 1 root root 45633536 Jul 13 12:11 tmp.orinEI

Here tmp.orinEI is the final uploaded file and lighttpd-upload-XXXXX being the temporary files lighttpd generates. While the system of course has some limited memory constraints, there is potential that it still will fit the intermediate lighttpd-upload-XXXX files while they are generated and then removed as soon as they are added to the tmp.XXXX file. Thus, I do wonder if there is an option already to allow lighttpd to put the lighttpd-upload-XXXX file to /tmp (a tmpfs mount in memory) but combine these temp files still in another (e.g. on-disk directory) directory like /usr/local/tmp which will then have enough space for large file uploads.

Any help would highly appreciated.


Replies (6)

RE: Different location for chunked "lighttpd-upload" files and final post upload file - Added by gstrauss almost 3 years ago

The lighttpd modules from the base installation do not directly produce a temporary file with the template name tmp.XXXXXX. Maybe that is produced by your backend scripts which read the request body from lighttpd?

lighttpd server.upload-dirs supports a list of directories, so you can use something like server.upload-dirs = ( "/dev/shm", "/var/tmp" ) (See #2588 for more details)

RE: Different location for chunked "lighttpd-upload" files and final post upload file - Added by gstrauss almost 3 years ago

(For reference, the commit for #2588 is commit 77bd4512 and part of lighttpd 1.4.40)

BTW, the reason lighttpd uses temporary files of a limited size (default 1 MB (fuzzy) for server.upload-temp-file-size) is so that the temporary file can be removed (and the space freed) as soon as that one file is read by the backend, or sent to the client. This is intentional to avoid storing the entire result (in either direction) and taking up a growing amount of disk space for the entire length of the request and response.

If you need to minimize disk space used, then you can use server.stream-request-body = 2 and server.stream-response-body = 2, though that might cause your backend process to remain running (using memory and other resources) much longer, rather than quickly offloading the response to lighttpd for lighttpd to send the response to the client.

RE: Different location for chunked "lighttpd-upload" files and final post upload file - Added by gstrauss over 2 years ago

@jens-maus do were you able to come up with a solution? Do you have further related questions?

RE: Different location for chunked "lighttpd-upload" files and final post upload file - Added by gstrauss over 2 years ago

@jens-maus do were you able to come up with a solution? Do you have further related questions?

I think the answer to your question is that lighttpd does not combine the temporary files, and that tmp.orinEI was likely created by your backend script.

server.upload-dirs = ( "/dev/shm", "/usr/local/tmp" ) will fall back to "/usr/local/tmp" if "/dev/shm" fills up. This is a good solution if "/dev/shm" is in-memory and you are trying to reduce writes of temporary files to embedded flash "/usr/local/tmp". However, your backend script should probably accumulate the input in "/usr/local/tmp" where you know that you have enough space. If filling up "/tmp" or "/dev/shm" is something that must be avoided, then you can set server.upload-dirs = ( "/usr/local/tmp" ) so that lighttpd creates temp files in "/usr/local/tmp". lighttpd uses temporary files for the request body when the request body is > 64KB (or when request body is > 128KB when server.stream-request-body = 2)

server.upload-temp-file-size controls the size of the temp files lighttpd creates (and is a fuzzy limit that lighttpd may slightly exceed while draining kernel socket buffers, before moving on to the next temp file). It looks like you have already changed that to 16MB, up from the default 1MB.

RE: Different location for chunked "lighttpd-upload" files and final post upload file - Added by jens-maus over 2 years ago

Yes, thanks to your comments I was able to solve my issue by using server.upload-dirs = ( "/dev/shm", "/usr/local/tmp" ) and server.stream-XXX-body=2. So thanks for the help. Now my application seems to utilize lighttpd way better in that regard!

RE: Different location for chunked "lighttpd-upload" files and final post upload file - Added by gstrauss over 2 years ago

FYI: I created Resource Tuning wiki doc to expand on some of the conversation here. I hope it is useful to readers.

    (1-6/6)