Forums » Development »
[Solved] Different location for chunked "lighttpd-upload" files and final post upload file
Added by jens-maus over 3 years ago
For one of my application using lighttpd I am supporting to upload files via POST requests.
There, I am using server.stream-request-body = 1
as well as server.upload-dirs = ( "/usr/local/tmp" )
to let lighttpd stream to temp files on disk (because of limited memory constraints). However, looking at the upload process, I do see the following files being created in /usr/local/tmp:
-rw------- 1 root root 17025248 Jul 13 12:11 lighttpd-upload-dDWIDa
-rw------- 1 root root 16778792 Jul 13 12:11 lighttpd-upload-orl9Tc
-rw------- 1 root root 1294336 Jul 13 12:11 lighttpd-upload-pkMkgb
-rw------- 1 root root 45633536 Jul 13 12:11 tmp.orinEI
Here tmp.orinEI
is the final uploaded file and lighttpd-upload-XXXXX
being the temporary files lighttpd generates. While the system of course has some limited memory constraints, there is potential that it still will fit the intermediate lighttpd-upload-XXXX
files while they are generated and then removed as soon as they are added to the tmp.XXXX
file. Thus, I do wonder if there is an option already to allow lighttpd to put the lighttpd-upload-XXXX
file to /tmp
(a tmpfs mount in memory) but combine these temp files still in another (e.g. on-disk directory) directory like /usr/local/tmp
which will then have enough space for large file uploads.
Any help would highly appreciated.
Replies (6)
RE: Different location for chunked "lighttpd-upload" files and final post upload file - Added by gstrauss over 3 years ago
The lighttpd modules from the base installation do not directly produce a temporary file with the template name tmp.XXXXXX
. Maybe that is produced by your backend scripts which read the request body from lighttpd?
lighttpd server.upload-dirs
supports a list of directories, so you can use something like server.upload-dirs = ( "/dev/shm", "/var/tmp" )
(See #2588 for more details)
RE: Different location for chunked "lighttpd-upload" files and final post upload file - Added by gstrauss over 3 years ago
(For reference, the commit for #2588 is commit 77bd4512 and part of lighttpd 1.4.40)
BTW, the reason lighttpd uses temporary files of a limited size (default 1 MB (fuzzy) for server.upload-temp-file-size
) is so that the temporary file can be removed (and the space freed) as soon as that one file is read by the backend, or sent to the client. This is intentional to avoid storing the entire result (in either direction) and taking up a growing amount of disk space for the entire length of the request and response.
If you need to minimize disk space used, then you can use server.stream-request-body = 2
and server.stream-response-body = 2
, though that might cause your backend process to remain running (using memory and other resources) much longer, rather than quickly offloading the response to lighttpd for lighttpd to send the response to the client.
RE: Different location for chunked "lighttpd-upload" files and final post upload file - Added by gstrauss over 3 years ago
@jens-maus do were you able to come up with a solution? Do you have further related questions?
RE: Different location for chunked "lighttpd-upload" files and final post upload file - Added by gstrauss over 3 years ago
@jens-maus do were you able to come up with a solution? Do you have further related questions?
I think the answer to your question is that lighttpd does not combine the temporary files, and that tmp.orinEI
was likely created by your backend script.
server.upload-dirs = ( "/dev/shm", "/usr/local/tmp" )
will fall back to "/usr/local/tmp" if "/dev/shm" fills up. This is a good solution if "/dev/shm" is in-memory and you are trying to reduce writes of temporary files to embedded flash "/usr/local/tmp". However, your backend script should probably accumulate the input in "/usr/local/tmp" where you know that you have enough space. If filling up "/tmp" or "/dev/shm" is something that must be avoided, then you can set server.upload-dirs = ( "/usr/local/tmp" )
so that lighttpd creates temp files in "/usr/local/tmp". lighttpd uses temporary files for the request body when the request body is > 64KB (or when request body is > 128KB when server.stream-request-body = 2
)
server.upload-temp-file-size
controls the size of the temp files lighttpd creates (and is a fuzzy limit that lighttpd may slightly exceed while draining kernel socket buffers, before moving on to the next temp file). It looks like you have already changed that to 16MB, up from the default 1MB.
RE: Different location for chunked "lighttpd-upload" files and final post upload file - Added by jens-maus over 3 years ago
Yes, thanks to your comments I was able to solve my issue by using server.upload-dirs = ( "/dev/shm", "/usr/local/tmp" )
and server.stream-XXX-body=2
. So thanks for the help. Now my application seems to utilize lighttpd way better in that regard!
RE: Different location for chunked "lighttpd-upload" files and final post upload file - Added by gstrauss over 3 years ago
FYI: I created Resource Tuning wiki doc to expand on some of the conversation here. I hope it is useful to readers.