Project

General

Profile

Forums » Development »

[Solved] Using lighttpd + web_dav and large file uploads

Added by DaGoodBoy over 15 years ago

I'm running Debian stable + lighty (lighttpd-1.4.x branch @ r2357) with lighttpd-mod-webdav. The file system is ext3 mounted with noatime. It is running on a fairly lightweight system with 1GHz cpu and average IDE disks. This is a semi-embedded system, so my hardware choices are limited.

After I upload a 1.2GB file to the system (via PUT request over ssl) lighttpd becomes unresponsive for up to 2 minutes while the file is reassembled and the temporary files are deleted. The IO wait runs up past 80% and connections just time out on the client. When I take a peek at lighttpd using strace, I see this going on and on:

open("/var/tmp/lighttpd-upload-35qU8L", O_RDONLY|O_LARGEFILE) = 17
mmap2(NULL, 1049600, PROT_READ, MAP_SHARED, 17, 0) = 0xb7891000
close(17)                               = 0
write(11, "=\206\232\252\22\20QH\313(j\305W\351\362\367\16\37\16a"..., 1049600) = 1049600
unlink("/var/tmp/lighttpd-upload-35qU8L") = 0
munmap(0xb7891000, 1049600)             = 0

Each time the kernel flushes the file system unlink() to disk, there is a visible pause in the strace output. It seems wasteful somehow to tie up lighttpd waiting for disk activity like this. I imagine waiting on disk I/O probably makes things much simpler in the code, but would it be possible to make the file system activity asynchronous and let lighttpd get back to serving requests while the file is shuffled around on disk?

Is there some obvious config setting I've missed that helps with this issue? Should I open a bug on this? If I wanted to take a crack at writing a patch for this, do you guys have any suggestions on an approach or common potholes to avoid? Thanks in advance.


Replies (4)

RE: Using lighttpd + web_dav and large file uploads - Added by darix over 15 years ago

slow disks will always kill lighttpd's performance.

RE: Using lighttpd + web_dav and large file uploads - Added by icy over 15 years ago

There is no way to prevent that in 1.4, there are no threads that could do the work in the background.
1.5 has gthread-aio but I am not sure if that helps with uploaded files, probably not. Would have to check the source.
In 2.0, we will probably have a thread to do that stuff or think of some other way to do it without having the server block in such cases.
Thanks for mentioning.

RE: [Solved] Using lighttpd + web_dav and large file uploads - Added by gstrauss over 3 years ago

In lighttpd 1.4.54, lighttpd mod_webdav was rewritten. It now uploads the file into the target directory as a temporary file, and renames it into place, so the unnecessary copying no longer occurs.

    (1-4/4)