[Solved] mod_webdav - PUT files with < 64kb Content-Length reults in zero length file
Added by levi over 3 years ago
Lighttpd version: 1.4.55
Server Config: https://paste.lighttpd.net/D7#OUNrY4YQcAOOynGKNatTYyli
Server Environment: docker: alpine linux 3.12
Client: Cadaver 0.23.3
a PUT request for a file < 64kb in length results in a zero length file on the server, but PUTing files > 64kb works as expected.
Debug logs look like this:
2020-08-01 21:56:05: (connections.c.774) fd: 9 request-len: 155 \nPUT /data/Selection_004.png HTTP/1.1\r\nUser-Agent: cadaver/0.23.3 neon/0.30.2\r\nConnection: TE\r\nTE: trailers\r\nHost: localhost:3003\r\nConte nt-Length: 25162\r\n\r\n 2020-08-01 21:56:05: (response.c.447) -- splitting Request-URI 2020-08-01 21:56:05: (response.c.448) Request-URI : /data/Selection_004.png 2020-08-01 21:56:05: (response.c.449) URI-scheme : http 2020-08-01 21:56:05: (response.c.450) URI-authority : localhost:3003 2020-08-01 21:56:05: (response.c.451) URI-path (raw) : /data/Selection_004.png 2020-08-01 21:56:05: (response.c.452) URI-path (clean): /data/Selection_004.png 2020-08-01 21:56:05: (response.c.453) URI-query : 2020-08-01 21:56:05: (mod_access.c.177) -- mod_access_uri_handler called 2020-08-01 21:56:05: (response.c.598) -- before doc_root 2020-08-01 21:56:05: (response.c.599) Doc-Root : /srv/todo-all-the-things 2020-08-01 21:56:05: (response.c.600) Rel-Path : /data/Selection_004.png 2020-08-01 21:56:05: (response.c.601) Path : 2020-08-01 21:56:05: (response.c.643) -- after doc_root 2020-08-01 21:56:05: (response.c.644) Doc-Root : /srv/todo-all-the-things 2020-08-01 21:56:05: (response.c.645) Rel-Path : /data/Selection_004.png 2020-08-01 21:56:05: (response.c.646) Path : /srv/todo-all-the-things/data/Selection_004.png 2020-08-01 21:56:05: (response.c.125) Response-Header: \nHTTP/1.1 204 No Content\r\nETag: "2269318771"\r\nAccess-Control-Allow-Origin: *\r\nAccess-Control-Allow-Methods: *\r\nAccess-Control-Expose-Headers: *\r\n Access-Control-Allow-Headers: *\r\nAccess-Control-Max-Age: 600\r\nTiming-Allow-Origin: *\r\nDate: Sat, 01 Aug 2020 21:56:05 GMT\r\nServer: lighttpd/1.4.55\r\n\r\n 2020-08-01 21:56:11: (connections.c.1402) connection closed - keep-alive timeout: 9 2020-08-01 22:00:30: (connections.c.774) fd: 9 request-len: 156 \nPUT /data/Selection_005.png HTTP/1.1\r\nUser-Agent: cadaver/0.23.3 neon/0.30.2\r\nConnection: TE\r\nTE: trailers\r\nHost: localhost:3003\r\nConte nt-Length: 156974\r\n\r\n 2020-08-01 22:00:30: (response.c.447) -- splitting Request-URI 2020-08-01 22:00:30: (response.c.448) Request-URI : /data/Selection_005.png 2020-08-01 22:00:30: (response.c.449) URI-scheme : http 2020-08-01 22:00:30: (response.c.450) URI-authority : localhost:3003 2020-08-01 22:00:30: (response.c.451) URI-path (raw) : /data/Selection_005.png 2020-08-01 22:00:30: (response.c.452) URI-path (clean): /data/Selection_005.png 2020-08-01 22:00:30: (response.c.453) URI-query : 2020-08-01 22:00:30: (mod_access.c.177) -- mod_access_uri_handler called 2020-08-01 22:00:30: (response.c.598) -- before doc_root 2020-08-01 22:00:30: (response.c.599) Doc-Root : /srv/todo-all-the-things 2020-08-01 22:00:30: (response.c.600) Rel-Path : /data/Selection_005.png 2020-08-01 22:00:30: (response.c.601) Path : 2020-08-01 22:00:30: (response.c.643) -- after doc_root 2020-08-01 22:00:30: (response.c.644) Doc-Root : /srv/todo-all-the-things 2020-08-01 22:00:30: (response.c.645) Rel-Path : /data/Selection_005.png 2020-08-01 22:00:30: (response.c.646) Path : /srv/todo-all-the-things/data/Selection_005.png 2020-08-01 22:00:30: (response.c.125) Response-Header: \nHTTP/1.1 201 Created\r\nETag: "2762402138"\r\nAccess-Control-Allow-Origin: *\r\nAccess-Control-Allow-Methods: *\r\nAccess-Control-Expose-Headers: *\r\nAcc ess-Control-Allow-Headers: *\r\nAccess-Control-Max-Age: 600\r\nTiming-Allow-Origin: *\r\nContent-Length: 0\r\nDate: Sat, 01 Aug 2020 22:00:30 GMT\r\nServer: lighttpd/1.4.55\r\n\r\n 2020-08-01 22:00:36: (connections.c.1402) connection closed - keep-alive timeout: 9
Which results in files like this:
/var/log/lighttpd # ls -l /srv/todo-all-the-things/data/ total 156 -rw-r--r-- 1 lighttpd lighttpd 0 Aug 1 21:56 Selection_004.png -rw-r--r-- 1 lighttpd lighttpd 156974 Aug 1 22:00 Selection_005.png
This is similar to: https://redmine.lighttpd.net/boards/2/topics/9060 but I'm using 1.4.55, and I'm not using a modified version of mod_webdav. The issue linked in that post (#2958) doesn't seem to be occurring here, or at least there's nothing about a SEGFAULT in my logs.
My configuration is very simple, and if this were a lighttpd bug I'm sure someone else would have run into it before me, so does that mean it's likely a dependency issue? I'm using lighttpd installed by this package: https://pkgs.alpinelinux.org/package/edge/main/x86_64/lighttpd
There's a build log here showing dependencies: https://build.alpinelinux.org/buildlogs/build-edge-x86_64/main/lighttpd/lighttpd-1.4.55-r1.log
No really obvious problem in the build log but I don't know what I'm looking for.
Replies (9)
RE: mod_webdav - PUT files with < 64kb Content-Length reults in zero length file - Added by gstrauss over 3 years ago
If you have strace inside the docker image, it would be useful to strace lighttpd to see what happens to the temporary file created by the PUT.
RE: mod_webdav - PUT files with < 64kb Content-Length reults in zero length file - Added by gstrauss over 3 years ago
What libc is used on Alpine Linux? (I think musl) I have not tested mod_webdav with musl and so there may be some differences from glibc not accounted for in mod_webdav.
RE: mod_webdav - PUT files with < 64kb Content-Length reults in zero length file - Added by gstrauss over 3 years ago
There is an optimized code path in mod_webdav for uploaded files < 64KB on systems #if (defined(__linux__) || defined(__CYGWIN__)) && defined(O_TMPFILE), and so that is where I started looking.
I tested a minimal build of https://download.lighttpd.net/lighttpd/releases-1.4.x/lighttpd-1.4.55.tar.xz using the musl-gcc wrapper (on Fedora 32 x86_64), and things worked for me for files < 64KB and > 64KB.
An strace on your system might help identify the issue.
(As an aside, I also tested the lighttpd master branch and found a bug that I had recently introduced (only on master) which coincidentally exhibited the behavior your reported. That bug has been fixed on master.)
RE: mod_webdav - PUT files with < 64kb Content-Length reults in zero length file - Added by levi over 3 years ago
I tried a new docker config using ubuntu base and the package from their repo (also using 1.4.55), and see the same behavior.
An strace from this ubuntu base is here: https://paste.lighttpd.net/H7#kEPoKj6PCXrndUX9s8yllhyp
I'll put together a minimal docker config so you can see if you can reproduce.
RE: mod_webdav - PUT files with < 64kb Content-Length reults in zero length file - Added by gstrauss over 3 years ago
Thanks for the strace.
openat(AT_FDCWD, 0x564cf7c21aa0, O_RDWR|O_NOCTTY|O_APPEND|O_NONBLOCK|O_CLOEXEC|O_TMPFILE, 0666) = -1 EOPNOTSUPP (Operation not supported) getpid() = 9 openat(AT_FDCWD, 0x564cf7c21aa0, O_RDWR|O_CREAT|O_EXCL, 0600) = 10 fcntl(10, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE) fcntl(10, F_SETFL, O_RDWR|O_APPEND|O_LARGEFILE) = 0 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 unlink(0x564cf7c21aa0) = 0 getpid() = 9 write(10, 0x564cf7c191bb, 25162) = 25162 linkat(AT_FDCWD, 0x7ffe70cd6dc0, AT_FDCWD, 0x564cf7b18250, AT_SYMLINK_FOLLOW) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, 0x564cf7b18250, O_WRONLY|O_CREAT|O_EXCL|O_NOCTTY|O_TRUNC|O_NONBLOCK|O_NOFOLLOW|O_CLOEXEC, 0666) = 11 close(10) = 0 fstat(11, 0x7ffe70cd6ca0) = 0 close(11) = 0
It appears the Linux O_TMPFILE is not supported in this environment -- possibly not supported by musl. Do you know what version of musl is in your environment?
Anyway, that's probably not the issue since lighttpd falls back to creating its own temporary file. The issue appears to be linkat() failing. Your strace does not display the string values, but linkat() is failing with ENOENT, probably on "/proc/self/fd/10". Is /proc mounted and available in the docker image?
If linkat() fails, as it is in your case, maybe lighttpd can do better than leaving an empty file. I'll think about alternative approaches. ...Maybe I should check for existence of /proc/self/fd/ at lighttpd startup and disable the linkat() optimization if /proc/self/fd/ is not present.
RE: mod_webdav - PUT files with < 64kb Content-Length reults in zero length file - Added by gstrauss over 3 years ago
This fixes the fallback case where linkat() fails. (I tested a simple use case.)
--- a/src/mod_webdav.c
+++ b/src/mod_webdav.c
@@ -4222,10 +4222,12 @@ mod_webdav_write_single_file_chunk (request_st * const r, chunkqueue * const cq)
/*assert(cq->first->next != NULL);*/
chunk * const c = cq->first;
cq->first = c->next;
+ const off_t len = chunkqueue_length(cq);
if (mod_webdav_write_cq(r, cq, c->file.fd)) {
/*assert(cq->first == NULL);*/
c->next = NULL;
cq->first = cq->last = c;
+ c->file.length = len;
return 1;
}
else {
RE: mod_webdav - PUT files with < 64kb Content-Length reults in zero length file - Added by gstrauss over 3 years ago
BTW, what is the filesystem you are using? I am curious if the limitation on Linux O_TMPFILE is due to the underlying filesystem.
RE: mod_webdav - PUT files with < 64kb Content-Length reults in zero length file - Added by levi over 3 years ago
Success!
I managed to get lighttpd to build from source, which was a bit of a journey, and of course your patch resolves this issue.
regarding filesystem, "df -khT" in the container lists the filesystem type as "overlay", which I assume is a docker thing? The underlying host filesystem is ext4.
regarding access to /proc, I'm out of my depth here. Not sure if this helps but I haven't done anything in my Dockerfile that would effect the default /proc structure.
Anyhow, thanks very much for your help.
RE: [Solved] mod_webdav - PUT files with < 64kb Content-Length reults in zero length file - Added by gstrauss over 3 years ago
Thanks for the update. The patch will be part of the next release of lighttpd. (lighttpd 1.4.56)