[Solved] Webdav + rclone backup
Added by meeb5 over 3 years ago
Hi all,
I'm encountering a problem when I'm trying to backup files to a webdav share via rclone:
Starting from an empty folder:
touch a
rclone sync . test:/ # rclone remote has been setup beforehand
ERROR : a: Failed to copy: read metadata failed: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>403 Forbidden</title>
</head>
<body>
<h1>403 Forbidden</h1>
</body>
</html>: 403 Forbidden
-- SNIP --
2021/10/15 20:32:20 ERROR : Attempt 2/3 succeeded
# running rclone again, will work without errors, as long as we don't add another file, or change an existing one
rclone sync . test:/
The access log shows:
fd13::*** webdav.domain.com user [15/Oct/2021:20:32:31 +0200] "PUT /dav/test/b HTTP/2.0" 201 0 "https://webdav.domain.com/dav/test/" "rclone/v1.55.1-DEV" fd13::*** webdav.domain.com user [15/Oct/2021:20:32:31 +0200] "PROPFIND /dav/test/b HTTP/2.0" 403 341 "https://webdav.domain.com/dav/test/" "rclone/v1.55.1-DEV" fd13::*** webdav.domain.com user [15/Oct/2021:20:32:31 +0200] "PROPFIND /dav/test/ HTTP/2.0" 207 2151 "https://webdav.domain.com/dav/test/" "rclone/v1.55.1-DEV"
# lighttpd -v lighttpd/1.4.60-devel-lighttpd-1.4.59-181-gbac34c1a (ssl) - a light and fast webserver # uname -a FreeBSD webdav.domain.com 12.2-RELEASE-p9 FreeBSD 12.2-RELEASE-p9 2ee62d665f0(HEAD) TRUENAS amd64
I'm not sure if this a problem with rclone or lighttpd, but since the PROPFIND works the second time, for me this looks like lighttpd is the issue here?
Cheers
| minimal.conf (1.75 KB) minimal.conf | config file | ||
| tail_access_error.log (7.97 KB) tail_access_error.log | access and error log |
Replies (2)
RE: Webdav + rclone backup - Added by gstrauss over 3 years ago
If you strace lighttpd, I would be interested in seeing what happens with the PROPFIND that results in a 403 Forbidden after the PUT returns 201 Created for the same file.
I have a hunch that some changes in lighttpd 1.4.60 result in the PROPFIND being processed prior to the PUT, though I would still expect 404 unless the stat() returned something other than ENOENT. [Edit: disproved; hunch incorrect]
lighttpd/1.4.60-devel-lighttpd-1.4.59-181-gbac34c1a from version bac34c1a is quite old. Please upgrade your tree when you have a chance.
Aside: server.follow-symlink = "disable" is not very performant and is only a deterrent. The checks done for symlinks are not air-tight.
RE: Webdav + rclone backup - Added by gstrauss over 3 years ago
2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: PUT /dav/a HTTP/1.1 2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: Host: localhost:8080 2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: User-Agent: rclone/v1.55.1-DEV 2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: Content-Length: 0 2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: Content-Type: application/octet-stream 2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: Referer: http://localhost:8080/dav/ 2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: Accept-Encoding: gzip 2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: 2021-10-16 01:09:08: (response.c.164) fd:8 resp: HTTP/1.1 201 Created 2021-10-16 01:09:08: (response.c.164) fd:8 resp: ETag: "808694291" 2021-10-16 01:09:08: (response.c.164) fd:8 resp: Content-Length: 0 2021-10-16 01:09:08: (response.c.164) fd:8 resp: Date: Sat, 16 Oct 2021 05:09:08 GMT 2021-10-16 01:09:08: (response.c.164) fd:8 resp: Server: lighttpd/1.4.61-devel-lighttpd-1.4.60-16-g20a19e1d 2021-10-16 01:09:08: (response.c.164) fd:8 resp: 2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: PROPFIND /dav/a HTTP/1.1 2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: Host: localhost:8080 2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: User-Agent: rclone/v1.55.1-DEV 2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: Depth: 1 2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: Referer: http://localhost:8080/dav/ 2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: Accept-Encoding: gzip 2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: 2021-10-16 01:09:08: (response.c.164) fd:8 resp: HTTP/1.1 403 Forbidden 2021-10-16 01:09:08: (response.c.164) fd:8 resp: Content-Type: text/html 2021-10-16 01:09:08: (response.c.164) fd:8 resp: Content-Length: 341 2021-10-16 01:09:08: (response.c.164) fd:8 resp: Date: Sat, 16 Oct 2021 05:09:08 GMT 2021-10-16 01:09:08: (response.c.164) fd:8 resp: Server: lighttpd/1.4.61-devel-lighttpd-1.4.60-16-g20a19e1d 2021-10-16 01:09:08: (response.c.164) fd:8 resp:
rclone is sending a request PROPFIND /dav/a HTTP/1.1 with Depth: 1 and lighttpd has specific code to reject with 403 any Depth other than Depth: 0 for things (files) which are not collections (dirs).
This was imported from a development branch in lighttpd 1.4.54: https://git.lighttpd.net/lighttpd/lighttpd1.4/src/commit/50aae03c31f9dddb489092c050f9dbec21281118/src/mod_webdav.c#L3626 and originates from earlier lighttpd webdav code which attempted opendir() if Depth: 1 was specified.
RFC 4918 Section 10.2. Depth Header
If a resource does not have internal members, then the Depth header MUST be ignored.
The above suggests that the lighttpd code should ignore Depth for files.
Here is a patch for lighttpd:
--- a/src/mod_webdav.c
+++ b/src/mod_webdav.c
@@ -4014,9 +4014,8 @@ mod_webdav_propfind (request_st * const r, const plugin_config * const pconf)
http_status_set_error(r, 403);
return HANDLER_FINISHED;
}
- else if (0 != pb.depth) {
- http_status_set_error(r, 403);
- return HANDLER_FINISHED;
+ else {
+ pb.depth = 0;
}
pb.proplist.ptr = NULL;
As the previous behavior in lighttpd has been present for a very, very long time, you must be the first to use rclone with lighttpd. Since the issue you reported has not come up before with other WebDAV clients and lighttpd, maybe rclone should avoid specifying Depth: 1 when rclone does not expect the target to be a collection?
In any case, the above patch to lighttpd mod_webdav will be part of lighttpd 1.4.61, which I expect to release within the next month.