Project

General

Profile

[Solved] Webdav + rclone backup

Added by meeb5 about 2 months ago

Hi all,

I'm encountering a problem when I'm trying to backup files to a webdav share via rclone:

Starting from an empty folder:

touch a
rclone sync . test:/  # rclone remote has been setup beforehand
ERROR : a: Failed to copy: read metadata failed: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>403 Forbidden</title>
 </head>
 <body>
  <h1>403 Forbidden</h1>
 </body>
</html>: 403 Forbidden

-- SNIP --
2021/10/15 20:32:20 ERROR : Attempt 2/3 succeeded

# running rclone again, will work without errors, as long as we don't add another file, or change an existing one
rclone sync . test:/

The access log shows:

fd13::*** webdav.domain.com user [15/Oct/2021:20:32:31 +0200] "PUT /dav/test/b HTTP/2.0" 201 0 "https://webdav.domain.com/dav/test/" "rclone/v1.55.1-DEV" 
fd13::*** webdav.domain.com user [15/Oct/2021:20:32:31 +0200] "PROPFIND /dav/test/b HTTP/2.0" 403 341 "https://webdav.domain.com/dav/test/" "rclone/v1.55.1-DEV" 
fd13::*** webdav.domain.com user [15/Oct/2021:20:32:31 +0200] "PROPFIND /dav/test/ HTTP/2.0" 207 2151 "https://webdav.domain.com/dav/test/" "rclone/v1.55.1-DEV" 

# lighttpd -v
lighttpd/1.4.60-devel-lighttpd-1.4.59-181-gbac34c1a (ssl) - a light and fast webserver
# uname -a
FreeBSD webdav.domain.com 12.2-RELEASE-p9 FreeBSD 12.2-RELEASE-p9 2ee62d665f0(HEAD) TRUENAS  amd64

I'm not sure if this a problem with rclone or lighttpd, but since the PROPFIND works the second time, for me this looks like lighttpd is the issue here?

Cheers

minimal.conf (1.75 KB) minimal.conf config file
tail_access_error.log (7.97 KB) tail_access_error.log access and error log

Replies (2)

RE: Webdav + rclone backup - Added by gstrauss about 2 months ago

If you strace lighttpd, I would be interested in seeing what happens with the PROPFIND that results in a 403 Forbidden after the PUT returns 201 Created for the same file.

I have a hunch that some changes in lighttpd 1.4.60 result in the PROPFIND being processed prior to the PUT, though I would still expect 404 unless the stat() returned something other than ENOENT. [Edit: disproved; hunch incorrect]

lighttpd/1.4.60-devel-lighttpd-1.4.59-181-gbac34c1a from version bac34c1a is quite old. Please upgrade your tree when you have a chance.

Aside: server.follow-symlink = "disable" is not very performant and is only a deterrent. The checks done for symlinks are not air-tight.

RE: Webdav + rclone backup - Added by gstrauss about 2 months ago

2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: PUT /dav/a HTTP/1.1
2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: Host: localhost:8080
2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: User-Agent: rclone/v1.55.1-DEV
2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: Content-Length: 0
2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: Content-Type: application/octet-stream
2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: Referer: http://localhost:8080/dav/
2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: Accept-Encoding: gzip
2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: 
2021-10-16 01:09:08: (response.c.164) fd:8 resp: HTTP/1.1 201 Created
2021-10-16 01:09:08: (response.c.164) fd:8 resp: ETag: "808694291" 
2021-10-16 01:09:08: (response.c.164) fd:8 resp: Content-Length: 0
2021-10-16 01:09:08: (response.c.164) fd:8 resp: Date: Sat, 16 Oct 2021 05:09:08 GMT
2021-10-16 01:09:08: (response.c.164) fd:8 resp: Server: lighttpd/1.4.61-devel-lighttpd-1.4.60-16-g20a19e1d
2021-10-16 01:09:08: (response.c.164) fd:8 resp: 
2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: PROPFIND /dav/a HTTP/1.1
2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: Host: localhost:8080
2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: User-Agent: rclone/v1.55.1-DEV
2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: Depth: 1
2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: Referer: http://localhost:8080/dav/
2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: Accept-Encoding: gzip
2021-10-16 01:09:08: (connections.c.775) fd:8 rqst: 
2021-10-16 01:09:08: (response.c.164) fd:8 resp: HTTP/1.1 403 Forbidden
2021-10-16 01:09:08: (response.c.164) fd:8 resp: Content-Type: text/html
2021-10-16 01:09:08: (response.c.164) fd:8 resp: Content-Length: 341
2021-10-16 01:09:08: (response.c.164) fd:8 resp: Date: Sat, 16 Oct 2021 05:09:08 GMT
2021-10-16 01:09:08: (response.c.164) fd:8 resp: Server: lighttpd/1.4.61-devel-lighttpd-1.4.60-16-g20a19e1d
2021-10-16 01:09:08: (response.c.164) fd:8 resp: 

rclone is sending a request PROPFIND /dav/a HTTP/1.1 with Depth: 1 and lighttpd has specific code to reject with 403 any Depth other than Depth: 0 for things (files) which are not collections (dirs).
This was imported from a development branch in lighttpd 1.4.54: https://git.lighttpd.net/lighttpd/lighttpd1.4/src/commit/50aae03c31f9dddb489092c050f9dbec21281118/src/mod_webdav.c#L3626 and originates from earlier lighttpd webdav code which attempted opendir() if Depth: 1 was specified.

RFC 4918 Section 10.2. Depth Header

If a resource does not have internal members, then the Depth header MUST be ignored.

The above suggests that the lighttpd code should ignore Depth for files.

Here is a patch for lighttpd:

--- a/src/mod_webdav.c
+++ b/src/mod_webdav.c
@@ -4014,9 +4014,8 @@ mod_webdav_propfind (request_st * const r, const plugin_config * const pconf)
         http_status_set_error(r, 403);
         return HANDLER_FINISHED;
     }
-    else if (0 != pb.depth) {
-        http_status_set_error(r, 403);
-        return HANDLER_FINISHED;
+    else {
+        pb.depth = 0;
     }

     pb.proplist.ptr  = NULL;

As the previous behavior in lighttpd has been present for a very, very long time, you must be the first to use rclone with lighttpd. Since the issue you reported has not come up before with other WebDAV clients and lighttpd, maybe rclone should avoid specifying Depth: 1 when rclone does not expect the target to be a collection?

In any case, the above patch to lighttpd mod_webdav will be part of lighttpd 1.4.61, which I expect to release within the next month.

    (1-2/2)