Project

General

Profile

Actions

Bug #1674

closed

Copying (big) files onto server fails with WinXP clients

Added by Anonymous almost 16 years ago. Updated 3 months ago.

Status:
Obsolete
Priority:
Normal
Category:
mod_webdav
Target version:
-
ASK QUESTIONS IN Forums:
No

Description

This is a problem I've already asked in the Forum but got no response.

I run a small webserver with lighttpd (1.4.19) and WebDAV. The server environment is somthing like a homebrew busybox environment running on an old P166 with 48MB RAM.

All software was compiled on a Debian Linux 3.1 box whith no updates as it was a CD install without internet connection. (I write this to let you know that it's not the latest build environment, if that matters.)

The problem is the following: If I copy a file onto the server which is bigger than about 10MB the process fails and XP reports that the copying of one or more files failed.

I can download file of any size, and the problem doesn't occur if I use cadaver to copy the file onto the server.
Thus I assume it has something to do with the combination lighttpd and WinXP.
The reason might be an uncorret behaviour of any side, but I think it should be investigated because most clients are usually Win.

I've connected the build box to the LAN and checked if the problem existes on it too. It is a AMD XP3200 with 1.5GB RAM, and thus several times faster.
The problem seems to exist with it too, but I need to copy much bugger files to trigger it.

The problem seems to arise while lighttpd copies the temporary files in /var/tmp into the destination file, but I'm not 100% sure.

It looks like a timeout problem to me and thus, the slower the machine, the easier it is to trigger.

I've compiled and tested lighttpd on a Ubuntu 8.04 box with latest updates too and wasn't able to trigger it there but it is only connected via a 10M link so I didn't test really big files with it.

-- trashy993-hosting

Actions #1

Updated by Anonymous almost 16 years ago

I've tested it on the Ubunbtu box, too. A fairly large file is needed to trigger it (>1GB) but it is possible, thus it's obviously not a problem with the old build system.

-- trashy993-hosting

Actions #2

Updated by stbuehler over 15 years ago

What exact operation did you do?

If I copy a file onto the server which is bigger than about 10MB the process fails and XP reports that the copying of one or more files failed.

html form upload, webdav, ... ?

error.log from lighttpd?

And if your upload times out... what do you think lighty should do? wait forever?...

Actions #3

Updated by Anonymous over 15 years ago

Replying to stbuehler:

What exact operation did you do?

I tried to do file operations via webdav. Copy file onto the server, delete files on the server and download files from the server.

If I copy a file onto the server which is bigger than about 10MB the process fails and XP reports that the copying of one or more files failed.

html form upload, webdav, ... ?

The whole bugreport is about copying files using webdav. Sorry, I thought that was clear.

error.log from lighttpd?

Well. The error log doesn't say anthing really useful.

192.168.0.1 - - +0200 "PUT /dav/user1/big.file HTTP/1.1" 413 0 "-" "Microsoft Data Access Internet Publishing Provider DAV"
192.168.0.1 192.168.0.13:1234 user1 +0200 "HEAD /dav/user1/upi-41ah.pdf HTTP/1.1" 404 0 "-" "Microsoft Data Access Internet Publishing Provider DAV"
192.168.0.1 192.168.0.13:1234 user1 +0200 "PUT /dav/user1/upi-41ah.pdf HTTP/1.1" 201 0 "-" "Microsoft Data Access Internet Publishing Provider DAV"
192.168.0.1 192.168.0.13:1234 user1 +0200 "DELETE /dav/user1/upi-41ah.pdf HTTP/1.1" 204 0 "-" "Microsoft Data Access Internet Publishing Provider DAV"
192.168.0.1 192.168.0.13:1234 user1 +0200 "HEAD /dav/user1/big.file HTTP/1.1" 404 0 "-" "Microsoft Data Access Internet Publishing Provider DAV"
192.168.0.1 - - +0200 "PUT /dav/user1/big.file HTTP/1.1" 413 0 "-" "Microsoft Data Access Internet Publishing Provider DAV"
192.168.0.1 192.168.0.13:1234 user1 +0200 "HEAD /dav/user1/big.file HTTP/1.1" 404 0 "-" "Microsoft Data Access Internet Publishing Provider DAV"
192.168.0.1 192.168.0.13:1234 user1 +0200 "PUT /dav/user1/big.file HTTP/1.1" 201 0 "-" "Microsoft Data Access Internet Publishing Provider DAV"
192.168.0.1 192.168.0.13:1234 user1 +0200 "PROPFIND /dav/user1 HTTP/1.1" 207 1172 "-" "Microsoft Data Access Internet Publishing Provider DAV"
192.168.0.1 192.168.0.13:1234 user1 +0200 "DELETE /dav/user1/big.file HTTP/1.1" 204 0 "-" "Microsoft Data Access Internet Publishing Provider DAV"
192.168.0.1 192.168.0.13:1234 user1 +0200 "PROPFIND /dav/user1 HTTP/1.1" 207 674 "-" "Microsoft Data Access Internet Publishing Provider DAV"
192.168.0.1 192.168.0.13:1234 user1 +0200 "PROPFIND /dav/user1 HTTP/1.1" 207 674 "-" "Microsoft Data Access Internet Publishing Provider DAV"

and

2008-05-25 17:20:01: (request.c.1132) request-size too long: -0 -> 413
2008-05-25 17:20:31: (request.c.1132) request-size too long: -0 -> 413
2008-05-25 19:54:43: (server.c.1475) server stopped by UID = 0 PID = 8105

Note that those 2 413 errors are not the errors I'm talking about. The one I'm talking about is the last PUT in access.log and as you can see didnÄt produce any error in the logfile.

And if your upload times out... what do you think lighty should do? wait forever?...

I've already explained what the problem seems to be.
Ok. I'll rewrite that part a little bit more detailed:

As far as I have found out, the following happens:
While the data ist transmitted to lighty (via webdav), it puts the received data
into several seperate files in its temporary directory.
After the whole file has been transmitted, it copies the small chunks together into the destination directory, where the user wanted to put the file.

The problem seems to be the following: On a slow or busy machine with big files,
it may take quite long to copy e.g. 1.5 GB from the temp-dir over to its destination.
I don't know if lighty needs to send any acknowledgement after having finished or if the client wants something else while lighty is still copying, but the problem seems to occur during this part of the transmission process.

I don't know what's going wrong in detail (why the client exactly prints the error
message), maybe a wireshark trace could reveal that, but I have no time for that.

Obviosly, not all clients seem to behave the same way. WinXP and Win2k seem to have/trigger the problem, while cadaver doesn't.

Actions #4

Updated by stbuehler over 15 years ago

  • Status changed from New to Fixed
  • Resolution set to wontfix
ok, let me summarize:
  • lighttpd does not timeout
  • the file is received completely and not corrupted (and http response is 201 Created)
  • only the webdav client (winxp / win2k) has a timeout, as the reponse is sent after the tmp-files are copied into the destination file

I don't see how we could fix that without a mod_webdav rewrite - i am sorry, but this is a wontfix :(.

So your server machine should be fast enough to copy the files you want to upload before your client timeouts.

Actions #5

Updated by Anonymous over 15 years ago

Well, actually I haven't investigated what exactly happens. I wrote only what I think that happens.
To find out the exact reason, further investigation would be needed (to find out why some clients timeout).

I think that this problem should be fixed.
Why?
Because the status quo is that the maximum filesize you can copy depends on you server speed (mostly on the harddisk speed and load to be more precise) and maybe on server load.
So for those (XP/2k) clients, which are maybe 90% (just a guess, but you know what mean) of the clients out there the maximum filesize is maybe most of the time sufficient, but totally unpredictable to be precise.

Of course, after it has aborted, you could check the filesize to be sure and be happy.
(The problem is if you wanted to copy more than one file, because the client aborts.)
But try to explain that to the average client user.

My opinion is that those problems render the webdav support unusable.
At least for me and those who want to use webdav with lighty as a small and
convinent replacement for ftp.

So, what do we need to do?
Investigate it to understand really.
We could (for example) not use chunks but write everything in one file and move that file to destination after a complete transfer. That would at least remove the problem if the temp-dir and destination dir are on the same filesystem, because moving is fast.
Alternatively we could drop the usage of a common temp-dir completely and use something like an "incoming" dir below the document root, which is below the document root and use the file-moving tactic. This way, chances are high, that in 99% of the lighty users we make a move on the same filesystem as the document root usually doesn't spread aross multiple filesystems.

Well, that are just my proposals on solving the problem.
(Note that I haven't read the webdav source, so I don't know if this is possible
without a rewrite.)

I may investigate this myself but not now and not in the near future, because I don't have time for that.

-- trashy993-hosting

Actions #6

Updated by Anonymous over 15 years ago

I've had some time and did some Wireshark tests.

Interestingly I found out that under my current test
conditions the error seems to show up much earlier,
at least sometimes.

The error message on the client appears long before all
of the file is transmitted or after completion like I reported before.

I did the following:
1) Start copying a 50MB file onto the server with WebDAV.
2) Waited till the client error message popped up.
This was at about 10MB packet capture size (shown by Wireshark).
Strangely, the client kept sending while the error message box
was on the screen and stopped about 4MB later.
(I think that the client flushed its buffers.)

So what is the conclusion? I'm now convinced that this is a silly
Windows client problem and seems to have to be solved on the client side.

I'm very sorry for bugging you and (maybe) wasting your time.

-- trashy993-hosting

Actions #7

Updated by stbuehler over 15 years ago

  • Status changed from Fixed to Wontfix
Actions #8

Updated by gstrauss 3 months ago

  • Description updated (diff)
  • Status changed from Wontfix to Obsolete
  • Target version deleted (1.5.0)
  • ASK QUESTIONS IN Forums set to No

Issues such as this were very likely fixed long ago in lighttpd mod_webdav.
lighttpd mod_webdav writes a file upload into a temporary file next to the target, and then atomically renames the temporary file into place once the upload is complete.

Please file a new issue if that is not the case.

Actions

Also available in: Atom