Project

General

Profile

[Abandoned] Unable to upload large files (of size 4GB and above) using lighttpd 1.4.65

Added by skchaitanya 6 months ago

Hi,

We make use of lighttpd within our embedded system and currently experiencing an issue with file uploads of size 4GB or more.
The uploads of file with size 4GB or more fails stating "500 Internal Server Error" and this is seen from version 1.4.62 and above

We experienced some errors to configure lighttpd versions 1.4.60 and 1.4.61 (as described in Bug #3116). As it was fixed in 1.4.62, we were able to configure with lighttpd v1.4.62. So don't know if this issue exists with 1.4.60 and 1.4.61 versions.

Since we are using OpenSSL 1.0.2, had some issues with configuring lighttpd 1.4.65; and even that was resolved with the Revision 222d8416: [mod_openssl] compile compat w/ openssl < 1.1.0 . But even on 1.4.65 the upload of files of size 4GB+ fails.

We are using a CGI program/script (that in-turn makes a curl call) to upload the files; and it works without any hassle for files less than 4GB, but fails with size of 4GB or more.

Also, the upload process worked seamlessly (without any issue) until lighttpd version 1.4.59, even for files of size more than 4GB.

Attached is the lighttpd conf file that we make use of.

Can I get some insights/hints on what changes with lighttpd (I know there are a lot of changes between 1.4.59 and the latest) would be affecting the large file uploads ?


Replies (9)

RE: Unable to upload large files (of size 4GB and above) using lighttpd 1.4.65 - Added by gstrauss 6 months ago

We are using a CGI program/script (that in-turn makes a curl call) to upload the files; and it works without any hassle for files less than 4GB, but fails with size of 4GB or more.

We make use of lighttpd within our embedded system

Is your embedded system running 32-bit lighttpd?

What is receiving the 4 GB upload? A CGI script on the server?

What kind of storage space do you have available on this embedded system? Your lighttpd.conf indicates that you are using lighttpd's defaults for file uploads, which means that lighttpd is storing the entire 4 GB upload in temporary files. See server.stream-request-body and server.upload-dirs

RE: Unable to upload large files (of size 4GB and above) using lighttpd 1.4.65 - Added by gstrauss 6 months ago

Aside: please review lighttpd 1.4.65 release notes
mod_uploadprogress is deprecated. A replacement is available with lua mod_uploadprogress using lighttpd mod_magnet

RE: Unable to upload large files (of size 4GB and above) using lighttpd 1.4.65 - Added by gstrauss 6 months ago

We are using a CGI program/script (that in-turn makes a curl call) to upload the files; and it works without any hassle for files less than 4GB, but fails with size of 4GB or more.

What do the request headers look like? I see that you have disabled "h2proto" in lighttpd.conf.
Does the request set Content-Length? Or is it using Transfer-Encoding: chunked?

RE: Unable to upload large files (of size 4GB and above) using lighttpd 1.4.65 - Added by gstrauss 6 months ago

I did a simple test with lighttpd 1.4.65 (64-bit) and upload of a file over 4 GB worked for me.

You should strace lighttpd and look for an error which might indicate that you are running out of disk space, e.g.
pwritev(9, [{iov_base="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"..., iov_len=32756}], 1, 4044324864) = -1 ENOSPC (No space left on device)

RE: Unable to upload large files (of size 4GB and above) using lighttpd 1.4.65 - Added by gstrauss 6 months ago

My client:
perl -e '$x = 0; do { print "x"x(1024*1024); } while (($x+=(1024*1024)) < 4*1024*1024*1024+(1024*1024));' | curl -s --http1.1 -v -k -T - https://localhost:8443/up.cgi 2>&1 | egrep -v "TLS header, Supplemental data|5 bytes data"

up.cgi

#!/usr/bin/perl

$x = $ENV{CONTENT_LENGTH} || 0;
while ($x) {
    $rd = sysread(STDIN, $data, $x > 1024*1024 ? 1024*1024 : $x);
    last if ($rd < 0);
    $x -= $rd;
}

print "Status: 200\nRead-CL: $ENV{CONTENT_LENGTH}\n\n" if ($x == 0);
print "Status: 500\nUnread: $x\n\n" if ($x != 0);

RE: Unable to upload large files (of size 4GB and above) using lighttpd 1.4.65 - Added by skchaitanya 5 months ago

Thanks for the response gstrauss.
Apologies for the delay in response as I am out of station for a couple of days.
Will try out whatever you have said and let you know about the progress by early next week.

Meanwhile, Although I will confirm once with using strace; I don't think we are running out of disk space as the 4GB+ upload works fine with lighttpd v1.4.59 (unless there have been changes to these as well in the newer versions).

RE: Unable to upload large files (of size 4GB and above) using lighttpd 1.4.65 - Added by gstrauss 5 months ago

I have shown that a simple approach to uploading a file > 4 GB does not reproduce the issue (tested on both a 32-bit lighttpd and a 64-bit lighttpd), so your issue is waiting for you to provide more details, which includes answering the questions that I previously asked.

Maybe your CGI application (if compiled) was not built with large file support?

RE: Unable to upload large files (of size 4GB and above) using lighttpd 1.4.65 - Added by skchaitanya 5 months ago

Apologies for the delay.
Answering the questions you have asked.

Is your embedded system running 32-bit lighttpd?

No, it is running 64-bit version.

What is receiving the 4 GB upload? A CGI script on the server?

Yes.

Does the request set Content-Length?

Yes, it sets Content-Length

What kind of storage space do you have available on this embedded system? Your lighttpd.conf indicates that you are using lighttpd's defaults for file uploads, which means that lighttpd is storing the entire 4 GB upload in temporary files.

Yes, we are using the lighttpd defaults and aware that it stores the entire 4GB in temp files.
I did monitor the disk space during the upload, we still had around 10-12GB of free space.

Maybe your CGI application (if compiled) was not built with large file support?

I am not sure whether it was built with large file support or not, but we are providing the largefile support compilation option (so I think it is).

mod_uploadprogress is deprecated. A replacement is available with lua mod_uploadprogress using lighttpd mod_magnet

Thanks for this information. We are facing some issues configuring lighttpd with-lua option, but yeah that is a separate problem that we need to tackle.

Also as I said earlier, this greater than 4GB upload works fine with 1.4.59 version of lighttpd; so is there any changes with file uploads/disk space handling or anything that could be affecting here ? Do you think using the lua mod_uploadprogress of mod_magnet might resolve this ?

RE: Unable to upload large files (of size 4GB and above) using lighttpd 1.4.65 - Added by gstrauss 5 months ago

Also as I said earlier, this greater than 4GB upload works fine with 1.4.59 version of lighttpd; so is there any changes with file uploads/disk space handling or anything that could be affecting here ?

Possibly, which is why I asked the questions I did.

Do you think using the lua mod_uploadprogress of mod_magnet might resolve this ?

No, that would be unrelated.

As an aside: taking a week or more of latency to reply to posts suggests that this is not a priority for you.

You should: test without server.max-worker = 4; comment it out. You're on an embedded system.
You should: try the test code that I posted above.
You should: try the test code that I posted above to a test location which is not TLS (i.e. http not https)
You should: strace lighttpd and save the output (during a 4 GB upload) to a log file that can be analyzed.
You should: strace whatever process is receiving the upload on the server, whether that is a CGI or proxy.

If possible, you should test using a newer version of openssl, i.e. a version that is not end-of-lifed.

    (1-9/9)