Lighttpd

I now made tests with
server.upload-dirs = ( "/opt-pdd", "/tmp" )

server.stream-request-body = 1
and:
server.upload-temp-file-size = 15728640 //whole upload would fit in here
server.upload-temp-file-size = 16384
server.upload-temp-file-size = 4096

All three without success. It fails at different progress points and I got different number of lighttpd-upload* files during upload.

I am on a small embedded device with slow flash disk
Is server.stream-request-body = 1 the right choice?
Is this behavior expected?
Is the configuration the problem?

I also built against 1.4.65. The error occurs here as well. So the regression was between 1.4.64 and 1.4.65. Maybe this is helpful.

Thank you for helping narrow it down to between 1.4.64 and 1.4.65. I'll dig into this issue this weekend, if not sooner.

Is server.stream-request-body = 1 the right choice?

Probably not for you. Are you processing the file by the CGI while it is being uploaded? Or do you process it after the file has been uploaded completely? If the latter, then server.stream-request-body = 0 is the right choice for you.

What client and what client version are you using to upload the file?

Is this behavior expected?

(chunk.c.780) opening temp-file failed: /opt-pdd/lighttpd-upload-y6zw4v: Interrupted system call

That is not expected.

man 2 open

EINTR While blocked waiting to complete an open of a slow device (e.g., a FIFO; see fifo(7)), the call was interrupted by a signal handler; see signal(7).

What filesystem type is your flash? If your system has enough memory for the file uploads, you might use "/dev/shm" as the first entry in server.upload-dirs = ( "/opt-pdd", "/tmp" ), or place "/tmp" first if "/tmp" is tmpfs.

lighttpd does not currently use SA_RESTART with SIGINT. Doing so globally across the entire lighttpd process might lead to unintented consequences and would need to be explored more deeply. Instead, maybe I'll add a retry loop to retry up to 3 times if the mkostemp() fails with EINTR.

It may be useful to me if you could compare config.h from your builds of lighttpd 1.4.64 and lighttpd 1.4.65.

Is lighttpd receiving a signal which interrupts the system call? What signal? Try capturing strace -o strace.log -p <lighttpd-pid> during any upload request.

strace was attached in the first post. Was it valuable? If not I make another one on Monday.

Besides version number change (expected), I do not see any substantive differences between 1.4.64_config.h and 1.4.65_config.h

From the original strace:

13:03:00.600816 open("/opt-pdd/lighttpd-upload-y6zw4v", O_RDWR|O_CREAT|O_EXCL|O_CLOEXEC, 0600) = -1 EINTR (Interrupted system call)
13:03:00.634685 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1389, si_uid=0, si_status=0, si_utime=0, si_stime=1} ---
13:03:00.636150 rt_sigreturn({mask=[]}) = -1 EINTR (Interrupted system call)
13:03:00.639544 send(6, "<27>Aug 25 13:03:00 lighttpd[1264]: (chunk.c.780) opening temp-file failed: /opt-pdd/lighttpd-upload-y6zw4v: Interrupted system call", 132, MSG_NOSIGNAL) = 132

Would you strace lighttpd at startup to confirm that SA_RESTART is being set for SIGCHLD?

The original strace appears to show your CGI program exiting before the upload is complete. Are you perhaps backgrounding the CGI to detach the process from lighttpd?

While I am still interested in tracking down why this behavior changed for you, if your CGI script does not need to process the request body until after the request body is completely uploaded, then server.stream-request-body = 0 is recommended.

I attached a startup trace. files.cgi is the actual upload cgi. I had to stop the strace before the lost connection because of disk space.

00:33:19.175820 rt_sigaction(SIGCHLD, {sa_handler=0x100074f0, sa_mask=[], sa_flags=SA_RESTART|SA_SIGINFO|SA_NOCLDSTOP}, NULL, 8) = 0

I see that SA_RESTART is being set for SIGCHLD, so why lighttpd call to open() is being interrupted and not restarted is a mystery. open() is explicitly listed as restartable in man 7 signal for a signal configured with SA_RESTART.

Does stat() count as "processing the request body"?

Yes, but you might instead consider using mod_magnet lua mod_uploadprogress for that part.

I tried it again with server.stream-request-body = 0. It works , but with up to 90000 lighttpd-upload* tmp files. Do you judge this to be harmless?

Did you clear the temporary directories before you restarted with server.stream-request-body = 0 ?
(lighttpd is unable to clean up those temporary files if created by mkostemp and not reported as success by mkostemp due to EINTR)
Did you modify server.upload-temp-file-size ? The default is 1 MB. If you see lots and lots of temp files, what are their sizes?

Second I tried server.stream-request-body = 0 and server.upload-dirs = ( "/tmp", "/opt-pdd" ). /tmp is fast RAM, but only 5 Mb free disk space. The connection was lost after /tmp was filled up. Shouldn't there be a fallback to /opt-pdd?

Yes, I have previously tested that the fallback works when ENOSPC occurs. Do you have an strace of that failing?

I attached a startup trace. files.cgi is the actual upload cgi. I had to stop the strace before the lost connection because of disk space.

00:33:19.175820 rt_sigaction(SIGCHLD, {sa_handler=0x100074f0, sa_mask=[], sa_flags=SA_RESTART|SA_SIGINFO|SA_NOCLDSTOP}, NULL, 8) = 0

I see that SA_RESTART is being set for SIGCHLD, so why lighttpd call to open() is being interrupted and not restarted is a mystery. open() is explicitly listed as restartable in man 7 signal for a signal configured with SA_RESTART.

... If you know some C, I would suggest writing a small test program to see if you could interrupt an open() in a program with signal SIGCHLD with SA_RESTART set by sending a SIGCHLD to the program during the open().

If you recently upgraded the OS or glibc on your system, I would test with an older version. If you haven't upgraded your OS or glibc recently, I would check for and test an upgraded version.

To repeat:
lighttpd is unable to clean up those temporary files if created by mkostemp and not reported as success by mkostemp due to EINTR.
You must manually delete those files before you continue your testing.
lighttpd is not creating 90000 lighttpd-upload* tmp files for a single 10 MB upload. (By default, lighttpd might create ten (10) temp files, each 1 MB in size)

... If you know some C, I would suggest writing a small test program to see if you could interrupt an open() in a program with signal SIGCHLD with SA_RESTART set by sending a SIGCHLD to the program during the open().

Do you maybe have a kind of "stackoverflow" example where I could start?
Linux is 3.10.28, libc-2.18.so, Unfortunately I can not change this.

What is INTMAX_MAX defined to on your embedded system? Is INTMAX_MAX properly defined in your build environment?
Use the appropriate cross-compiler flags with the following to get the value for the embedded system build.
echo "#include <stdint.h>" | gcc -dM -E - | grep "define INTMAX_MAX"

I have not been able to reproduce the many, many tempfiles you are seeing.

For a CGI backend and server.stream-request-body = 0, the default lighttpd behavior is to create a single tempfile and then attach that to CGI stdin. I confirmed that I see this behavior on my machine. Similarly, I confirmed that when server.stream-request-body = 1, lighttpd creates multiple 1MB tempfiles if the backend delays reading CGI stdin.

In your strace lighttpd_tmpdir.trace.gz, I see that lighttpd is performing a single pwritev() to a tempfile, writing approx 4k - 16k. When more data is available, the tempfile is closed and a new one is opened. That should not happen unless the tempfile has exceeded cq->upload_temp_file_size: https://git.lighttpd.net/lighttpd/lighttpd1.4/src/branch/master/src/chunk.c#L801

Did you notice any change in temporary files sizes when you tested with different settings of server.upload-temp-file-size = 1048576 ? Please make sure that you properly restart lighttpd after making changes to lighttpd.conf.

Please do a clean and full build of lighttpd 1.4.66 with the patch above to test.

echo "#include <stdint.h>" | powerpc-linux-gcc -dM -E - | grep "define INTMAX_MAX" 
#define INTMAX_MAX (__INT64_C(9223372036854775807))

I see no differences between 1.4.65 and 1.4.66
I have disabled javascript, so no processing of the uploaded file with stat
I have no /dev/shm but /tmp is a ramdisk with 5Mb free.

1.4.66 patched WORKING (unpatched is not working)
1.4.65 patched WORKING
server.upload-dirs = ( "/tmp", "/opt-pdd/" )
server.stream-request-body = 1
several (5-9?) 1Mb lighttpd-upload- files distributed between /tmp and /opt-pdd

1.4.66 patched WORKING
1.4.65 patched WORKING
server.upload-dirs = ( "/opt-pdd" )
server.stream-request-body = 1
one lighttpd-upload- file < 1MB in /opt-pdd
indeed it is NOT working if I process the uploaded tmp file with stat() -> connection loss

1.4.65 patched WORKING
1.4.65 patched WORKING
server.upload-dirs = ( "/opt-pdd" )
server.stream-request-body = 0
about 1000 lighttpd-upload- files in /opt-pdd

1.4.66 patched WORKING
1.4.65 patched WORKING
server.upload-dirs = ( "/tmp", "/opt-pdd" )
server.stream-request-body = 0
about 1000 lighttpd-upload- files distributed between /tmp and /opt-pdd
size between 384 - 16000 bytes, 16000 seems to be the maximum!
I tried server.upload-temp-file-size = 8192, but this is not respected. max filesize is still 16000

So it seems
server.upload-dirs = ( "/tmp", "/opt-pdd/" )
server.stream-request-body = 1
is what you expect with
server.upload-dirs = ( "/tmp", "/opt-pdd/" )
server.stream-request-body = 0 ?

It may be of interest, the powerpc cpu is 32bit big-endian

I do not think that I understand the way the you structured your post with "WORKING" and then describing not working.

With server.upload-dirs = ( "/opt-pdd" ) and server.stream-request-body = 0, I expect a single temp file in /opt-pdd.

With server.upload-dirs = ( "/tmp", "/opt-pdd/" ) and server.stream-request-body = 0, I expect a handful tempfiles, with one in /tmp, and then the rest in /opt-pdd.

With server.upload-dirs = ( "/tmp", "/opt-pdd/" ) and server.stream-request-body = 1, I expect a handful tempfiles, with some in /tmp, and then the rest in /opt-pdd, but the number also depends on how quickly the backend processes the request body.

It may be of interest, the powerpc cpu is 32bit big-endian

Yes. it is.

Did you test again with lighttpd 1.4.64? Is that still working WITHOUT creating an excessing number of temp files?

You posted

server.upload-dirs = ( "/opt-pdd" )
server.stream-request-body = 0
about 1000 lighttpd-upload- files in /opt-pdd

                /* store request body in single tempfile if not streaming request body*/
                r->reqbody_queue.upload_temp_file_size = INTMAX_MAX;

cq->upload_temp_file_size is of type (off_t) and lighttpd should always be built with 64-bit file support, so (off_t) should be the same size as (int64_t)

In lighttpd 1.4.60, mod_cgi was changed to use a single temp file if not streaming.
https://git.lighttpd.net/lighttpd/lighttpd1.4/commit/70195d069d7baf7ecb53db186176f1dee61817f5
You can try setting server.feature-flags += ("cgi.tempfile-accum" => "disable") to disable this feature and see if that makes the number of temp files with server.stream-request-body = 0 reduce to about 10 temp files.

What does your compiler say for echo "#include <stdint.h>" | powerpc-linux-gcc -dM -E - | grep OFF ?

Your issue likely stems from your system not sizing (off_t) to (int64_t) for large filesystem support.

--- a/src/mod_cgi.c
+++ b/src/mod_cgi.c
@@ -1028,7 +1028,8 @@ URIHANDLER_FUNC(cgi_is_handled) {
            && !(r->conf.stream_request_body /*(if not streaming request body)*/
                 & (FDEVENT_STREAM_REQUEST|FDEVENT_STREAM_REQUEST_BUFMIN))) {
                /* store request body in single tempfile if not streaming request body*/
-               r->reqbody_queue.upload_temp_file_size = INTMAX_MAX;
+               r->reqbody_queue.upload_temp_file_size =
+                 (off_t)((1uLL << (sizeof(off_t)*8-1))-1);
        }

        {

If (sizeof(off_t) != sizeof(int64_t)), your cross-compiler is not operating as lighttpd expects since lighttpd expects to be able to force large file support in lighttpd builds. The patch above attempts to work around that. On your system, you should see an excessive number of upload temp files created if using server.stream-request-body = 0 and any version of lighttpd 1.4.60 through lighttpd 1.4.66.

Separately, your system is broken in some way if open() is being interrupted by SIGCHLD when SIGCHLD has been configured with sigaction() and flag SA_RESTART. Workarounds for you have been suggested in a post further above, including deferring your fork and backgrounding of your CGI until after the upload request body has been fully received, or server.stream-request-body = 1 with server.upload-temp-file-size = 15728640 //whole upload would fit in here, along with the pwritev() retry patch further above.

Did you test again with lighttpd 1.4.64? Is that still working WITHOUT creating an excessing number of temp files?

I tried it again. No it is not working at all. Number of temp files > 10000.

I do not think that I understand the way the you structured your post with "WORKING" and then describing not working

With WORKING I meant: I get a complete upload file

that on your system it results in a negative number in r->reqbody_queue.upload_temp_file_size such that tests in chunk.c:

You are exactly right. I did some printf at startup to test:

server.c:2070:main  sizeof(int64_t): 8
server.c:2071:main  sizeof(off_t): 4
Error size differs
server.c:2073:main  INTMAX_MAX: 9223372036854775807
server.c:2075:main  intmaxcast: -4294966784

Now I have
1.4.66 two patches WORKING
server.upload-dirs = ( "/opt-pdd" )
server.stream-request-body = 0
server.feature-flags += ("cgi.tempfile-accum" => "disable")
ten lighttpd-upload- files in /opt-pdd

1.4.66 two patches WORKING
server.upload-dirs = ( "/opt-pdd" )
server.stream-request-body = 1
server.upload-temp-file-size = 15728640
one lighttpd-upload- files in /opt-pdd

I am very happy with it. Thank you for your great work and much patience.
I will be out of office for the next week if there are any questions for me.

Will the two patches merged to the master?

Your problem stems from your own build configuration:
$ ./configure --host=powerpc-linux --build=i386-gnu-linux --prefix=/usr --libdir=/lib/lighttpd --sysconfdir=/etc/lighttpd --with-pam --disable-lfs --without-pcre --without-pcre2 --without-bzip2 --without-zlib --with-openssl=/tmp/NMS/powerpc/eldk-5.5.3

The --disable-lfs is not recommended. Why did you include it? What do you think it gains you?

If your interest is in executable size, then you should consider using mbedtls instead of openssl.

Please consider removing --disable-lfs from your build. There may very well be other places in lighttpd that fail with small off_t. You are not using a recommended or well-tested config.

Also, your system mishandling SA_RESTART is a separate issue with your system.

@dani254 while this issue will be closed, you still have an issue on your system:

Your system is broken in some way if open() is being interrupted by SIGCHLD when SIGCHLD has been configured with sigaction() and flag SA_RESTART.

While the changes above make the interrupted open() much less likely to occur, the problem on your system remains and you should look into it.