Lighttpd

You missed the streaming options in Docs_ConfigurationOptions.
server.stream-response-body = 1

Another convention for doing this is to fork your script to continue processing in a child process. The main script can then send the response and exit. (Once the main script closes stdout, lighttpd will send a TERM signal to the initial CGI procress.)

Thanks for your response. However, I have already tried to use "server.stream-response-body = 1" (which I forgot to mention, I am afraid) but without success.

And regarding the suggestion to fork the script so that the script continues processing in a child process. AFAIK then the child process cannot send any further progress output to the browser as soon as the parent exited properly. Or did I miss something?

If you tried server.stream-response-body = 1, then please verify your config applies that setting to the CGI, and try something like Perl or Python instead of your shell script. Please make sure that your shell script is behaving as you think it is. lighttpd is unable to send things which are not first sent to it.

You are correct that the child process can not send further output once the main CGI exits.

https://stackoverflow.com/questions/3465619/how-to-make-output-of-any-shell-command-unbuffered
Among other things, the page notes that stdbuf does not work on statically linked executables. If I am not mistaken, ash is statically linked. You'll have to use pipe tricks instead, or better, reconsider your choices in scripting language, especially since syntax like eval $(echo ${QUERY_STRING//&/;}) potentially contains multiple exploitable security holes.

HI again,

I have continued my testing in trying to find a solution for my above mentioned problem in not getting any continues progress information while my cgi upload and processing script is running. Thanks for your comments also about stdbuf not working on statically linked binaries (I didn't keep that in mind).

For testing purposes and to understand the issue more throughoutly I have written a small C-application that should help to demonstrate the problem and remove potential issues hided in the shell output buffering, etc. The source code of this C-application is as follows:

#include <stdio.h>
#include <unistd.h>

void main(void)
{
  setvbuf(stdout, NULL,  _IONBF, 0);

  printf("Content-Type: text/plain\r\n");
  printf("\r\n");
  fflush(stdout);

  printf("TEST\r\n");
  int i=0;
  for(i=0; i < 10; i++)
  {
    printf(".");
    fflush(stdout);
    sleep(1);
  }
  printf("\r\n");
  fflush(stdout);
  printf("DONE\r\n");
}

As you can see I am using setvbuf() and fflush() to ensure that the stdout buffer is flushed as soon as possible so that progress information is outputted throughout each step of using printf() here. I have then compiled this C application and used it as a cgi script instead of the ash shell script for being the form-data cgi script during uploading data via a web browser. However, even with that unbuffered C application lighttpd is not returning me the single dots (.) throughout the 10 seconds of execution of the C application once the file has been uploaded. In fact, the web browser only displays the complete outputted text in one chunk once this C application finished no matter if I have "server.stream-response-body" set to 0 or 1 in lighttpd.conf.

This makes me wonder if lighttpd is really performing correctly as you suggested since this C application should really run completly unbuffered and thus lighttpd should immediately forward the output to the webbrowser. Or do I still miss something here?

Oh well, I actually found the culprit. It actually is so simple that it is already strange that this should really be the issue :)

All I had to change was NOT to use "Content-Type: text/plain" as the first output string but to use "Content-Type: text/html" instead. I still don't fully understand why this makes the big difference but this actually solved the problem (is there something within lighttpd that it only directly forwards the response to the browser if the content-type of the response is text/html?) and now I continuously get each single printf() outputted immediately. In fact, I just tested it with my ash script and actually this also works now as expected.

More info: This even gets worse. In fact, it seems to be browser specific as for Chrome/Chromium it seems to be enough that "Content-Type: text/html" is specified in the response while for Firefox Content-Type: text/html; charset=iso-8859-1 is required to let it directly render the response received from the cgi. Well, for Firefox it doesn't seem to be relevant if the content type is text/plain or text/html, but the charset have to be specified. Really awkward and I still don't know why they are behaving so strange regarding receiving continues responses.

To summarize, there was no bug in lighttpd. Your issue turned out to be you getting fooled by your browser(s). In the future, please remove the browser by testing with telnet or curl with debugging enabled.

Yes, indeed. Sorry for having bothered you. Hopefully someone else will step over this discussion and learn the same lesson. However, I am still curious why on earth browsers require the content-type to actually render responses right away. However, I guess this is a discussion for a different fora and shouldn't be discussed here.

I am still curious why on earth browsers require the content-type to actually render responses right away.

The very definition of "Content-Type" is the answer to that question. Without being told the type of content, how should a GUI browser know whether to attempt to render it as text or to pop up a dialog to save a binary or to pop up an application to play it as a video or ... ?