Lighttpd

lighttpd 1.4.33 is fairly old. Lastest version is 1.4.39. You might google for later packages for your system. Here's one from debian: http://http.us.debian.org/debian/pool/main/l/lighttpd/lighttpd_1.4.39-1_amd64.deb

While it is possible you've found a bug, it's unlikely that you're the first to find it in such an old version. Would you try a newer version?

More likely, this is an issue with your PHP script. Does it help if you unlink($tmp_file) before exec() of zip? Please see the documentation for tempnam(). http://php.net/manual/en/function.tempnam.php

4.0.3 This function's behavior changed in 4.0.3. The temporary file is also created to avoid a race condition where the file might appear in the filesystem between the time the string was generated and before the script gets around to creating the file. Note, that you need to remove the file in case you need it no more, it is not done automatically.

Importantly, you might want to log the $tmp_file name. If the filename generated is not unique, then lighttpd may have cached the file size of a previous version of the file. (This will be fixed in the next release of lighttpd, to be 1.4.40.)

Ok.

I installed 1.4.39 (bit nervous to do so since it's a live system), but got there, and the problem persists.

No the problem is not inside the PHP. I have tried and deleted all the files from the temp folder, so it is 100% empty, again the temp file on the server is OK, but the download version of the file it isn't.

The difference between the 2 files is 1129 bytes, the first 1129 bytes of the file are not being downloaded. Manually setting the content-length through PHP fixes the problem, but like said I shouldn't manually set the content length.

It is curious that settings Content-Length in the PHP script is affecting what the client downloads, since mod_fastcgi replaces Content-Length when X-LIGHTTPD-send-file or X-Sendfile is used, ignoring what was sent by the PHP script.

What type of computer system is this? (desktop, raspberry pi, etc) Is this a virtual machine?
What is the filesystem of /tmp? You might temporarily try /var/tmp or some other place for the temp files.

What is the lighttpd.conf backend configured for server.network-backend?
Try: server.network-backend = "writev"

You might also enable some additional logging:
debug.log-request-header = "enable"
debug.log-response-header = "enable"
Is it possible that the client is sending a Range header? What other headers are you adding to the response? Are you telling the client the result is cacheable?

What other modules do you have enabled in lighttpd.conf?

- The server is a Dedicated server Intel Xeon E3 1245v2, 32GB ECC RAM, 2x2TB SOFT RAID. Ubuntu 14.04, kernel 3.10.23
- The PC I'm using is a Windows 10 Pro x64 laptop (no relevance I guess).
- Filesystem for /tmp/ is EXT4
- Tried different folder instead of /tmp/ same result
- I'm using: server.network-backend = "linux-sendfile", but I did test it with writev as well same result.
- No client is not sending range request.

server.modules = (
        "mod_expire",
        "mod_access",
        "mod_alias",
        "mod_compress",
        "mod_redirect",
        "mod_rewrite",
        "mod_accesslog",
        "mod_setenv",
        "mod_status",
        )

- Debug output:

2016-04-21 22:30:48: (request.c.311) fd: 9 request-len: 1070 \nGET /new/get_zip.php?id=20945716492a2c67l HTTP/1.1\r\nHost: www.mybibnumber.com\r\nConnection: keep-alive\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\nUpgrade-Insecure-Requests: 1\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36\r\nReferer: http://www.mybibnumber.com/new/my_order.php?id=20945716492a2c67l\r\nAccept-Encoding: gzip, deflate, sdch\r\nAccept-Language: en-GB,en;q=0.8,en-US;q=0.6,hu;q=0.4,ro;q=0.2\r\nCookie: visitor_id=80.47.151.12456b8d53ed0cad; fblo_953743471366672=y; PHPSESSID=7od8rr63p8spu6su6j08ur1iv4; channeloriginator=Direct_Traffic; channelcloser=Direct_Traffic; channelflow=Direct_Traffic%7Cother%7C0; utag_main=v_id:0154112b092800140fe696c9cf110506e002e0660086e$_sn:1$_ss:1$_pn:1%3Bexp-session$_st:1460578713704$ses_id:1460576913704%3Bexp-session$dc_visit:1$dc_event:1%3Bexp-session; s_cc=true; s_sq=%5B%5BB%5D%5D; s_fid=50460C5AB77819AF-2F8FA6ED5206FF0A; visitor_id=80.47.151.12456b8d53ed0cad; cart=single; _ga=GA1.2.197011593.1446641250\r\n\r\n 
2016-04-21 22:30:48: (response.c.124) Response-Header: \nHTTP/1.1 200 OK\r\nX-Powered-By: PHP/5.5.9-1ubuntu4.16\r\nContent-Disposition: attachment; filename="20945716492a2c67l.zip"\r\nContent-Encoding: gzip\r\nVary: Accept-Encoding\r\nContent-type: text/html\r\nContent-Length: 12945443\r\nDate: Thu, 21 Apr 2016 20:30:48 GMT\r\nServer: lighttpd/1.4.39\r\n\r\n

The funny thing is that the debug has the correct content length specified: 12945443, but the download file is only 12944314

I have tried IE 11, Edge, Firefox and they all download 12944314 bytes.

Other settings I'm using (tried disabling them but the problem persists)

server.max-fds = 4096
server.max-connections = 2048
server.max-keep-alive-requests = 32
server.max-keep-alive-idle = 4
server.event-handler = "linux-sysepoll" 
server.network-backend = "linux-sendfile" 
server.stat-cache-engine = "fam" 
server.use-noatime = "enable"

More info:
I have tested a curl against the link, and the CURL result is correct regardless even if I not set the content size from PHP.

I took my wife's laptop running Windows 8, and without the:

header("Content-Length: ".filesize($tmp_file.".zip"));

The download failed with network error. Adding back the header the download was completed successfully.

You're gzipping the .zip files, as indicated by the response header "Content-Encoding: gzip"

Disable mod_compress or configure it to skip .zip files. Why have you configured compress.filetype to compress .zip files?!?

No I do not have set mod_compress to compress ZIP. I might be a novice but I'm not that stupid.

My compress settings are as follows:

compress.cache-dir          = "/var/cache/lighttpd/compress/" 
compress.filetype           = ( "application/javascript", "text/css", "text/html", "text/plain" )

That being said I do have zlib.compression on in php.ini but that still shouldn't double ZIP. The only other header I have in the PHP is:

header("Content-Disposition: attachment; filename=\"$MYVARIABLE1" . ".zip\"");

Now if I disable this header, the content won't be double zipped any longer, BUT then the result stays as:

Content-Length:12945443
Content-type:text/html
Date:Fri, 22 Apr 2016 08:32:35 GMT
Server:lighttpd/1.4.39
X-Powered-By:PHP/5.5.9-1ubuntu4.16
Request Headers
view source

Which causes the file to be opened in the browser instead of being saved as a file.

Disabling zlib.compression on runtime fixed the issue, which does make sense till an extent. The only question remains if this should be the expected behaviour or not?

Also, if you reverse the order of the headers it will work as well.

With zlib.compression enabled in PHP if your header order is:

header("Content-Disposition: attachment; filename=\"$MYVARIABLE1" . ".zip\"");
header("X-LIGHTTPD-send-file: $tmp_file" . ".zip");

The Zip will be corrupted due to double zip header, but if you reverse the order like:

header("X-LIGHTTPD-send-file: $tmp_file" . ".zip");
     header("Content-Disposition: attachment; filename=\"$MYVARIABLE1" . ".zip\"");

It will work correctly. Reading the zlib.compression documentation it does make sense, the only slight problem is in cases user does not have access to PHP config and or server config, it can take forever to find the reasons, might be worth mentioning somewhere in documentation. Again it will only cause problems if you actually try to download zip, other content will be just fine.

It is not a bug I can confirm that, more like a quirks I will say.

I am glad your conscience holds you above responsibility.

Please see the following links.
http://www.techsupportalert.com/content/how-ask-question-when-you-want-technical-help.htm
http://www.catb.org/esr/faqs/smart-questions.html

The above will likely help reduce your frustration (and mine) that led to your comment

it can take forever to find the reasons,

Your issue is with your PHP. The PHP config and server config affect the results, and not all users have access to them, but your interaction with your PHP code is within your control.

If you search for "Content-Disposition" in http://php.net/manual/en/function.header.php, you may find the user-contributed comments useful. I am still uncertain that I understand exactly what you are trying to achieve, but it appears to me that you do not want PHP output compression to add Content-Encoding: gzip. Perhaps you want to set header("Content-Type: application/zip") or some other content-type such as header("Content-Type: application/octet-stream") so that PHP does not add a default Content-Type: text/html and PHP compression does not participate in responses where you use X-LIGHTTPD-send-file

Hi Sorry didn't want to sound like an ignorant person, neither frustrated and definitely had no insulting intention. The comment referred to preventing others walking blindly into the same trap I did.

Summarizing it all up, it look like:

If you have zlib.compression enabled in PHP, that will kick in on the first time you set any header except if your first header is X-Sendfile one.

When using compression in PHP the order of the headers does matter, putting X-Sendfile first will ensure that the headers are fully manipulated by Lighttpd, with the possibility to adjust some of them like "Content-Disposition" at a later stage.

However if you put any other header before X-Sendfile the zlib.compression will first try to set the header for compression, resulting in occasional broken download (like the example above).

The question remains if, X-Sendfile should overwrite this or actually PHP should take priority. To me it looks both of them want to set the ZIP header but finally neither the Web Server neither PHP sends it and the client receives the data without the zip header, albeit the content length is set correctly in all cases.

Happy if everything stays as it is, just probably either PHP Zlib, either Lighttpd documentation should mention that having this and this can result in this, as normally you wouldn't consider switching of Zlib compression if you use X-Sendfile.

Did you see my note about setting header("Content-Type: application/zip")?

See http://php.net/manual/en/zlib.configuration.php#ini.zlib.output-compression and ini_set() http://php.net/manual/en/function.ini-set.php

Read the fine manual and disable transparent output compression when you send X-Sendfile

<?php
ini_set("zlib.output_compression", "Off");
?>