Project

General

Profile

[Solved] Lighttpd mod_compress is creating zero length files. Why?!

Added by justaguyusinglighttpd almost 8 years ago

Is there a known issue or misconfiguration which will allow Lighttpd to create empty, zero length, files in the /var/cache/lighttpd/compress directory?

I'm seeing files like this:
bootstrap.min.css-gzip-123840-118984-1460379377 CSS-GZIP-123840-118984-1460379377-File 0

Lighttpd is serving these "empty" files, I can see an etag and content-length:0 in Chrome's developer tools; IF the files are updated\amended then lighttpd re-compresses and servers the files with content.

I'm not sure how to reproduce or create these files. We're seeing them on remote systems which have been in storage for a while.

It's on a Raspbian Jessie system, lighttpd/1.4.35.

The configuration file is:

Lighttpd.conf:

server.modules = (
"mod_access",
"mod_alias",
"mod_cgi",
"mod_compress",
"mod_redirect"
)

server.document-root = "/home/http"
server.upload-dirs = ( "/var/cache/lighttpd/uploads" )
#server.errorlog = "/var/log/lighttpd/error.log"
server.pid-file = "/var/run/lighttpd.pid"
server.username = "www-data"
server.groupname = "www-data"
server.port = 80

server.errorfile-prefix = "/home/http/errorfile-"
url.access-deny = ( "~", ".inc" )
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )

compress.cache-dir = "/var/cache/lighttpd/compress/"
compress.filetype = ( "application/javascript", "text/css", "text/html", "text/plain" )

  1. default listening port for IPv6 falls back to the IPv4 port
    include_shell "/usr/share/lighttpd/use-ipv6.pl " + server.port
    include_shell "/usr/share/lighttpd/create-mime.assign.pl"
    include_shell "/usr/share/lighttpd/include-conf-enabled.pl"

Any ideas or guidance on how to prevent this happening much appreciated.


Replies (6)

RE: Lightttpd mod_compress is creating zero length files. Why?! - Added by gstrauss almost 8 years ago

I'm trying to narrow down why this might be happening for you.

Is it possible that /var/cache/lighttpd/compress/ is (temporarily?) being filled and running out of disk space?

(This would, of course, probably be easier if this were more reproducible)
Even if you do not know the mechanism that causes this to occur, how frequently is this occurring for you?
Did you compile lighttpd yourself, and if so, would you be will to test with instrumented code?

RE: Lightttpd mod_compress is creating zero length files. Why?! - Added by justaguyusinglighttpd almost 8 years ago

Thanks for replying.

No, it's not compiled. It's installed using apt-get. The systems I'm seeing problems with have been installed by copying an image of a master, working, raspberry pi system's SD card. I only spotted the issue earlier today. The systems showing the problem are off-site being tested. I need to check whether the master image has the zero length files in the compress directory or whether they're being generated after the systems are run. So far 2-4 units are showing problems; I haven't seen the issue return after clearing our the compress directory.

I've seen disk space has been reported as an issue before. It's something I'll keep in mind, but the systems should all have plenty of free disk space.

We could test with instrumented code, however our test systems aren't yet showing the behaviour, and I wouldn't want to install it on a client's network. I guess, I need to find a way to reproduce the issue on the test systems. I'll work on it some more tomorrow and post if I manage to reproduce the problem.

RE: Lightttpd mod_compress is creating zero length files. Why?! - Added by gstrauss almost 8 years ago

I did a quick review of the mod_compress caching code and mod_compress removes the cache file if an error occurs while writing the cache file.

Now, I also noticed that the code does not check the return value from close(), for which I will submit a fix. However, if close() is failing, then that would suggest the failure is a side-effect of more serious issues, such as running out of disk space.

The current version of lighttpd in master (to be released as 1.4.40 in the next few months) contains a few fixes which make lighttpd more robust. These include configuring SA_RESTART if SIGCHLD is received for CGI, which means fewer interrupted system calls, as well as more robust handling of files which might change while being accessed. As long as the target file is updated in a temporary file and then renamed into place, the latest lighttpd code handles this caes more robustly than prior versions.

If you are able to even occassionally duplicate this on your tests systems, I recommend testing with the tip of the master branch of lighttpd to see if the problem goes away.

RE: Lighttpd mod_compress is creating zero length files. Why?! - Added by justaguyusinglighttpd almost 8 years ago

A colleague has now reproduced this on one of our test systems.

As part of our production test process we start all our units up, browse to a web application hosted by Lighhttpd on the unit, and then pull the mains power. On reboot zero length files are found in the compress directory.

Since the units are expected to be powered up 24/7 we're going to get around this by clearing out the compress directory on boot up.

It sounds like the more robust 1.4.40 may resolve this issue through it's use of temporary files. I will attempt to reproduce the issue, test with the master branch over the next few days, and report back.

Thanks for the support.

RE: Lighttpd mod_compress is creating zero length files. Why?! - Added by gstrauss almost 8 years ago

Based on your description to reproduce the issue, this is not fixed in 1.4.40. Clearing the compress directory upon reboot seems like a good idea in this case since both zero-length and partially written files might be invalid. Even were lighttpd to use temporary files when writing to cache directory (and then renaming into place), these temporary files might be partially flushed to disk and might take up disk space, eventually filling the cache directory.

RE: [Solved] Lighttpd mod_compress is creating zero length files. Why?! - Added by gstrauss about 7 years ago

lighttpd 1.4.40 and later fix issues with handling mmap on jffs2 filesystem (used on some embedded devices, likely including the one on which you run Raspbian)

    (1-6/6)