Feature #2666
closedhandle filesystems without mmap() support
Description
Hello,
I was using lighttpd v1.4.33 on an embedded arm board that works perfectly. When I upgrade to v1.4.36 I am running into problems. First was the dreaded request entity too large error when trying to upload a 30 meg file. I was able to manually patch chunk.c with the line below.
OLD
&& FILE_CHUNK != dest->last->type
NEW
&& FILE_CHUNK == dest->last->type
The patch solved the request entity error, but now am getting different errors. Below is the log file entries.
On Boot all is well.
2015-08-21 11:18:52: (log.c.164) server started
Trying to upload 30 meg file. Dies with 503 - Service Not Available
2015-08-21 11:42:41: (network_writev.c.266) toSend is negative: -524272 4623748077596442624 1048560 0 2015-08-21 11:42:41: (network_writev.c.293) write failed: Bad address 9 2015-08-21 11:42:41: (mod_fastcgi.c.2937) write failed: Bad address 14
Pressing reboot button on web interface dies with 503 - Service Not Available. All this does is send a post request that would run a system(reboot) command.
2015-08-21 11:44:01: (network_writev.c.237) mmap failed: Invalid argument /data/lighttpd-upload-rq9i9A 9 2015-08-21 11:44:01: (mod_fastcgi.c.2937) write failed: Invalid argument 22
Below is my .conf file.
# lighttpd configuration file
#
# use it as a base for lighttpd 1.0.0 and above
#
# $Id: lighttpd.conf,v 1.7 2004/11/03 22:26:05 weigon Exp $
############ Options you really have to take care of ####################
## modules to load
# at least mod_access and mod_accesslog should be loaded
# all other module should only be loaded if really neccesary
# - saves some time
# - saves memory
server.modules = (
"mod_rewrite",
# "mod_redirect",
# "mod_alias",
"mod_access",
# "mod_cml",
# "mod_trigger_b4_dl",
"mod_auth",
"mod_status",
"mod_setenv",
"mod_fastcgi",
# "mod_proxy",
# "mod_simple_vhost",
# "mod_evhost",
# "mod_userdir",
"mod_cgi",
# "mod_compress",
# "mod_ssi",
# "mod_usertrack",
# "mod_expire",
# "mod_secdownload",
# "mod_rrdtool",
"mod_accesslog" )
## debugging
# 0 for off, 1 for 'auth-ok' messages, 2 for verbose debugging
auth.debug = 0
## for htpasswd
auth.backend = "htdigest"
auth.backend.htdigest.userfile = "/etc/http_passwd"
auth.require = ( "/" =>
(
"method" => "digest",
"realm" => "root",
"require" => "valid-user"
)
)
status.status-url = "/custom/test/test.php"
server.document-root = "/usr/local/webif"
server.errorlog = "/var/log/lighttpd.log"
# files to check for if .../ is requested
index-file.names = ( "index.php", "index.html",
"index.htm", "default.htm" )
# mimetype mapping
mimetype.assign = (
".pdf" => "application/pdf",
".sig" => "application/pgp-signature",
".spl" => "application/futuresplash",
".class" => "application/octet-stream",
".ps" => "application/postscript",
".torrent" => "application/x-bittorrent",
".dvi" => "application/x-dvi",
".gz" => "application/x-gzip",
".pac" => "application/x-ns-proxy-autoconfig",
".swf" => "application/x-shockwave-flash",
".tar.gz" => "application/x-tgz",
".tgz" => "application/x-tgz",
".tar" => "application/x-tar",
".zip" => "application/zip",
".mp3" => "audio/mpeg",
".m3u" => "audio/x-mpegurl",
".wma" => "audio/x-ms-wma",
".wax" => "audio/x-ms-wax",
".ogg" => "application/ogg",
".wav" => "audio/x-wav",
".gif" => "image/gif",
".jpg" => "image/jpeg",
".jpeg" => "image/jpeg",
".png" => "image/png",
".xbm" => "image/x-xbitmap",
".xpm" => "image/x-xpixmap",
".xwd" => "image/x-xwindowdump",
".css" => "text/css",
".html" => "text/html",
".htm" => "text/html",
".js" => "text/javascript",
".asc" => "text/plain",
".c" => "text/plain",
".cpp" => "text/plain",
".log" => "text/plain",
".conf" => "text/plain",
".text" => "text/plain",
".txt" => "text/plain",
".dtd" => "text/xml",
".xml" => "text/xml",
".mpeg" => "video/mpeg",
".mpg" => "video/mpeg",
".mov" => "video/quicktime",
".qt" => "video/quicktime",
".avi" => "video/x-msvideo",
".asf" => "video/x-ms-asf",
".asx" => "video/x-ms-asf",
".wmv" => "video/x-ms-wmv",
".bz2" => "application/x-bzip",
".tbz" => "application/x-bzip-compressed-tar",
".tar.bz2" => "application/x-bzip-compressed-tar"
)
server.tag = "lighttpd"
url.access-deny = ( "~", ".inc" )
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
server.port = 80
server.pid-file = "/var/run/lighttpd.pid"
fastcgi.server = ( ".php" => ((
"bin-path" => "/usr/bin/php-cgi",
"socket" => "/tmp/php.socket",
"max-procs" => "1",
"bin-environment" => (
"PHP_FCGI_CHILDREN" => "1",
"PHP_FCGI_MAX_REQUESTS" => "500"
)
)))
cgi.assign = ( ".cgi" => "" )
#file upload
server.max-request-size = 1000000
server.network-backend = "writev"
server.upload-dirs=( "/data" )
Updated by stbuehler over 9 years ago
- Description updated (diff)
- Priority changed from Normal to High
- Target version changed from 1.4.x to 1.4.37
I have to admit this looks quite bad; the toSend is negative: -524272 4623748077596442624 1048560 0 message is hard to read because of some type mismatch in the output (it tries to read an off_t 64-bit from an 32-bit size_t parameter, resulting in 4623748077596442624; I need to fix that ofc).
Updated by stbuehler over 9 years ago
I still have no clue what the actual problem is.
I had a rewrite of the "network mmap/sendfile/write(v)" code pending for a long time anyway, so I just cleaned it up and committed it. Perhaps you could try to reproduce it with current SVN/git?
Updated by Dan over 9 years ago
Hello,
Thanks for looking into this.
Cross compiled new code from SVN
[root@RMS-200 /root]# lighttpd -v lighttpd/1.4.36-devel-3018M (ssl) - a light and fast webserver Build-Date: Aug 22 2015 12:31:58
Log file on boot.
2015-08-22 12:43:45: (log.c.164) server started
Attempting to upload 30 meg file. After a few seconds, web browser reports connection reset.
2015-08-22 12:47:20: (chunk.c.461) denying upload as opening temp-file for upload failed: Too many open files
Thanks,
Dan
Updated by Dan over 9 years ago
Small update or obsevation. On reboot, even without attempting to upload a file, the web server dishs out pages and works for the most part but pressing any button that would POST data produces 503 Internal Server Error and log file entries like below.
2015-08-22 12:54:38: (mod_cgi.c.1047) mmap failed: Invalid argument /data/lighttpd-upload-LJmYpW 10
Updated by stbuehler over 9 years ago
Dan wrote:
Hello,
Thanks for looking into this.
Cross compiled new code from SVN
> [root@RMS-200 /root]# lighttpd -v > lighttpd/1.4.36-devel-3018M (ssl) - a light and fast webserver > Build-Date: Aug 22 2015 12:31:58 >
Looks like you have some custom modifications.
Attempting to upload 30 meg file. After a few seconds, web browser reports connection reset.
> 2015-08-22 12:47:20: (chunk.c.461) denying upload as opening temp-file for upload failed: Too many open files >
Huh, this really should be fixed in that revision. Sure your modifications didn't break anything?
Small update or obsevation. On reboot, even without attempting to upload a file, the web server dishs out pages and works for the most part but pressing any button that would POST data produces 503 Internal Server Error and log file entries like below.
> 2015-08-22 12:54:38: (mod_cgi.c.1047) mmap failed: Invalid argument /data/lighttpd-upload-LJmYpW 10 >
This could mean the data length was 0; I just committed a rewrite of the mmap code in mod_cgi which should handle this case.
Could you please retest with current SVN, and either get rid of your local patch or show us the diff?
Updated by Dan over 9 years ago
Hello,
For a test I went back one version to 1.4.35 and this works perfectly. I can upload large files and on my web interface the buttons that send POST data work as expected.
I am using buildroot to cross compile the lighttpd web server for ARM. Since version 1.4.26 all I have needed to do is bump the lighttpd version number in the buildroot make file and then re-compile.
I downloaded the 3028 svn and ran ./autogen.sh and then let buildroot recompile it.
[root@RMS-200 /root]# lighttpd -v lighttpd/1.4.36-devel-3028M (ssl) - a light and fast webserver Build-Date: Aug 23 2015 14:02:11
Without trying an upload, clicking on a button on my web interface causes the browser to say 503 Internal Server Error. Logfile output is below.
2015-08-23 14:13:47: (log.c.164) server started 2015-08-23 14:22:45: (mod_cgi.c.1047) mmap failed: Invalid argument /data/lighttpd-upload-Pfbb6X 7
In my config file there is reference to /data folder.
#file upload server.max-request-size = 1000000 server.network-backend = "writev" server.upload-dirs=( "/data" )
Why would the error message in the log file be referencing the data folder at all since an upload was not tried, only clicking a button?
Thanks,
Dan
Updated by stbuehler over 9 years ago
clicking a button is very likely a POST request, which is an upload - just not uploading a file, only some form data.
Updated by stbuehler over 9 years ago
You need to fix your setup; you claim to be on svn 3028 (I still don't know whether you have any modifications, or whether svn just adds "M" for untracked files...), but your mod_cgi output clearly shows that this is not the case.
Please make sure you don't have old modules lying around.
Updated by Dan over 9 years ago
Here's what I have been doing. Delete entire lighttpd-1.4.36 folder. Downloaded the svn and run ./autogen.sh. Rebuild package. Stop lighttpd on target. Manually copy new binaries and all so's to target. Fire up light server. This procedure works with all previous versions. I will try it again 1 step at a time to double make sure.
Updated by Dan over 9 years ago
So I did the whole procedure again. Emptied the contents of the buildroot/lighttpd-1.4.36 folder. I even deleted all .so files in the /usr/lib/lighttpd/ folder and the binaries in /usr/sbin on the target board. Rebuilt the svn again by first using ./autogen.sh. Copied the snv files to the buildroot/lighttpd-1.4.36 folder that I previously emptied. Typed make from the top level buildroot folder. No errors. Copied lighttpd and angel binaries from light src folder to /usr/sbin on target. chmod binaries. Copied all .so files from src/.libs to /usr/lib/lighttpd/ on the target board. Started webserver manually by lighttpd -f /etc/lighttpd/lighttpd.conf and did a lighttpd -v. Output below. The only thing I skipped this time was to strip the binaries to save 40K.
[root@RMS-200 /]# lighttpd -v lighttpd/1.4.36-devel-3028M (ssl) - a light and fast webserver Build-Date: Aug 24 2015 20:22:54
Refresh web browser on main page works. Clicking a web button gives 500 error and the logfile output below.
2015-08-24 20:47:05: (log.c.164) server started 2015-08-24 20:47:08: (mod_cgi.c.1261) cleaning up CGI: process died with signal 11 2015-08-24 20:47:18: (mod_cgi.c.1047) mmap failed: Invalid argument /data/lighttpd-upload-TPwyRO 9
If I am doing something wrong, I don't know what it is. I have done this same procedure from version 1.4.26 all the way up to 1.4.35 and it works everytime. My build environment has not changed. Some more info in case it helps. The target board uses plain vanila linux with uClibc v0.9.30.1 and kernel 2.6.33. Php v5.3.29.
I was using light 1.4.33 previously, so going up two version numbers is a win for now.
Thanks,
Dan
Updated by stbuehler over 9 years ago
2015-08-24 20:47:05: (log.c.164) server started 2015-08-24 20:47:08: (mod_cgi.c.1261) cleaning up CGI: process died with signal 11 2015-08-24 20:47:18: (mod_cgi.c.1047) mmap failed: Invalid argument /data/lighttpd-upload-TPwyRO 9
source:branches/lighttpd-1.4.x/src/log.c@3028#L164
source:branches/lighttpd-1.4.x/src/mod_cgi.c@3028#L1261
source:branches/lighttpd-1.4.x/src/mod_cgi.c@3028#L1047
None of these three line numbers are matching the source revision. They should be 194 for "server started", 1329 for "cleaning up CGI" and 773 for "mmap failed" (which should also include more details).
I just did a fresh svn checkout:
$ svn checkout svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x lighttpd-1.4.x-svn $ cd lighttpd-1.4.x-svn $ ./autogen.sh $ mkdir build $ cd build $ ../configure '--with-pic' '--with-mysql' '--with-ldap' '--with-attr' '--with-openssl' '--with-pcre' '--with-zlib' '--with-bzip2' '--with-fam' '--with-webdav-props' '--with-webdav-locks' '--with-gdbm' '--with-memcache' '--with-lua' '--prefix=/usr' '--enable-extra-warnings' '--with-libev' '--with-libunwind' '--with-valgrind' $ make -j4 $ ./src/lighttpd -v lighttpd/1.4.37-devel-3028 (ssl) - a light and fast webserver Build-Date: Aug 25 2015 07:04:57
So you really should see a 1.4.37-devel* version here. I really don't know what you are doing... but my best guess is your svn checkout is broken.
Updated by Dan about 9 years ago
Agreed, there must be something wrong with my svn. Can you tar up your folder right after you run the ./autogen.sh and make it available for download. I would like to try building from that point.
Thanks,
Dan
Updated by stbuehler about 9 years ago
http://debian.lighttpd.net/lighttpd1.4-nightlies/pool/main/l/lighttpd/ at the bottom has the latest tar.xz. (the ".orig" filename part is just for the debian packaging, but the contents are the "upstream" tar.xz package)
Updated by Dan about 9 years ago
Hello,
So I downloaded lighttpd_1.4.37~snap-20150823-130438-r3028.orig.tar.xz. Then unxz, then untar. Copy to build environment. Make. All builds without errors. Copy binaries and .so's to the target.
[root@RMS-200 /root]# lighttpd -v lighttpd/1.4.37 (ssl) - a light and fast webserver Build-Date: Aug 25 2015 15:00:31
Click button on web interface, get 500 error.
2015-08-25 15:04:28: (log.c.194) server started 2015-08-25 15:04:54: (mod_cgi.c.773) mmap failed: Invalid argument /data/lighttpd-upload-RyrUFJ 9 0 32
The version does not say devel as it should, so I don't know what I am doing wrong. I don't want you to fight with this anymore. I left a little something in your PayPal tip jar for your trouble.
Thanks,
Dan
Updated by stbuehler about 9 years ago
- Priority changed from High to Normal
Don't worry so much :)
The tar.xz doesn't have the svn revision number, so "1.4.37" is perfectly normal here.
I think the mmap call itself looks perfectly valid (start at offset 0, length 32, file descriptor 9, ...), so maybe the system/kernel/filesystem/apparmor/.. whatever doesn't like mmap.
I think it might be reasonably to fall back to a simple read() (perhaps dynamic, perhaps as a configure option) - I'm not giving up yet :)
I hope we got at least rid of some of the more worrying log messages including large/negative numbers and the Bad address messages, and will downgrade the issue to "Normal" again.
Thanks for tipping, and thanks for responding to my requests.
Updated by Dan about 9 years ago
You are very welcome. Let me know when you have another nightly snapshot I can try. No rush.
Thanks,
Dan
Updated by stbuehler about 9 years ago
- Target version changed from 1.4.37 to 1.4.38
Updated by stbuehler about 9 years ago
I figured out that there is one "small" change since 1.4.35 - lighty tries to always cache request bodies on disk, and before it didn't cache request bodies smaller than 64k.
I fixed this in r3046, so small request bodies won't be cached on disk anymore, and this should workaround the mmap() issue some people have been seeing for small requests (and I think it was always broken for large uploads if the filesystem doesn't support mmap).
Updated by stbuehler about 9 years ago
- Has duplicate Bug #2677: lighttpd-1.4.37:mmap() call fails in handling POST request added
Updated by stbuehler about 9 years ago
- Subject changed from Trying to upload 30 meg file. Dies with 503 - Service Not Available to handle filesystems without mmap() support
- Target version changed from 1.4.38 to 1.4.x
Updated by gstrauss almost 9 years ago
This issue might be the same as in https://redmine.lighttpd.net/issues/2715 for which a patch has been committed to trunk. JFFS2 flash file system does not support mmap PROT_READ MAP_SHARED, so mod_cgi was changed to use mmap PROT_READ MAP_PRIVATE on the temporary file. mmap PROT_READ MAP_PRIVATE does appear to work on JFFS2.
Updated by stbuehler almost 9 years ago
while #2715 might solve problems with some file systems, it certainly is not a generic fix for missing mmap() support.
Updated by gstrauss almost 9 years ago
Agreed, the title "handle filesystems without mmap() support" is not addressed generically by https://redmine.lighttpd.net/issues/2715
However, the problem reported in the error messages do appear to be the same EINVAL "Invalid argument" issue that appeared with mod_cgi with the JFFS2 flash filesystem not supporting MAP_SHARED.
Also, the network backends should not use mmap() unless that backend is specifically requested in the configuration, which would be ill-advised on a system that does not support mmap(). :)
Updated by gstrauss over 8 years ago
- Tracker changed from Bug to Feature
Updated by gstrauss over 8 years ago
- Status changed from New to Patch Pending
- Target version changed from 1.4.x to 1.4.40
Submitted pull request https://github.com/lighttpd/lighttpd1.4/pull/57
Note that this pull request does not touch network_write_mmap.c. Admins should not select this optional backend if serving files from a filesystem which does not support mmap().
Updated by gstrauss over 8 years ago
- Status changed from Patch Pending to Fixed
Also available in: Atom