Project

General

Profile

Feature #2666

handle filesystems without mmap() support

Added by Dan over 1 year ago. Updated 12 months ago.

Status:
Fixed
Priority:
Normal
Assignee:
-
Category:
core
Target version:
Start date:
2015-08-21
Due date:
% Done:

0%

Missing in 1.5.x:

Description

Hello,

I was using lighttpd v1.4.33 on an embedded arm board that works perfectly. When I upgrade to v1.4.36 I am running into problems. First was the dreaded request entity too large error when trying to upload a 30 meg file. I was able to manually patch chunk.c with the line below.

OLD

&& FILE_CHUNK != dest->last->type

NEW

&& FILE_CHUNK == dest->last->type

The patch solved the request entity error, but now am getting different errors. Below is the log file entries.

On Boot all is well.

2015-08-21 11:18:52: (log.c.164) server started 

Trying to upload 30 meg file. Dies with 503 - Service Not Available

2015-08-21 11:42:41: (network_writev.c.266) toSend is negative: -524272 4623748077596442624 1048560 0 
2015-08-21 11:42:41: (network_writev.c.293) write failed: Bad address 9 
2015-08-21 11:42:41: (mod_fastcgi.c.2937) write failed: Bad address 14 

Pressing reboot button on web interface dies with 503 - Service Not Available. All this does is send a post request that would run a system(reboot) command.

2015-08-21 11:44:01: (network_writev.c.237) mmap failed: Invalid argument /data/lighttpd-upload-rq9i9A 9 
2015-08-21 11:44:01: (mod_fastcgi.c.2937) write failed: Invalid argument 22 

Below is my .conf file.

# lighttpd configuration file
#
# use it as a base for lighttpd 1.0.0 and above
#
# $Id: lighttpd.conf,v 1.7 2004/11/03 22:26:05 weigon Exp $

############ Options you really have to take care of ####################

## modules to load
# at least mod_access and mod_accesslog should be loaded
# all other module should only be loaded if really neccesary
# - saves some time
# - saves memory
server.modules              = (
                                "mod_rewrite",
#                               "mod_redirect",
#                               "mod_alias",
                                "mod_access",
#                               "mod_cml",
#                               "mod_trigger_b4_dl",
                                "mod_auth",
                                "mod_status",
                                "mod_setenv",
                                "mod_fastcgi",
#                               "mod_proxy",
#                               "mod_simple_vhost",
#                               "mod_evhost",
#                               "mod_userdir",
                                "mod_cgi",
#                               "mod_compress",
#                               "mod_ssi",
#                               "mod_usertrack",
#                               "mod_expire",
#                               "mod_secdownload",
#                               "mod_rrdtool",
                                "mod_accesslog" )

## debugging
# 0 for off, 1 for 'auth-ok' messages, 2 for verbose debugging
auth.debug                 = 0

## for htpasswd
auth.backend               = "htdigest" 
auth.backend.htdigest.userfile = "/etc/http_passwd" 

auth.require = ( "/" =>
 (
  "method" => "digest",
  "realm" => "root",
  "require" => "valid-user" 
 )
)

status.status-url = "/custom/test/test.php" 

server.document-root        = "/usr/local/webif" 

server.errorlog             = "/var/log/lighttpd.log" 

# files to check for if .../ is requested
index-file.names            = ( "index.php", "index.html",
                                "index.htm", "default.htm" )

# mimetype mapping
mimetype.assign             = (
  ".pdf"          =>      "application/pdf",
  ".sig"          =>      "application/pgp-signature",
  ".spl"          =>      "application/futuresplash",
  ".class"        =>      "application/octet-stream",
  ".ps"           =>      "application/postscript",
  ".torrent"      =>      "application/x-bittorrent",
  ".dvi"          =>      "application/x-dvi",
  ".gz"           =>      "application/x-gzip",
  ".pac"          =>      "application/x-ns-proxy-autoconfig",
  ".swf"          =>      "application/x-shockwave-flash",
  ".tar.gz"       =>      "application/x-tgz",
  ".tgz"          =>      "application/x-tgz",
  ".tar"          =>      "application/x-tar",
  ".zip"          =>      "application/zip",
  ".mp3"          =>      "audio/mpeg",
  ".m3u"          =>      "audio/x-mpegurl",
  ".wma"          =>      "audio/x-ms-wma",
  ".wax"          =>      "audio/x-ms-wax",
  ".ogg"          =>      "application/ogg",
  ".wav"          =>      "audio/x-wav",
  ".gif"          =>      "image/gif",
  ".jpg"          =>      "image/jpeg",
  ".jpeg"         =>      "image/jpeg",
  ".png"          =>      "image/png",
  ".xbm"          =>      "image/x-xbitmap",
  ".xpm"          =>      "image/x-xpixmap",
  ".xwd"          =>      "image/x-xwindowdump",
  ".css"          =>      "text/css",
  ".html"         =>      "text/html",
  ".htm"          =>      "text/html",
  ".js"           =>      "text/javascript",
  ".asc"          =>      "text/plain",
  ".c"            =>      "text/plain",
  ".cpp"          =>      "text/plain",
  ".log"          =>      "text/plain",
  ".conf"         =>      "text/plain",
  ".text"         =>      "text/plain",
  ".txt"          =>      "text/plain",
  ".dtd"          =>      "text/xml",
  ".xml"          =>      "text/xml",
  ".mpeg"         =>      "video/mpeg",
  ".mpg"          =>      "video/mpeg",
  ".mov"          =>      "video/quicktime",
  ".qt"           =>      "video/quicktime",
  ".avi"          =>      "video/x-msvideo",
  ".asf"          =>      "video/x-ms-asf",
  ".asx"          =>      "video/x-ms-asf",
  ".wmv"          =>      "video/x-ms-wmv",
  ".bz2"          =>      "application/x-bzip",
  ".tbz"          =>      "application/x-bzip-compressed-tar",
  ".tar.bz2"      =>      "application/x-bzip-compressed-tar" 
 )

server.tag                 = "lighttpd" 

url.access-deny             = ( "~", ".inc" )

static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )

server.port                = 80

server.pid-file            = "/var/run/lighttpd.pid" 

fastcgi.server = ( ".php" => ((
                     "bin-path" => "/usr/bin/php-cgi",
                     "socket" => "/tmp/php.socket",
                     "max-procs" => "1",
                     "bin-environment" => (
                         "PHP_FCGI_CHILDREN" => "1",
                         "PHP_FCGI_MAX_REQUESTS" => "500" 
                         )
                 )))

cgi.assign = ( ".cgi" => "" )

#file upload
server.max-request-size = 1000000
server.network-backend = "writev" 
server.upload-dirs=( "/data" )


Related issues

Duplicated by Bug #2677: lighttpd-1.4.37:mmap() call fails in handling POST request Duplicate 2015-10-06

Associated revisions

Revision 3b6fd58f (diff)
Added by gstrauss 12 months ago

[mod_webdav] lseek,read if fs can not mmap (#2666, fixes #962)

For uploaded files or other request body, fall back to
lseek(),read() if filesystem does not support mmap()

(mmap(), if supported, is utilized regardless of --enable-mmap
since request body is either in memory or stored in temporary
files controlled by lighttpd)

x-ref:
"WebDAV upload-> mmap failed: operation not permitted"
https://redmine.lighttpd.net/issues/962
"handle filesystems without mmap() support"
https://redmine.lighttpd.net/issues/2666

github: closes #55

Revision c9b56735 (diff)
Added by gstrauss 12 months ago

[mod_compress] use mmap and trap SIGBUS (#2666, fixes #1879)

use mmap and trap SIGBUS in mod_compress
(if lighttpd build with --enable-mmap)

mod_compress has not used mmap since Feb 2012 (see #2391)

x-ref:
"Lighttpd 1.4.20 Crash (SIGBUS in mod_compress)"
https://redmine.lighttpd.net/issues/1879
"Crash SIGBUS"
https://redmine.lighttpd.net/issues/2391
"handle filesystems without mmap() support"
https://redmine.lighttpd.net/issues/2666

github: closes #56

History

#1 Updated by stbuehler over 1 year ago

  • Description updated (diff)
  • Priority changed from Normal to High
  • Target version changed from 1.4.x to 1.4.37

I have to admit this looks quite bad; the toSend is negative: -524272 4623748077596442624 1048560 0 message is hard to read because of some type mismatch in the output (it tries to read an off_t 64-bit from an 32-bit size_t parameter, resulting in 4623748077596442624; I need to fix that ofc).

#2 Updated by stbuehler over 1 year ago

I still have no clue what the actual problem is.

I had a rewrite of the "network mmap/sendfile/write(v)" code pending for a long time anyway, so I just cleaned it up and committed it. Perhaps you could try to reproduce it with current SVN/git?

#3 Updated by Dan over 1 year ago

Hello,

Thanks for looking into this.

Cross compiled new code from SVN

[root@RMS-200 /root]# lighttpd -v
lighttpd/1.4.36-devel-3018M (ssl) - a light and fast webserver
Build-Date: Aug 22 2015 12:31:58

Log file on boot.

2015-08-22 12:43:45: (log.c.164) server started 

Attempting to upload 30 meg file. After a few seconds, web browser reports connection reset.

2015-08-22 12:47:20: (chunk.c.461) denying upload as opening temp-file for upload failed: Too many open files

Thanks,
Dan

#4 Updated by Dan over 1 year ago

Small update or obsevation. On reboot, even without attempting to upload a file, the web server dishs out pages and works for the most part but pressing any button that would POST data produces 503 Internal Server Error and log file entries like below.

2015-08-22 12:54:38: (mod_cgi.c.1047) mmap failed:  Invalid argument /data/lighttpd-upload-LJmYpW 10 

#5 Updated by stbuehler over 1 year ago

Dan wrote:

Hello,

Thanks for looking into this.

Cross compiled new code from SVN

> [root@RMS-200 /root]# lighttpd -v
> lighttpd/1.4.36-devel-3018M (ssl) - a light and fast webserver
> Build-Date: Aug 22 2015 12:31:58
> 

Looks like you have some custom modifications.

Attempting to upload 30 meg file. After a few seconds, web browser reports connection reset.

> 2015-08-22 12:47:20: (chunk.c.461) denying upload as opening temp-file for upload failed: Too many open files
> 

Huh, this really should be fixed in that revision. Sure your modifications didn't break anything?

Small update or obsevation. On reboot, even without attempting to upload a file, the web server dishs out pages and works for the most part but pressing any button that would POST data produces 503 Internal Server Error and log file entries like below.

> 2015-08-22 12:54:38: (mod_cgi.c.1047) mmap failed:  Invalid argument /data/lighttpd-upload-LJmYpW 10
> 

This could mean the data length was 0; I just committed a rewrite of the mmap code in mod_cgi which should handle this case.

Could you please retest with current SVN, and either get rid of your local patch or show us the diff?

#6 Updated by Dan over 1 year ago

Hello,

For a test I went back one version to 1.4.35 and this works perfectly. I can upload large files and on my web interface the buttons that send POST data work as expected.

I am using buildroot to cross compile the lighttpd web server for ARM. Since version 1.4.26 all I have needed to do is bump the lighttpd version number in the buildroot make file and then re-compile.

I downloaded the 3028 svn and ran ./autogen.sh and then let buildroot recompile it.

[root@RMS-200 /root]# lighttpd -v
lighttpd/1.4.36-devel-3028M (ssl) - a light and fast webserver
Build-Date: Aug 23 2015 14:02:11

Without trying an upload, clicking on a button on my web interface causes the browser to say 503 Internal Server Error. Logfile output is below.

2015-08-23 14:13:47: (log.c.164) server started 
2015-08-23 14:22:45: (mod_cgi.c.1047) mmap failed:  Invalid argument /data/lighttpd-upload-Pfbb6X 7 

In my config file there is reference to /data folder.

#file upload
server.max-request-size = 1000000
server.network-backend = "writev" 
server.upload-dirs=( "/data" )

Why would the error message in the log file be referencing the data folder at all since an upload was not tried, only clicking a button?

Thanks,
Dan

#7 Updated by stbuehler over 1 year ago

clicking a button is very likely a POST request, which is an upload - just not uploading a file, only some form data.

#8 Updated by Dan over 1 year ago

Ok, that makes sense.

#9 Updated by stbuehler over 1 year ago

You need to fix your setup; you claim to be on svn 3028 (I still don't know whether you have any modifications, or whether svn just adds "M" for untracked files...), but your mod_cgi output clearly shows that this is not the case.

Please make sure you don't have old modules lying around.

#10 Updated by Dan over 1 year ago

Here's what I have been doing. Delete entire lighttpd-1.4.36 folder. Downloaded the svn and run ./autogen.sh. Rebuild package. Stop lighttpd on target. Manually copy new binaries and all so's to target. Fire up light server. This procedure works with all previous versions. I will try it again 1 step at a time to double make sure.

#11 Updated by Dan over 1 year ago

So I did the whole procedure again. Emptied the contents of the buildroot/lighttpd-1.4.36 folder. I even deleted all .so files in the /usr/lib/lighttpd/ folder and the binaries in /usr/sbin on the target board. Rebuilt the svn again by first using ./autogen.sh. Copied the snv files to the buildroot/lighttpd-1.4.36 folder that I previously emptied. Typed make from the top level buildroot folder. No errors. Copied lighttpd and angel binaries from light src folder to /usr/sbin on target. chmod binaries. Copied all .so files from src/.libs to /usr/lib/lighttpd/ on the target board. Started webserver manually by lighttpd -f /etc/lighttpd/lighttpd.conf and did a lighttpd -v. Output below. The only thing I skipped this time was to strip the binaries to save 40K.

[root@RMS-200 /]# lighttpd -v
lighttpd/1.4.36-devel-3028M (ssl) - a light and fast webserver
Build-Date: Aug 24 2015 20:22:54

Refresh web browser on main page works. Clicking a web button gives 500 error and the logfile output below.

2015-08-24 20:47:05: (log.c.164) server started 
2015-08-24 20:47:08: (mod_cgi.c.1261) cleaning up CGI: process died with signal 11 
2015-08-24 20:47:18: (mod_cgi.c.1047) mmap failed:  Invalid argument /data/lighttpd-upload-TPwyRO 9

If I am doing something wrong, I don't know what it is. I have done this same procedure from version 1.4.26 all the way up to 1.4.35 and it works everytime. My build environment has not changed. Some more info in case it helps. The target board uses plain vanila linux with uClibc v0.9.30.1 and kernel 2.6.33. Php v5.3.29.

I was using light 1.4.33 previously, so going up two version numbers is a win for now.

Thanks,
Dan

#12 Updated by stbuehler over 1 year ago

2015-08-24 20:47:05: (log.c.164) server started 
2015-08-24 20:47:08: (mod_cgi.c.1261) cleaning up CGI: process died with signal 11 
2015-08-24 20:47:18: (mod_cgi.c.1047) mmap failed:  Invalid argument /data/lighttpd-upload-TPwyRO 9

source:branches/lighttpd-1.4.x/src/log.c@3028#L164
source:branches/lighttpd-1.4.x/src/mod_cgi.c@3028#L1261
source:branches/lighttpd-1.4.x/src/mod_cgi.c@3028#L1047

None of these three line numbers are matching the source revision. They should be 194 for "server started", 1329 for "cleaning up CGI" and 773 for "mmap failed" (which should also include more details).

I just did a fresh svn checkout:

$ svn checkout svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x lighttpd-1.4.x-svn
$ cd lighttpd-1.4.x-svn
$ ./autogen.sh
$ mkdir build
$ cd build
$ ../configure '--with-pic' '--with-mysql' '--with-ldap' '--with-attr' '--with-openssl' '--with-pcre' '--with-zlib' '--with-bzip2' '--with-fam' '--with-webdav-props' '--with-webdav-locks' '--with-gdbm' '--with-memcache' '--with-lua' '--prefix=/usr' '--enable-extra-warnings' '--with-libev' '--with-libunwind' '--with-valgrind'
$ make -j4
$ ./src/lighttpd -v
lighttpd/1.4.37-devel-3028 (ssl) - a light and fast webserver
Build-Date: Aug 25 2015 07:04:57

So you really should see a 1.4.37-devel* version here. I really don't know what you are doing... but my best guess is your svn checkout is broken.

#13 Updated by Dan over 1 year ago

Agreed, there must be something wrong with my svn. Can you tar up your folder right after you run the ./autogen.sh and make it available for download. I would like to try building from that point.

Thanks,
Dan

#14 Updated by stbuehler over 1 year ago

http://debian.lighttpd.net/lighttpd1.4-nightlies/pool/main/l/lighttpd/ at the bottom has the latest tar.xz. (the ".orig" filename part is just for the debian packaging, but the contents are the "upstream" tar.xz package)

#15 Updated by Dan over 1 year ago

Hello,

So I downloaded lighttpd_1.4.37~snap-20150823-130438-r3028.orig.tar.xz. Then unxz, then untar. Copy to build environment. Make. All builds without errors. Copy binaries and .so's to the target.

[root@RMS-200 /root]# lighttpd -v
lighttpd/1.4.37 (ssl) - a light and fast webserver
Build-Date: Aug 25 2015 15:00:31

Click button on web interface, get 500 error.

2015-08-25 15:04:28: (log.c.194) server started 
2015-08-25 15:04:54: (mod_cgi.c.773) mmap failed: Invalid argument /data/lighttpd-upload-RyrUFJ 9 0 32

The version does not say devel as it should, so I don't know what I am doing wrong. I don't want you to fight with this anymore. I left a little something in your PayPal tip jar for your trouble.

Thanks,
Dan

#16 Updated by stbuehler over 1 year ago

  • Priority changed from High to Normal

Don't worry so much :)

The tar.xz doesn't have the svn revision number, so "1.4.37" is perfectly normal here.

I think the mmap call itself looks perfectly valid (start at offset 0, length 32, file descriptor 9, ...), so maybe the system/kernel/filesystem/apparmor/.. whatever doesn't like mmap.

I think it might be reasonably to fall back to a simple read() (perhaps dynamic, perhaps as a configure option) - I'm not giving up yet :)

I hope we got at least rid of some of the more worrying log messages including large/negative numbers and the Bad address messages, and will downgrade the issue to "Normal" again.

Thanks for tipping, and thanks for responding to my requests.

#17 Updated by Dan over 1 year ago

You are very welcome. Let me know when you have another nightly snapshot I can try. No rush.

Thanks,
Dan

#18 Updated by stbuehler over 1 year ago

  • Target version changed from 1.4.37 to 1.4.38

#19 Updated by stbuehler over 1 year ago

I figured out that there is one "small" change since 1.4.35 - lighty tries to always cache request bodies on disk, and before it didn't cache request bodies smaller than 64k.

I fixed this in r3046, so small request bodies won't be cached on disk anymore, and this should workaround the mmap() issue some people have been seeing for small requests (and I think it was always broken for large uploads if the filesystem doesn't support mmap).

#20 Updated by stbuehler over 1 year ago

  • Duplicated by Bug #2677: lighttpd-1.4.37:mmap() call fails in handling POST request added

#21 Updated by stbuehler over 1 year ago

  • Subject changed from Trying to upload 30 meg file. Dies with 503 - Service Not Available to handle filesystems without mmap() support
  • Target version changed from 1.4.38 to 1.4.x

#22 Updated by gstrauss about 1 year ago

This issue might be the same as in https://redmine.lighttpd.net/issues/2715 for which a patch has been committed to trunk. JFFS2 flash file system does not support mmap PROT_READ MAP_SHARED, so mod_cgi was changed to use mmap PROT_READ MAP_PRIVATE on the temporary file. mmap PROT_READ MAP_PRIVATE does appear to work on JFFS2.

#23 Updated by stbuehler about 1 year ago

while #2715 might solve problems with some file systems, it certainly is not a generic fix for missing mmap() support.

#24 Updated by gstrauss about 1 year ago

Agreed, the title "handle filesystems without mmap() support" is not addressed generically by https://redmine.lighttpd.net/issues/2715

However, the problem reported in the error messages do appear to be the same EINVAL "Invalid argument" issue that appeared with mod_cgi with the JFFS2 flash filesystem not supporting MAP_SHARED.

Also, the network backends should not use mmap() unless that backend is specifically requested in the configuration, which would be ill-advised on a system that does not support mmap(). :)

#25 Updated by gstrauss about 1 year ago

  • Tracker changed from Bug to Feature

#26 Updated by gstrauss about 1 year ago

  • Status changed from New to Patch Pending
  • Target version changed from 1.4.x to 1.4.40

Submitted pull request https://github.com/lighttpd/lighttpd1.4/pull/57

Note that this pull request does not touch network_write_mmap.c. Admins should not select this optional backend if serving files from a filesystem which does not support mmap().

#27 Updated by gstrauss 12 months ago

  • Category set to core

#28 Updated by gstrauss 12 months ago

  • Status changed from Patch Pending to Fixed

Also available in: Atom