Project

General

Profile

Bug #2715

mmap error while uploading file in 1.4.39

Added by Dan over 1 year ago. Updated over 1 year ago.

Status:
Fixed
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
2016-02-08
Due date:
% Done:

100%

Missing in 1.5.x:

Description

Hello,

I am getting the following error while trying to upload a 78K file on a home rolled buildroot linux distro.

2016-02-08 10:19:26: (log.c.194) server started 
2016-02-08 10:19:58: (mod_cgi.c.780) mmap failed: Invalid argument /data/lighttpd-upload-StCx4S 8 0 80192

Lighttpd version:

[root@RMS-300 /root]# lighttpd -v
lighttpd/1.4.39 (ssl) - a light and fast webserver
Build-Date: Jan  3 2016 17:50:32

Some relevant config I am using.

#file upload
server.max-request-size = 1000000
server.network-backend = "writev" 
server.upload-dirs=( "/data" )

Please forgive me if I have not provided enough details. Everything else works as expected.

Thanks,
Dan Pattison

Associated revisions

Revision 3075 (diff)
Added by stbuehler over 1 year ago

[mod_cgi] use MAP_PRIVATE to mmap temporary file instead of MAP_SHARED (fixes #2715)

Flash filesystem JFFS2 does not support mmap PROT_READ MAP_SHARED,
though it does support mmap PROT_READ MAP_PRIVATE

Although MAP_SHARED is preferred, CGI input body is fully collected
prior to handler invoking the CGI, so the temporary file is never
modified after being mapped. Since the request input body is specific
to request and is temporary file, mmap PROT_READ MAP_PRIVATE works fine.

From: Glenn Strauss <>

Revision 3fd80ff8 (diff)
Added by gstrauss over 1 year ago

[mod_cgi] use MAP_PRIVATE to mmap temporary file instead of MAP_SHARED (fixes #2715)

Flash filesystem JFFS2 does not support mmap PROT_READ MAP_SHARED,
though it does support mmap PROT_READ MAP_PRIVATE

Although MAP_SHARED is preferred, CGI input body is fully collected
prior to handler invoking the CGI, so the temporary file is never
modified after being mapped. Since the request input body is specific
to request and is temporary file, mmap PROT_READ MAP_PRIVATE works fine.

From: Glenn Strauss <>

git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@3075 152afb58-edef-0310-8abb-c4023f1b3aa9

History

#1 Updated by gstrauss over 1 year ago

The mmap() man page on Linux states that mmap() will return EINVAL for one of the following:

       EINVAL We don't like addr, length, or offset (e.g., they are too large,
              or not aligned on a page boundary).

       EINVAL (since Linux 2.6.12) length was 0.

       EINVAL flags  contained neither MAP_PRIVATE or MAP_SHARED, or contained
              both of these values.

Since you mentioned that you have a custom build ("home rolled buildroot linux distro"), I would recommend the first place to check is config.h at the top of the lighttpd source tree. If HAVE_SYS_MMAN_H is not defined, then lighttpd src/sys-mmap.h will #define MAP_SHARED 0, which will result in EINVAL from the mmap() call, since a non-zero value of MAP_PRIVATE or MAP_SHARED was not passed to mmap().

Then again, src/mod_cgi.c does not #include "sys-mmap.h" -- inconsistency, sigh -- so maybe my suggestion is a red herring. You might still choose to temporarily modify line 780 of mod_cgi.c to print out PROT_READ and MAP_SHARED values, since the other values are already included in the error message you provided above.

    log_error_write(srv, __FILE__, __LINE__, "ssbdoodd", "mmap failed:",
                    strerror(errno), c->file.name, c->file.fd, c->file.mmap.offset, (off_t) c->file.mmap.length,
                    PROT_READ, MAP_SHARED);

#2 Updated by Dan over 1 year ago

Hello,

Thank you for looking into this.

In config.h I have:

#define HAVE_SYS_MMAN_H 1

I added the extra items (PROT_READ and MAP_SHARED) to the error message on line 780 as suggested. I also added the text -new so I could tell for sure my changes made it into the mod_cgi.so file. I was expecting to see something added to the end of the output. Below is a snippet from the log file, it is the same as before, except for my -new text.

2016-02-09 11:01:46: (log.c.194) server started 
2016-02-09 11:09:30: (mod_cgi.c.780) mmap failed-new: Invalid argument /data/lighttpd-upload-3j2k3x 8 0 80192 

Let me know if I should try something else.

Thank you,
Dan Pattison

#3 Updated by patrickdk over 1 year ago

Did you forget to add dd to the "ssbdoodd" string, if so, those extra params are just ignored.

#4 Updated by Dan over 1 year ago

Hello,

Yes, I did not notice that (shame). Below is another snippet from the log file.

2016-02-09 12:10:43: (log.c.194) server started 
2016-02-09 12:11:15: (mod_cgi.c.780) mmap failed-newest: Invalid argument /data/lighttpd-upload-YXMNQo 8 0 80186 1 1 

Thank you,
Dan Pattison

#5 Updated by gstrauss over 1 year ago

Those values look correct. The offset must be page-aligned, and 0 is page-aligned. The values for PROT_READ and MAP_SHARED are both 1 on Linux. File-size is non-zero. The fd is 8, and should have failed a few lines above in network_open_file_chunk() if it failed to open the file. Something is weird since those arguments all look correct.

Try adding this directly above the call to mmap() on line 779 in mod_cgi.c:

    struct stat st;
    if (0 != fstat(c->file.fd, &st))
       log_error_write(srv, __FILE__, __LINE__, "ssbd", "fstat failed:",
                       strerror(errno), c->file.name, c->file.fd);
    errno = ENOSYS;  /* something to see if the library function mmap() is actually being called */

As an alternative, would you strace the server pid and post the strace output line for the failing mmap() call?

#6 Updated by gstrauss over 1 year ago

If the fstat() succceeds, then it is probably worth looking at what fstat() reports as the current file size:

    struct stat st;
    if (0 != fstat(c->file.fd, &st))
       log_error_write(srv, __FILE__, __LINE__, "ssbd", "fstat failed:",
                       strerror(errno), c->file.name, c->file.fd);
    log_error_write(srv, __FILE__, __LINE__, "so", "fstat succeeded.  CGI input file size:", st.st_size);
    errno = ENOSYS;  /* something to see if the library function mmap() is actually being called */

#7 Updated by stbuehler over 1 year ago

Although the mmap man page says the error code would be ENODEV instead of EINVAL I think it's is possible the underlying filesystem doesn't support mmap; and mod_cgi wouldn't handle it anyway.

The (temp) files sent to the cgi backend shouldn't get smaller - they are owned by lighttpd.

#8 Updated by Dan over 1 year ago

Hello,

I added that extra code, but it did not fire.

2016-02-09 12:58:49: (log.c.194) server started 
2016-02-09 13:00:53: (mod_cgi.c.786) mmap failed-newest1: Invalid argument /data/lighttpd-upload-wOgiCO 8 0 80186 1 1 

More info if it helps

[root@RMS-300 /root]# uname -a
Linux RMS-300 2.6.33 #47 Tue Sep 8 23:45:56 PDT 2015 armv5tejl GNU/Linux

Thanks,

#9 Updated by Dan over 1 year ago

Hello,

Newest output.

2016-02-09 13:32:46: (log.c.194) server started 
2016-02-09 13:33:39: (mod_cgi.c.783) fstat succeeded.  CGI input file size: 80189 
2016-02-09 13:33:39: (mod_cgi.c.787) mmap failed-newest2: Invalid argument /data/lighttpd-upload-I3JN6J 8 0 80189 1 1 

Thanks,

#10 Updated by gstrauss over 1 year ago

stbuehler is probably on to something with his comment about the filesystem supporting mmap().

Would you try changing the following in your lighttpd.conf:

  server.upload-dirs = "/dev/shm" 

#11 Updated by Dan over 1 year ago

Hello,

OK, tried that. The webserver will not start with that config file change.

Feb  9 13:51:46 RMS-300 respawnd[773]: Child pid 896 exited with status 255. 
Feb  9 13:51:49 RMS-300 respawnd[773]: Child pid 897 exited with status 255. 
Feb  9 13:51:52 RMS-300 respawnd[773]: Child pid 898 exited with status 255. 
Feb  9 13:51:55 RMS-300 respawnd[773]: Child pid 899 exited with status 255. 
Feb  9 13:51:58 RMS-300 respawnd[773]: Child pid 900 exited with status 255. 
Feb  9 13:52:01 RMS-300 respawnd[773]: Child pid 908 exited with status 255. 
Feb  9 13:52:04 RMS-300 respawnd[773]: Child pid 910 exited with status 255. 

Thanks,

#12 Updated by gstrauss over 1 year ago

What is the current value of server.upload-dirs? "/data" ? Try:

    server.upload-dirs = "/tmp" 
or
    server.upload-dirs = "/var/tmp" 

Of course, make sure the target directory exists.

#13 Updated by gstrauss over 1 year ago

Of course, make sure the target directory exists and that the user the server is running under (e.g. 'lighttpd') has write access to that location.

#14 Updated by Dan over 1 year ago

Hello,

I figured out what the issue was.

2016-02-09 14:00:55: (configfile-glue.c.75) server.upload-dirs should have been a array of strings like ... = ( "..." ) 

Fixed the above error, and now it works.

Web output from my custom board.

RMS-300 Database Restore Operation
Recieved FILENAME: rms300(1).db
SIZE: 79872 bytes

Saving File...
File saved successfully as /tmp/rms300.db.new
Validating uploaded file....

Current Database Version: 2
Imported Database Version: 2

873 831 [OK]

Stopping Dependant Services...
  Stopping RMSD...
  Stopping snmpD...
Stopping Dependant Services...[OK]

Move new Sqlite3 database... [OK]

Restarting Dependant Services...
  Starting RMSD...
  Starting snmpD...
Restarting Dependant Services...[OK]

Database Restore Operation Successfull!

I need this to work at /data. Is there some file permission or something that I do not have set right? Dev and Data are both set the same though.

drwxr-xr-x    6 root     root            0 Dec 31  1969 data
drwxr-xr-x    8 root     root            0 Feb  9 13:53 dev

Thanks,

#15 Updated by gstrauss over 1 year ago

Sorry that I gave you incorrect syntax.

What are the underlying filesystem types of /data versus /dev/shm?

    df -h /dev/shm /data
    mount | egrep "^(/data|/dev/shm)" 

#16 Updated by gstrauss over 1 year ago

Also, would you check those permissions on the directories? /data and /dev/shm
Note: /dev/shm is a different mount from that of /dev.

What is the output of the following?

    ls -ld /data/ /dev/shm/

What is the user that lighttpd is running as?

A non-root user won't be able to write to /data with these permissions:
drwxr-xr-x 6 root root 0 Dec 31 1969 data
drwxr-xr-x 8 root root 0 Feb 9 13:53 dev

#17 Updated by Dan over 1 year ago

Hello,

Lighttpd is running as user root on an embedded arm board. /data is in a separate flash partition.

# /etc/fstab: static file system information.
#
# <file system>    <mount pt>    <type>    <options>               <dump> <pass>
/dev/mtdblock3    /data        jffs2    rw,noatime            0    0
proc        /proc        proc    defaults            0    0
devpts        /dev/pts    devpts    defaults,gid=5,mode=620        0    0
tmpfs        /tmp        tmpfs    defaults            0    0
tmpfs        /var        tmpfs    defaults            0    0
sysfs        /sys        sysfs    defaults            0    0
usbfs        /proc/bus/usb    usbfs    defaults    
[root@RMS-300 /dev]# ls -ld /data/ /dev/shm/
drwxr-xr-x    6 root     root            0 Dec 31  1969 /data/
drwxr-xr-x    2 root     root            0 Feb  9 14:02 /dev/shm/

Thanks,

#18 Updated by gstrauss over 1 year ago

https://sourceware.org/bugzilla/show_bug.cgi?id=5033 notes that JFFS2 does not support mmap() MAP_SHARED. Perhaps you could try MAP_PRIVATE instead of MAP_SHARED? (Sorry, I don't have a system on which to try this myself.)

Internet searches turn up warnings about wearing out flash more quickly if using mmap() for writable maps on flash file systems, but the use here in mod_cgi is read-only so if it works with MAP_PRIVATE, that should be fine, as the file is a temporary file specific to the request.

https://lkml.org/lkml/2015/9/3/66 also suggests a possible solution using MAP_PRIVATE with perf.

If this works for you, perhaps mod_cgi.c can be modified to use MAP_PRIVATE.

#19 Updated by Dan over 1 year ago

Hello,

MAP_PRIVATE fixed it. Previous versions of lighttpd worked for me but I notice it was using 0_RDONLY then and not MAP_SHARED. My update program was broken in 1.4.39 but worked in 1.4.35. I can patch future versions using the hack-by-hand method unless you make that a permenant change.

I want to thank you guys (gstrauss and stbuehler) for helping me with this. I believe you both have PHDs in awesomicity! I left a little something in your PayPal tip jar for the effort.

Thank you!
Dan Pattison

#20 Updated by gstrauss over 1 year ago

Glad to hear it works!

I submitted a pull request with a patch: https://github.com/lighttpd/lighttpd1.4/pull/16

#21 Updated by stbuehler over 1 year ago

  • Status changed from New to Fixed
  • % Done changed from 0 to 100

Applied in changeset r3075.

Also available in: Atom