Project

General

Profile

Actions

Bug #3235

closed

mod_dirlisting: embedded device unresponsive when accessing excessively large folders

Added by INS 6 months ago. Updated 5 months ago.

Status:
Missing Feedback
Priority:
Low
Category:
mod_dirlisting
Target version:
-
ASK QUESTIONS IN Forums:
No

Description

We use mod_dirlisting to list SD card content attached to an embedded device running linux. While browsing through folders containing large amount of files (10K+) lighttpd slows down significantly and sometimes even hangs completely until being restarted.

We can see that the server is busy creating the listing with high load and - until complete - does barely respond to any other requests. Things become worse when trying to access other folders while the listing is being processed. Sometimes we can get into a state where the server does not seem to complete the listing at all and hangs from there on.

OS: Linux (embedded), kernel version 4.19.91
lighttpd version: 1.4.72

Configuration:

server.modules = (
    "mod_indexfile",
    "mod_access",
    "mod_alias",
    "mod_setenv",
    "mod_redirect",
    "mod_deflate",
    "mod_dirlisting",
    "mod_fastcgi",
    "mod_proxy",
    "mod_staticfile",
    "mod_openssl",
)
mimetype.assign   += ( 
    ".jpg"  => "image/jpeg",
    ".jpeg" => "image/jpeg",
    ".png"  => "image/png",
    ".html" => "text/html",
    ".txt"  => "text/plain",
    ".js"   => "text/javascript",
    ".svg"  => "image/svg+xml",
    ".svgz" => "image/svg+xml",
    ".css"  => "text/css" 
)
index-file.names            = ( "firstpage.html" )
url.access-deny             = ( "~", ".inc" )
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
deflate.cache-dir          = "/tmp/compress/" 
deflate.allowed-encodings  = ( "br", "gzip", "deflate" )
$HTTP["url"] !~ "^/(sd)/" {
    deflate.mimetypes          = ( "text/javascript", "text/css", "text/html", "text/plain" )
}
server.http-parseopts = (
    "header-strict"           => "enable",# default
    "host-strict"             => "enable",# default
    "host-normalize"          => "enable",# default
    "url-normalize-unreserved"=> "enable",# recommended highly
    "url-normalize-required"  => "enable",# recommended
    "url-ctrls-reject"        => "enable",# recommended
    "url-path-2f-decode"      => "enable",# recommended highly (unless breaks app)
    "url-path-dotseg-remove"  => "enable"# recommended highly (unless breaks app)
)
server.document-root="/home/web" 
server.upload-dirs=("/tmp")
server.errorlog="/tmp/lighttpd.log" 
server.stream-request-body=2
server.stream-response-body=2
server.chunkqueue-chunk-sz=1024
server.port=80
ssl.openssl.ssl-conf-cmd = ("Options" => "-SessionTicket")
$HTTP["url"] =~ "^/sd($|/)" {
    dir-listing.activate = "enable" 
    dir-listing.set-footer = " " 
}
Actions #1

Updated by gstrauss 6 months ago

We use mod_dirlisting to list SD card content attached to an embedded device running linux. While browsing through folders containing large amount of files (10K+) lighttpd slows down significantly and sometimes even hangs completely until being restarted.

We can see that the server is busy creating the listing with high load and - until complete - does barely respond to any other requests. Things become worse when trying to access other folders while the listing is being processed. Sometimes we can get into a state where the server does not seem to complete the listing at all and hangs from there on.

Vague. Please read How to get support

Did this work for you with prior versions of lighttpd?
Are there any errors in the lighttpd error log?
Is CPU the bottleneck? Is disk I/O the bottleneck? Have you captured and reviewed strace or iostat or dstat (or other monitoring) output? Is /tmp filling up?
Have you tried testing with mod_deflate disabled?


Those questions above are about mod_dirlisting. However, you're on an embedded device. How frequently are files changing in this directory of 10k+ files? Have you considered generating your own firstpage.html when the contents of the directory change?

Why are you using ssl.openssl.ssl-conf-cmd = ("Options" => "-SessionTicket")? Is your embedded device that underpowered that doing so makes a difference?

Actions #2

Updated by gstrauss 6 months ago

If your device is sufficiently underpowered, you should strongly consider using mod_indexfile instead, but you also have the option of having mod_dirlisting provide unsorted json output of the directory listing.
https://example.org/path/to/dir/?json
and having the client parse the json and display however you would like it sorted.

Actions #3

Updated by gstrauss 6 months ago

Did you check if you were running out of memory on your embedded system?

Actions #4

Updated by gstrauss 6 months ago

  • Subject changed from mod_dirlisting: lighttpd becomes unresponsive when accessing folders with large amounts of files to mod_dirlisting: embedded device unresponsive when accessing excessively large folders
  • Status changed from New to Need Feedback
  • Target version deleted (1.4.xx)

If /bin/ls -f /path/to/large/folder works on the embedded device, but /bin/ls /path/to/large/folder causes the device to slow down, then the embedded device does not have sufficient resources to handle such a large folder using typical approaches. The best solution is to change the way that the files are structured to avoid such a large folder. If that is not possible, then specialized approaches to generate the directory listing are possible. If the embedded device is running out of memory, then you should try pointing lighttpd server.upload-dirs=("/tmp") to location that is not in-memory tmpfs so that lighttpd temporary files do not add to memory pressure on the embedded device. That might be sufficient to solve your problem. If it is not sufficient, then you will need specialized code (not mod_dirlisting) to use more expensive on-disk approaches to read, store and sort the directory contents, and then to generate the output. lighttpd mod_dirlisting generates a sorted HTML page for historical reasons, such as a client not enabling javascript. If you want lighttpd mod_dirlisting to stream the directory listing as unsorted, then you should have lighttpd mod_dirlisting generate a json response to a request such as https://example.org/path/to/dir/?json, and you should write your own client-side javascript to make the request for json directory listing, and then have the client javascript sort and display as desired.

Actions #5

Updated by gstrauss 5 months ago

  • Status changed from Need Feedback to Missing Feedback
  • Priority changed from Normal to Low

mod_dirlisting: embedded device unresponsive when accessing excessively large folders

Missing Feedback. OP unresponsive.

Actions

Also available in: Atom