AbsoLUAtion - The powerful combo of lighttpd + Lua¶
- Table of contents
- AbsoLUAtion - The powerful combo of lighttpd + Lua
- Requirements
- Links
- Code-Snippets
- Content Negotiation
- Fight DDoS
- Mod_Security
- security responses
- Time-to-First-Byte
- Client-Cert HTTP header field
- request limiter
- other solutions
- Sample Files
We want to build a central resource for lighttpd + Lua as this is one of the biggest advantages lighttpd has over other webservers. It's useful, handy, simple and sometimes a quite powerful combo which gives you some additional flexibility and offers you solutions to small and big problems other httpds can't solve!
Again, we hope that you, the users of lighttpd, support this page by contributing links, code-snippets or simply offer your lua scripts with small descriptions of what they do and how it helps lighttpd to do stuff you want it to do.
Requirements¶
- lighttpd mod_magnet
- Lua (v5.1; lighttpd 1.4.40+ should also support v5.2, v5.3, v5.4) http://www.lua.org
Links¶
- http://www.sitepoint.com/blogs/2007/04/10/faster-page-loads-bundle-your-css-and-javascript/
- WP-MultiUser http://www.bisente.com/blog/2007/04/08/lighttpd-wordpressmu-english/
- Dynamically generate thumbnails and cache them http://www.xarg.org/2010/04/dynamic-thumbnail-generation-on-the-static-server/
- Drupal/OpenAtrium simple cleanurl solution drupal-clean-url-lighttpd
- Authentication through openid and the likes https://github.com/chmduquesne/lighttpd-external-auth (blog post)
Dead links? You don´t like to be listed here? Please remove it. Thanks!
Code-Snippets¶
Apache .htaccess alternatives
Options for migrating .htaccess functionality to lighttpd
A common idiom in Apache .htaccess:RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php?q=$1 [L,QSA]
- url.rewrite-if-not-file (recommended if !-f is sufficient; does not check !-d)
- lua mod_rewrite (recommended; simpler than option below)
- mod_magnet rewrite paths without restarting request (below)
As lighttpd doesn't provide this is_file/is_dir check out of the box, again mod_magnet comes into play. I took the example for drupal from darix site.
Lets assume drupal is already installed under http://example.com/drupal/ you now add the magnet part to it.
$HTTP["url"] =~ "^/drupal" { # we only need index.php here. index-file.names = ( "index.php" ) # for clean urls magnet.attract-physical-path-to = ( "/etc/lighttpd/drupal.lua" ) }
The drupal.lua:
-- little helper function function file_exists(path) return lighty.stat(path) and true or false end function removePrefix(str, prefix) return str:sub(1,#prefix+1) == prefix.."/" and str:sub(#prefix+2) end -- prefix without the trailing slash local prefix = '/drupal' -- the magic ;) if (not file_exists(lighty.env["physical.path"])) then -- file still missing. pass it to the fastcgi backend request_uri = removePrefix(lighty.env["uri.path"], prefix) if request_uri then lighty.env["uri.path"] = prefix .. "/index.php" local uriquery = lighty.env["uri.query"] or "" lighty.env["uri.query"] = uriquery .. (uriquery ~= "" and "&" or "") .. "q=" .. request_uri lighty.env["physical.rel-path"] = lighty.env["uri.path"] lighty.env["request.orig-uri"] = lighty.env["request.uri"] lighty.env["physical.path"] = lighty.env["physical.doc-root"] .. lighty.env["physical.rel-path"] end end -- fallthrough will put it back into the lighttpd request loop -- that means we get the 304 handling for free. ;)
Overwrite default mime-type/content-type
Add "magnet.attract-physical-path-to = ( "/path-to/change-ctype.lua" )" to lighttpd.conf and save the following as "change-ctype.lua"
if (string.match(lighty.env["physical.rel-path"], ".swf")) then lighty.header["Content-Type"] = "text/html" end
Sending text-files as HTML
This is a bit simplistic, but it illustrates the idea: Take a text-file and cover it in a "< pre >" tag.
Config-file
magnet.attract-physical-path-to = (server.docroot + "/readme.lua")
readme.lua
lighty.content = { "<pre>", { filename = "/README" }, "</pre>" } lighty.header["Content-Type"] = "text/html" return 200
Redirect map
redirect-map.lua high performance redirect-map based on url-path
Simple maintenance script
You need three files, maint.up, maint.down and maint.html.
maint.html holds a simple html-page of what you want to display to your users while in maintenance-mode.
Add "magnet.attract-physical-path-to = ( "/path-to-your/maint.lua" )" to your lighttpd.conf, best is global section or within a host-section of your config, e.g. a board/forum/wiki you know a maintenance-mode is needed from time to time. If you want to switch to maintenance-mode, just copy maint.down to maint.lua in your "/path-to-your/" location, and lighttpd will display your maint.html to all users - without restarting anything - this can be done on-the-fly. Work is done and all is up again? Copy maint.up to maint.lua in your "/path-to-your/" location. Whats maint.up doing? Nothing, just going on with normal file serving :-)
maint.up - all is up, user will see normal pages
-- This is empty, nothing to do.
maint.down - lighttpd will show the maintenance page -> maint.html
-- lighty.header["X-Maintenance-Mode"] = "1" -- uncomment the above if you want to add the header lighty.content = { { filename = "/path-to-your/maint.html" } } lighty.header["Content-Type"] = "text/html" return 503 -- or return 200 if you want
custom error pages
mod_magnet can be used to implement any or all of the lighttpd directives: server.error-handler
, server.error-handler-404
, server.errorfile-prefix
, server.error-intercept
magnet.attract-response-start-to = ( "/absolute/path/to/script.lua" )
-- intercept all HTTP status errors (e.g. server.error-intercept = "enable") local r = lighty.r local http_status = r.req_item.http_status if (http_status < 400) -- not an HTTP status error; do not modify return 0 end -- send back custom error page (similar to server.errorfile-prefix) -- replace existing response body, if any local errfile = "/path/to/errfiles/" .. http_status .. ".html" local st = lighty.c.stat(errfile) if (not st) then errfile = "/path/to/errfiles/generic.html" fi r.resp_body:set({ { filename = errfile } }) -- (alternative: construct custom error page (similar to server.error-handler)) return http_status
selecting a random file from a directory
Say, you want to send a random file (ad-content) from a directory.
To simplify the code and to improve the performance we define:
- all images have the same format (e.g. image/png)
- all images use increasing numbers starting from 1
- a special index-file names the highest number
Config
server.modules += ( "mod_magnet" ) magnet.attract-physical-path-to = ("random.lua")
random.lua
dir = lighty.env["physical.path"] f = assert(io.open(dir .. "/index", "r")) maxndx = f:read("*all") f:close() ndx = math.random(maxndx) lighty.content = { { filename = dir .. "/" .. ndx }} lighty.header["Content-Type"] = "image/png" return 200
denying illegal character sequences in the URL
Instead of implementing mod_security, you might just want to apply filters on the content and deny special sequences that look like SQL injection.
A common injection is using UNION to extend a query with another SELECT query.
if (string.find(lighty.env["request.uri"], "UNION%s")) then return 400 end
Traffic Quotas
If you only allow your virtual hosts a certain amount for traffic each month and want to disable them if the traffic is reached, perhaps this helps:
host_blacklist = { ["www.example.org"] = 0 } if (host_blacklist[lighty.request["Host"]]) then return 404 end
Just add the hosts you want to blacklist into the blacklist table in the shown way.
Complex rewrites
If you want to implement caching on your document-root and only want to regenerate content if the requested file doesn't exist, you can attract the physical.path:
magnet.attract-physical-path-to = ( server.document-root + "/rewrite.lua" )
rewrite.lua
attr = lighty.stat(lighty.env["physical.path"]) if (not attr) then -- we couldn't stat() the file for some reason -- let the backend generate it lighty.env["uri.path"] = "/dispatch.fcgi" lighty.env["physical.rel-path"] = lighty.env["uri.path"] lighty.env["physical.path"] = lighty.env["physical.doc-root"] .. lighty.env["physical.rel-path"] end
Extension rewrites
If you want to hide your file extensions (like .php) you can attract the physical.path:
magnet.attract-physical-path-to = ( server.document-root + "/rewrite.lua" )
rewrite.lua
attr = lighty.stat(lighty.env["physical.path"] .. ".php") if (attr) then lighty.env["uri.path"] = lighty.env["uri.path"] .. ".php" lighty.env["physical.rel-path"] = lighty.env["uri.path"] lighty.env["physical.path"] = lighty.env["physical.doc-root"] .. lighty.env["physical.rel-path"] end
User tracking
... or how to store data globally in the script-context:
Each script has its own script-context. When the script is started it only contains the lua-functions and the special lighty.* name-space. If you want to save data between script runs, you can use the global-script context:
if (nil == _G["usertrack"]) then _G["usertrack"] = {} end if (nil == _G["usertrack"][lighty.request["Cookie"]]) then _G["usertrack"][lighty.request["Cookie"]] else _G["usertrack"][lighty.request["Cookie"]] = _G["usertrack"][lighty.request["Cookie"]] + 1 end print _G["usertrack"][lighty.request["Cookie"]]
The global-context is per script. If you update the script without restarting the server, the context will still be maintained.
WordpressMU
wpmu.lua
if (not lighty.stat(lighty.env["physical.path"])) then if (string.match(lighty.env["uri.path"], "^(/?[^/]*/)files/$")) then lighty.env["physical.rel-path"] = "index.php" else n, a = string.match(lighty.env["uri.path"], "^(/?[^/]*/)files/(.+)") if a then lighty.env["physical.rel-path"] = "wp-content/blogs.php" lighty.env["uri.query"] = "file=" .. a else n, a = string.match(lighty.env["uri.path"], "^(/[^/]*)/(wp-.*)") if a then lighty.env["physical.rel-path"] = a; else n, a = string.match(lighty.env["uri.path"], "^(/[^/]*)/(.*\.php)$") if a then lighty.env["physical.rel-path"] = a else lighty.env["physical.rel-path"] = "index.php" end end end end lighty.env["physical.path"] = lighty.env["physical.doc-root"] .. "/".. lighty.env["physical.rel-path"] end
Content Negotiation¶
content-negotiation.lua to parse Accept-Language and Accept-Encoding (#2678, #2736) to determine best target file
content-negotiation.lua
Related, see #1259 for lua code to try multiple extensions (a la Apache mod_autoext) to find target file
Fight DDoS¶
If your Server is under high load because of someone is flooding you with requests, a little bit lua might help you. ;) In our case we've got a lot of requests without a User-Agent in the request header.
if ( lighty.request["User-Agent"]== nil ) then file = io.open ("ips.txt","a") file:write(lighty.env["request.remote-ip"]) file:write("\n") file:close() return 200 end
The field request.remote-ip is available since lighttpd 1.4.23. The file ips.txt must be writeable by the lighttpd user (www-data). The bad guys in the ips.txt file can be dropped into the firewall with a little shell script.
lua can also be used to access a database and reject requests based on data in the database.
See sample reject-bad-actors.lua attached in Files section at the bottom of this page. It uses an mcdb constant database for fast lookups.
Mod_Security¶
Apache has mod_security available as a WAF (web application firewall) however this isn't available for other webservers. I've written a quick and dirty script to perform a similar task to mod_security using mod_magnet
lighttpd-mod_security-via-mod_magnet
I've been recently working on libmodsecurity binding for openresty which is a nginx+luajit combo. Using patched libmodsecurity and cffi-lua along with modsec.lua and mod_magnet lighttpd can perform incoming request inspection. Currently there is no way to perform request body (POST) and response inspection. Two last would involve additional lua entry points in lighty. Security rules could be obtained at coreruleset.
waf.lua for lighttpd:
local modsec = require "modsec" local ok, err = modsec.init("/etc/owasp/modsec.conf") if not ok then print(err) return end local transaction = modsec.transaction() if not transaction then print("Failed to initialize transaction") end -- evaluate connection info and request headers local req_attr = lighty.r.req_attr local url = req_attr["uri.scheme"] .. "://" .. req_attr["uri.authority"] .. req_attr["uri.path-raw"] .. (req_attr["uri.query"] and ("?" .. req_attr["uri.query"]) or "") local res, err = transaction:eval_connection(req_attr["request.remote-addr"],req_attr["request.remote-port"], req_attr["uri.authority"],req_attr["request.server-port"],url, req_attr["request.method"],req_attr["request.protocol"]) if err then print("Failed to evaluate connection: ",err) end local res, err = transaction:eval_request_headers(lighty.r.req_header) if err then print("Failed to evaluate request headers: ",err) end --[[ evaluate request body Currently no way to evaluate request body but this function must be run even with nil as arguments ]] local res, err = transaction:eval_request_body(nil,nil) if err then print("Failed to evaluate request body: ",err) end -- Here decision could be made upon modsecurity variables whether handle this request or not local score = tonumber(transaction.var.tx.anomaly_score) if score >= 8 then print("This request looks nasty overall score is: "..score) return 403 end
Example of owasp/modsec.conf
#This is libmodsecurity base configuration Include modsecurity.conf Include /opt/openresty/owasp/crs-setup.conf Include /opt/openresty/owasp/rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf Include /opt/openresty/owasp/rules/REQUEST-901-INITIALIZATION.conf Include /opt/openresty/owasp/rules/REQUEST-905-COMMON-EXCEPTIONS.conf Include /opt/openresty/owasp/rules/REQUEST-910-IP-REPUTATION.conf Include /opt/openresty/owasp/rules/REQUEST-911-METHOD-ENFORCEMENT.conf Include /opt/openresty/owasp/rules/REQUEST-912-DOS-PROTECTION.conf Include /opt/openresty/owasp/rules/REQUEST-913-SCANNER-DETECTION.conf Include /opt/openresty/owasp/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf Include /opt/openresty/owasp/rules/REQUEST-921-PROTOCOL-ATTACK.conf Include /opt/openresty/owasp/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf Include /opt/openresty/owasp/rules/REQUEST-931-APPLICATION-ATTACK-RFI.conf Include /opt/openresty/owasp/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf Include /opt/openresty/owasp/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf Include /opt/openresty/owasp/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf Include /opt/openresty/owasp/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf Include /opt/openresty/owasp/rules/REQUEST-943-APPLICATION-ATTACK-SESSION-FIXATION.conf Include /opt/openresty/owasp/rules/REQUEST-949-BLOCKING-EVALUATION.conf Include /opt/openresty/owasp/rules/RESPONSE-950-DATA-LEAKAGES.conf Include /opt/openresty/owasp/rules/RESPONSE-951-DATA-LEAKAGES-SQL.conf Include /opt/openresty/owasp/rules/RESPONSE-952-DATA-LEAKAGES-JAVA.conf Include /opt/openresty/owasp/rules/RESPONSE-953-DATA-LEAKAGES-PHP.conf Include /opt/openresty/owasp/rules/RESPONSE-954-DATA-LEAKAGES-IIS.conf Include /opt/openresty/owasp/rules/RESPONSE-959-BLOCKING-EVALUATION.conf Include /opt/openresty/owasp/rules/RESPONSE-980-CORRELATION.conf Include /opt/openresty/owasp/rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf
security responses¶
lua scripts can be deployed quickly as a security response in order to mitigate exploits against CGI and other backends.
-- reject request if request-line URL or request headers contain literal '{' -- -- If not expecting literal '{' in HTTP requests, then this heuristic to -- reject '{' thwarts certain classes of attacks on servers behind lighttpd. -- - Log4Shell CVE-2021-44228 (and related) (Apache log4j RCE) -- - Shellshock CVE-2014-6271 (and related) (Bash env RCE) -- local r = lighty.r -- check url-path (url-decoded) for literal '{' if (string.find(r.req_attr["uri.path"], "{")) then return 403 end -- check query-string (url-encoded) for literal '{' local q = r.req_attr["uri.query"] if q and (string.find(q, "{")) then return 403 end -- check request headers for literal '{' local req_header = r.req_header for _, v in pairs(req_header) do if (string.find(v, "{")) then return 403 end end return 0
Time-to-First-Byte¶
-- Time-to-First-Byte (TTFB) -- (requires lighttpd 1.4.65+) function elapsed_time(r) local start_sec, start_nsec = r.req_item.start_time() local elapsed_sec, elapsed_nsec = lighty.c.hrtime() elapsed_sec = elapsed_sec - start_sec elapsed_nsec = elapsed_nsec - start_nsec if (elapsed_nsec < 0) then elapsed_nsec = elapsed_nsec + 1000000000 elapsed_sec = elapsed_sec - 1 end return elapsed_sec, elapsed_nsec end -- save time-to-first-byte (ttfb) time in env for later use in logging -- (e.g. accesslog.format = "%h %V %u %t \"%r\" %>s %b %{ttfb}e %D") local r = lighty.r local elapsed_sec, elapsed_nsec = elapsed_time(r) r.req_env["ttfb"] = math.floor(elapsed_sec*1000000 + elapsed_nsec/1000) -- usecs -- (might use math.tointeger() in lua 5.3+ instead of math.floor())
Client-Cert HTTP header field¶
-- Client-Cert HTTP Header Field -- -- https://github.com/httpwg/http-extensions/blob/main/draft-ietf-httpbis-client-cert-field.md -- https://www.ietf.org/archive/id/draft-ietf-httpbis-client-cert-field-01.html -- -- (note: Client-Cert-Chain not implemented in lighttpd) -- -- This code intended for use with TLS client certificate verification -- for requests forwarded by lighttpd mod_proxy to other HTTP backends -- ssl.verifyclient.activate = "enable" -- ssl.verifyclient.exportcert = "enable" -- convert CGI environment variable SSL_CLIENT_CERT -- to structured field Client-Cert request header local r = lighty.r local cert = r.req_env["SSL_CLIENT_CERT"] if (cert) then cert = string.gsub(string.gsub(cert, "[-][^\n]+[-]", ":"), "%s", "") end -- set Client-Cert request header if lighttpd verified client certificate; -- unset Client-Cert request header if lighttpd did not verify client cert -- (to ignore Client-Cert request header if supplied by malicious client) r.req_header["Client-Cert"] = cert
request limiter¶
-- lua request limiter -- (requires lighttpd 1.4.65+) -- -- In this example, for a given path or set of paths (determined after -- request headers have been received), count number of total requests -- and number of requests matching the current client remote address. -- -- Adapt this to your site and your application needs. local count_limit = 8 -- max 8 total requests at a time to given paths local raddr_limit = 2 -- max 2 requests at a time per remote addr local addr = lighty.r.req_attr["request.remote-addr"] local ireq_attr, count, raddr = nil, 0, 0 for i in lighty.server.irequests() do ireq_attr = i.req_attr local path = ireq_attr["uri.path"] -- example: interested only in "^/$" and "\.php$" paths if (path == "/" or string.match(path, "%.php$")) then -- save desired info; i invalid outside lighty.server.irequests iteration count = count + 1 if (count > count_limit) then break end if (ireq_attr["request.remote-addr"] == addr) then raddr = raddr + 1 if (raddr > raddr_limit) then break end end end end -- check if counts exceed policy if (count > count_limit or raddr > raddr_limit) then -- (optional) logging --print(addr .. ' turned away. Too many connections.') -- 503 Service Unavailable lighty.r.resp_header["Retry-After"] = "2" return 503 -- Alternatives: -- could set Location response header to alternate site and send 302 redirect -- could rewrite the request to a static page for /site-busy.html -- (and be sure to also disable caching, e.g. Cache-Control: max-age=0) -- could reject requests with 403 Forbidden end -- Limitations: -- -- The above lua scans all active requests for matching remote address, but only -- sees the current lighttpd process. This may miss requests when there are -- multiple lighttpd workers or independent servers. -- -- For high-traffic sites, a more efficient solution may be to implement -- the request rate limiter in the backend daemon servicing the requests. -- Requests can be accepted and entered into backend queue up to a limit before -- turning away new requests. A fixed number of threads can service the queue.
other solutions¶
external-static
I´ve seen this nice solution somewhere where they host some files locally on their machines. If popularity gets to high, files are too big or for whatever reasons the files are moved to i think it was amazon´s S3 or akamai for faster serving or to cope with high traffic. You still can use your hostname, urls, collect stats from your logs - your users are just redirected with a 302 to the files they ask for.
2008-11-17: Found the source: presto-move-content-to-s3-with-no-code-changes
Request -> check for local copy -> 302 (if not stored locally) -> let users download from a big pipe
Add the following to your lighttpd.conf:
$HTTP["url"] =~ "^/static/[^/]+[.]gif([?].*)?$" { #match the files you want this to work for magnet.attract-physical-path-to = ( "/path-to-your/external-static.lua" ) }
Save the following to external-static.lua:
local filename = lighty.env["physical.path"] local stat = lighty.stat( filename ) if not stat then local static_name = string.match( filename, "static/([^/]+)$" ) lighty.header["Location"] = "http://<new-location-with-big-pipes>/" .. static_name return 302 end
Sample Files¶
Updated by gstrauss almost 2 years ago · 55 revisions