Project

General

Profile

AbsoLUAtion » History » Revision 44

Revision 43 (prauat, 2021-11-03 21:29) → Revision 44/54 (gstrauss, 2021-11-04 05:01)

h1. AbsoLUAtion - The powerful combo of lighttpd + Lua 

 {{>toc}} 

 We want to build a central resource for lighttpd + Lua as this is one of the biggest advantages lighttpd has over other webservers. It's useful, handy, simple and sometimes a quite powerful combo which gives you some additional flexibility and offers you solutions to small and big problems other httpds can't solve! 

 Again, we hope that you, the users of lighttpd, support this page by contributing links, code-snippets or simply offer your lua scripts with small descriptions of what they do and how it helps lighttpd to do stuff you want it to do. 


 h1. Requirements 

 * lighttpd [[lighttpd:Docs_ModMagnet|mod_magnet]] 
 * Lua (v5.1; lighttpd 1.4.40+ should also support v5.2, v5.3, v5.4) http://www.lua.org 

 h1. Links 

 * [[ModMagnetExamples#lua-examples-of-lighttpd-modules|lua examples of lighttpd modules]] 

 * http://www.sitepoint.com/blogs/2007/04/10/faster-page-loads-bundle-your-css-and-javascript/ 
 * WP-MultiUser http://www.bisente.com/blog/2007/04/08/lighttpd-wordpressmu-english/ 
 * Dynamically generate thumbnails and cache them http://www.xarg.org/2010/04/dynamic-thumbnail-generation-on-the-static-server/ 
 * Drupal/OpenAtrium simple cleanurl solution "drupal-clean-url-lighttpd":https://web.archive.org/web/20100820100050/http://sudhaker.com:80/web-development/drupal/drupal-clean-url-lighttpd.html 
 * Authentication through openid and the likes https://github.com/chmduquesne/lighttpd-external-auth ("blog post":http://blog.chmd.fr/using-openid-and-the-likes-to-protect-static-content-lighttpd.html) 

 Dead links? You don´t like to be listed here? Please remove it. Thanks! 

 h1. Code-Snippets 

 *The is_file/is_dir dilemma* 

 Known from Apache´s .htaccess: 
 Hint: see [[lighttpd:Docs_ModRewrite#urlrewrite-repeat-if-not-file|url.rewrite-if-not-file]] for the !-f part 

 <pre> 
 RewriteCond %{REQUEST_FILENAME} !-f 
 RewriteCond %{REQUEST_FILENAME} !-d 
 RewriteRule ^(.*)$ index.php?q=$1 [L,QSA] 
 </pre> 

 As lighttpd doesn't provide this is_file/is_dir check out of the box, again mod_magnet comes into play.    I took the example for drupal from darix site. 

 Lets assume drupal is already installed under http://example.com/drupal/ you now add the magnet part to it. 

 <pre> 
 $HTTP["url"] =~ "^/drupal" { 
     # we only need index.php here. 
     index-file.names = ( "index.php" ) 
     # for clean urls 
     magnet.attract-physical-path-to = ( "/etc/lighttpd/drupal.lua" ) 
 } 
 </pre> 

 The drupal.lua: 

 <pre> 
 -- little helper function 
 function file_exists(path) 
   return lighty.stat(path) and true or false 
 end 
 function removePrefix(str, prefix) 
   return str:sub(1,#prefix+1) == prefix.."/" and str:sub(#prefix+2) 
 end 

 -- prefix without the trailing slash 
 local prefix = '/drupal' 

 -- the magic ;) 
 if (not file_exists(lighty.env["physical.path"])) then 
     -- file still missing. pass it to the fastcgi backend 
     request_uri = removePrefix(lighty.env["uri.path"], prefix) 
     if request_uri then 
       lighty.env["uri.path"]            = prefix .. "/index.php" 
       local uriquery = lighty.env["uri.query"] or "" 
       lighty.env["uri.query"] = uriquery .. (uriquery ~= "" and "&" or "") .. "q=" .. request_uri 
       lighty.env["physical.rel-path"] = lighty.env["uri.path"] 
       lighty.env["request.orig-uri"]    = lighty.env["request.uri"] 
       lighty.env["physical.path"]       = lighty.env["physical.doc-root"] .. lighty.env["physical.rel-path"] 
     end 
 end 
 -- fallthrough will put it back into the lighttpd request loop 
 -- that means we get the 304 handling for free. ;) 
 </pre> 


 *Overwrite default mime-type/content-type* 

 Add "magnet.attract-physical-path-to = ( "/path-to/change-ctype.lua" )" to lighttpd.conf and save the following as "change-ctype.lua" 
 <pre> 
  if (string.match(lighty.env["physical.rel-path"], ".swf")) then 
     lighty.header["Content-Type"] = "text/html" 
 </pre> 


 *Sending text-files as HTML* 

 This is a bit simplistic, but it illustrates the idea: Take a text-file and cover it in a "< pre >" tag. 

 Config-file 

 <pre> 
   magnet.attract-physical-path-to = (server.docroot + "/readme.lua") 
 </pre> 

 readme.lua 

 <pre> 
   lighty.content = { "<pre>", { filename = "/README" }, "</pre>" } 
   lighty.header["Content-Type"] = "text/html" 
  
   return 200 
 </pre> 


 *Redirect map* 

 "redirect-map.lua":https://redmine.lighttpd.net/attachments/2065 redirect-map based on url-path 


 *Simple maintenance script* 

 You need three files, maint.up, maint.down and maint.html. 
 maint.html holds a simple html-page of what you want to display to your users while in maintenance-mode. 

 Add "magnet.attract-physical-path-to = ( "/path-to-your/maint.lua" )" to your lighttpd.conf, best is global section or within a host-section of your config, e.g. a board/forum/wiki you know a maintenance-mode is needed from time to time. If you want to switch to maintenance-mode, just copy maint.down to maint.lua in your "/path-to-your/" location, and lighttpd will display your maint.html to all users - without restarting anything - this can be done on-the-fly. Work is done and all is up again? Copy maint.up to maint.lua in your "/path-to-your/" location. Whats maint.up doing? Nothing, just going on with normal file serving :-) 

 maint.up - all is up, user will see normal pages 

 <pre> 
 -- This is empty, nothing to do. 
 </pre> 

 maint.down - lighttpd will show the maintenance page -> maint.html 

 <pre> 
 -- lighty.header["X-Maintenance-Mode"] = "1" 
 -- uncomment the above if you want to add the header 
 lighty.content = { { filename = "/path-to-your/maint.html" } } 
 lighty.header["Content-Type"] = "text/html" 
 return 503 
 -- or return 200 if you want 
 </pre> 


 *selecting a random file from a directory* 

 Say, you want to send a random file (ad-content) from a directory.  

 To simplify the code and to improve the performance we define: 

 * all images have the same format (e.g. image/png) 
 * all images use increasing numbers starting from 1 
 * a special index-file names the highest number 

 Config 

 <pre> 
   server.modules += ( "mod_magnet" ) 
   magnet.attract-physical-path-to = ("random.lua") 
 </pre> 

 random.lua 

 <pre> 
   dir = lighty.env["physical.path"] 

   f = assert(io.open(dir .. "/index", "r")) 
   maxndx = f:read("*all") 
   f:close() 

   ndx = math.random(maxndx) 

   lighty.content = { { filename = dir .. "/" .. ndx }} 
   lighty.header["Content-Type"] = "image/png" 

   return 200   
 </pre> 


 *denying illegal character sequences in the URL* 

 Instead of implementing mod_security, you might just want to apply filters on the content and deny special sequences that look like SQL injection.  

 A common injection is using UNION to extend a query with another SELECT query. 

 <pre> 
   if (string.find(lighty.env["request.uri"], "UNION%s")) then 
     return 400 
   end 
 </pre> 




 *Traffic Quotas* 

 If you only allow your virtual hosts a certain amount for traffic each month and want to disable them if the traffic is reached, perhaps this helps: 

 <pre> 
   host_blacklist = { ["www.example.org"] = 0 } 

   if (host_blacklist[lighty.request["Host"]]) then 
     return 404 
   end 
 </pre> 

 Just add the hosts you want to blacklist into the blacklist table in the shown way. 

 *Complex rewrites* 

 If you want to implement caching on your document-root and only want to regenerate content if the requested file doesn't exist, you can attract the physical.path: 

 <pre> 
   magnet.attract-physical-path-to = ( server.document-root + "/rewrite.lua" ) 
 </pre> 

 rewrite.lua 

 <pre> 
   attr = lighty.stat(lighty.env["physical.path"]) 

   if (not attr) then 
     -- we couldn't stat() the file for some reason 
     -- let the backend generate it 

     lighty.env["uri.path"] = "/dispatch.fcgi" 
     lighty.env["physical.rel-path"] = lighty.env["uri.path"] 
     lighty.env["physical.path"] = lighty.env["physical.doc-root"] .. lighty.env["physical.rel-path"] 
   end 
 </pre> 


 *Extension rewrites* 

 If you want to hide your file extensions (like .php) you can attract the physical.path: 

 <pre> 
   magnet.attract-physical-path-to = ( server.document-root + "/rewrite.lua" ) 
 </pre> 

 rewrite.lua 

 <pre> 
   attr = lighty.stat(lighty.env["physical.path"] .. ".php") 

   if (attr) then 
     lighty.env["uri.path"] = lighty.env["uri.path"] .. ".php" 
     lighty.env["physical.rel-path"] = lighty.env["uri.path"] 
     lighty.env["physical.path"] = lighty.env["physical.doc-root"] .. lighty.env["physical.rel-path"] 
   end 
 </pre> 


 *User tracking* 

 ... or how to store data globally in the script-context: 

 Each script has its own script-context. When the script is started it only contains the lua-functions and the special lighty.* name-space. If you want to save data between script runs, you can use the global-script context: 

 <pre> 
   if (nil == _G["usertrack"]) then 
     _G["usertrack"] = {} 
   end 
   if (nil == _G["usertrack"][lighty.request["Cookie"]]) then 
     _G["usertrack"][lighty.request["Cookie"]] 
   else  
     _G["usertrack"][lighty.request["Cookie"]] = _G["usertrack"][lighty.request["Cookie"]] + 1 
   end 

   print _G["usertrack"][lighty.request["Cookie"]] 
 </pre> 

 The global-context is per script. If you update the script without restarting the server, the context will still be maintained. 

 *WordpressMU* 

 wpmu.lua 

 <pre> 
 if (not lighty.stat(lighty.env["physical.path"])) then 
   if (string.match(lighty.env["uri.path"], "^(/?[^/]*/)files/$")) then 
     lighty.env["physical.rel-path"] = "index.php" 
   else 
     n, a = string.match(lighty.env["uri.path"], "^(/?[^/]*/)files/(.+)") 
     if a then 
       lighty.env["physical.rel-path"] = "wp-content/blogs.php" 
       lighty.env["uri.query"] = "file=" .. a 
     else 
       n, a = string.match(lighty.env["uri.path"], "^(/[^/]*)/(wp-.*)") 
       if a then 
         lighty.env["physical.rel-path"] = a; 
       else 
         n, a = string.match(lighty.env["uri.path"], "^(/[^/]*)/(.*\.php)$") 
         if a then 
           lighty.env["physical.rel-path"] = a 
         else 
           lighty.env["physical.rel-path"] = "index.php" 
         end 
       end 
     end 
   end 
   lighty.env["physical.path"] = lighty.env["physical.doc-root"] .. "/".. lighty.env["physical.rel-path"] 
 end 
 </pre> 

 h1. Content Negotiation 

 content-negotiation.lua to parse Accept-Language and Accept-Encoding (#2678, #2736) to determine best target file 
 "content-negotiation.lua":https://redmine.lighttpd.net/attachments/2012 

 Related, see #1259 for lua code to try multiple extensions (a la Apache mod_autoext) to find target file 

 h1. Fight DDoS 

 If your Server is under high load because of someone is flooding you with requests, a little bit lua might help you. ;) In our case we've got a lot of requests without a User-Agent in the request header. 
 <pre>if ( lighty.request["User-Agent"]== nil ) then 
         file = io.open ("ips.txt","a") 
         file:write(lighty.env["request.remote-ip"]) 
         file:write("\n") 
         file:close() 
         return 200 
 end</pre> 

 The field request.remote-ip is available since lighttpd 1.4.23. The file ips.txt must be writeable by the lighttpd user (www-data). The bad guys in the ips.txt file can be dropped into the firewall with a little shell script. 

 lua can also be used to access a database and reject requests based on data in the database. 
 See sample "reject-bad-actors.lua":https://redmine.lighttpd.net/attachments/2064 attached in Files section at the bottom of this page.    It uses an mcdb constant database for fast lookups. 

 h1. Do basic HTTP-Auth against a MySQL DB/Table 

 (Note: lighttpd [[Docs_ModAuth|mod_auth]] mod_authn_dbi is more secure and more performant than this simplistic example) 

 This script works for me, doing HTTP-Auth against a MySQL Table with Lua: 
 <pre> 
 -- vim: set ts=4 sw=4 sts=4 noai noet: 
 --[[ 
 MySQL Auth Lua Script for lighttpd 
 How to use: 
 1) add this to lighttpd.conf 
	 magnet.attract-raw-url-to = ( "/path/to/script/mysql_auth.lua" ) 
 2) Configure Database Access 
 3) Create this Table and fill it with users: 
	 CREATE TABLE IF NOT EXISTS `users` ( 
	   `fdMe` varchar(127) NOT NULL, 
	   `username` varchar(127) NOT NULL, 
	   `pass` varchar(127) NOT NULL, 
	   `realm` varchar(127) DEFAULT NULL, 
	   PRIMARY KEY (`fdMe`), 
	   KEY `username` (`username`), 
	   KEY `password` (`pass`), 
	   KEY `realm` (`realm`) 
	 ) ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='Users'; 
	
	 The column fdMe may/should be changed to a autoincrementing unsigned int or whatever your primary key is. 
	 If realm is NULL, a user has access to all realms! 
	 If you want to change this, change the SQL-Statement in checkAuthMySQL() 
 ]] 
 --[[ 
	 Config Variables 
 ]] 
 local dbConf = {} 
 dbConf.database = '' 
 dbConf.username = '' 
 dbConf.password = '' 
 dbConf.hostname = '' 
 --[[ 
	 Requires 
 ]] 
 -- Debian package: liblua5.1-socket2 
 -- required for Base64 De-/encoding. See: http://w3.impa.br/~diego/software/luasocket/home.html 
 require("mime") 
 -- Debian package: liblua5.1-sql-mysql-2 
 -- Lua Mysql Driver 
 require("luasql.mysql") 
 --[[ 
	 Function to send HTTP-Auth request 
 ]] 
 function doAuth() 
	 lighty.header["WWW-Authenticate"] = string.format('Basic realm="%s"', lighty.env["uri.authority"]) 
	 return 401 
 end 
 --[[ 
	 Function to check Auth Creds against MySQL Database 
 ]] 
 function checkAuthMySQL(user,pass) 
	 local MySQL = luasql.mysql() 
	 local con = MySQL:connect( 
			  dbConf.database 
			 ,dbConf.username 
			 ,dbConf.password 
			 ,dbConf.hostname 
		 ) 
	 local res = con:execute(string.format([[ 
		     SELECT * 
			 FROM `users` 
			 WHERE `username` = '%s' 
				 AND `pass` = '%s' 
				 AND (`realm` = '%s' OR `realm` IS NULL) 
			 ]],  
			 con:escape(user), 
			 con:escape(pass), 
			 con:escape(lighty.env["uri.authority"])) user, pass, lighty.env["uri.authority"]) 
		 ) 
	 local row = res:fetch ({}, "a") 
	 -- print(type(row)) 
	 -- close everything 
	 res:close() 
	 con:close() 
	 MySQL:close() 
	 if (not row) then 
		 return false 
	 else 
		 lighty.req_env['REMOTE_USER'] = user 
		 return true 
	 end 
 end 

 -- MAIN 
 --[[ 
	 Check for Authorization Header 
	 and force Basic Auth if not set. 
 ]] 
 if (not lighty.request.Authorization) then 
	 return doAuth() 
 end 
 --[[ 
	 Header found: check string for "Basic" and base64 encoded username & password 
	 - upb = User Password Base64 encoded 
 ]] 
 _, _, upb = string.find(lighty.request.Authorization, "^Basic%s+(.+)$") 
 if (not upb) then 
	 return doAuth() 
 end 

 up = mime.unb64(upb) -- Base64 Decode 
 _, _, username, password = string.find(up, "^(.+):(.+)$") -- split by ":" to get username and password supplied 
 if (not checkAuthMySQL(username, password)) then return doAuth() end 

 -- return nothing to proceed normal operation 
 return 
 </pre> 

 *Known Problems:* 
 * This Script is blocking!!! lighttpd will hang, if there are MySQL connection problems. 
 * Additionally, a whole new MySQL Connection is created with every request! So you shouldn't use this on High-Traffic Sites.  
 * Passwords are stored plain in MySQL - well, easy to fix. Look for MySQL's PASSWORD function.... 


 h1. Mod_Security 

 Apache has mod_security available as a WAF (web application firewall) however this isn't available for other webservers. I've written a quick and dirty script to perform a similar task to mod_security using mod_magnet 
 "lighttpd-mod_security-via-mod_magnet":https://web.archive.org/web/20160311005141/http://whmcr.com/2009/06/19/lighttpd-mod_security-via-mod_magnet 

 I've been recently working on libmodsecurity binding for openresty which is a nginx+luajit combo. Using patched "libmodsecurity":https://github.com/pr4u4t/ModSecurity and "cffi-lua":https://github.com/q66/cffi-lua along with "modsec.lua":https://luarocks.org/modules/pr4u4t/modsec and mod_magnet lighttpd can perform incoming request inspection. Currently there is no way to perform request body (POST) and response inspection. Two last would involve additional lua entry points in lighty. Security rules could be obtained at "coreruleset":https://github.com/coreruleset/coreruleset. 

 waf.lua for lighttpd: 
 <pre> 
 local modsec = require "modsec" 

 local ok, err = modsec.init("/etc/owasp/modsec.conf") 
 if not ok then 
         print(err) 
         return 
 end 

 local transaction = modsec.transaction() 

 if not transaction then 
         print("Failed to initialize transaction") 
 end 

 -- evaluate connection info and request headers 
 local req_attr = lighty.r.req_attr 
 local url = req_attr["uri.scheme"] 
          .. "://"  
          .. req_attr["uri.authority"] 
          .. req_attr["uri.path-raw"] 
          .. (req_attr["uri.query"] and ("?" .. req_attr["uri.query"]) or "")  

 local res, err = transaction:eval_connection(req_attr["request.remote-addr"],req_attr["request.remote-port"], 
                                                 req_attr["uri.authority"],req_attr["request.server-port"],url, 
                                                 req_attr["request.method"],req_attr["request.protocol"]) 

 if err then 
         print("Failed to evaluate connection: ",err) 
 end 

 local res, err = transaction:eval_request_headers(lighty.r.req_header) 

 if err then 
         print("Failed to evaluate request headers: ",err) 
 end 

 --[[ evaluate request body 
 Currently no way to evaluate request body 
 but this function must be run even with nil as arguments 
 ]] 

 local res, err = transaction:eval_request_body(nil,nil) 

 if err then 
         print("Failed to evaluate request body: ",err) 
 end 

 -- Here decision could be made upon modsecurity variables whether handle this request or not 
 local score = tonumber(transaction.var.tx.anomaly_score) 

 if score >= 8 then 
         print("This request looks nasty overall score is: "..score) 
         return 403 
 end 
 </pre> 

 Example of owasp/modsec.conf 
 <pre> 
 #This is libmodsecurity base configuration 
 Include modsecurity.conf 

 Include /opt/openresty/owasp/crs-setup.conf 
 Include /opt/openresty/owasp/rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf 
 Include /opt/openresty/owasp/rules/REQUEST-901-INITIALIZATION.conf 
 Include /opt/openresty/owasp/rules/REQUEST-905-COMMON-EXCEPTIONS.conf 
 Include /opt/openresty/owasp/rules/REQUEST-910-IP-REPUTATION.conf 
 Include /opt/openresty/owasp/rules/REQUEST-911-METHOD-ENFORCEMENT.conf 
 Include /opt/openresty/owasp/rules/REQUEST-912-DOS-PROTECTION.conf 
 Include /opt/openresty/owasp/rules/REQUEST-913-SCANNER-DETECTION.conf 
 Include /opt/openresty/owasp/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf 
 Include /opt/openresty/owasp/rules/REQUEST-921-PROTOCOL-ATTACK.conf 
 Include /opt/openresty/owasp/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf 
 Include /opt/openresty/owasp/rules/REQUEST-931-APPLICATION-ATTACK-RFI.conf 
 Include /opt/openresty/owasp/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf 
 Include /opt/openresty/owasp/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf 
 Include /opt/openresty/owasp/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf 
 Include /opt/openresty/owasp/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf 
 Include /opt/openresty/owasp/rules/REQUEST-943-APPLICATION-ATTACK-SESSION-FIXATION.conf 
 Include /opt/openresty/owasp/rules/REQUEST-949-BLOCKING-EVALUATION.conf 
 Include /opt/openresty/owasp/rules/RESPONSE-950-DATA-LEAKAGES.conf 
 Include /opt/openresty/owasp/rules/RESPONSE-951-DATA-LEAKAGES-SQL.conf 
 Include /opt/openresty/owasp/rules/RESPONSE-952-DATA-LEAKAGES-JAVA.conf 
 Include /opt/openresty/owasp/rules/RESPONSE-953-DATA-LEAKAGES-PHP.conf 
 Include /opt/openresty/owasp/rules/RESPONSE-954-DATA-LEAKAGES-IIS.conf 
 Include /opt/openresty/owasp/rules/RESPONSE-959-BLOCKING-EVALUATION.conf 
 Include /opt/openresty/owasp/rules/RESPONSE-980-CORRELATION.conf 
 Include /opt/openresty/owasp/rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf 
 </pre> 

 h1. other solutions 

 *external-static* 

 I´ve seen this nice solution somewhere where they host some files locally on their machines. If popularity gets to high, files are too big or for whatever reasons the files are moved to i think it was amazon´s S3 or akamai for faster serving or to cope with high traffic. You still can use your hostname, urls, collect stats from your logs - your users are just redirected with a 302 to the files they ask for. 

 2008-11-17: Found the source: "presto-move-content-to-s3-with-no-code-changes":https://web.archive.org/web/20120817090808/http://www.innerfence.com/blog/2008/05/31/presto-move-content-to-s3-with-no-code-changes/ 

   Request -> check for local copy -> 302 (if not stored locally) -> let users download from a big pipe 

 Add the following to your lighttpd.conf: 
 <pre> 
 $HTTP["url"] =~ "^/static/[^/]+[.]gif([?].*)?$" { #match the files you want this to work for 
   magnet.attract-physical-path-to = ( "/path-to-your/external-static.lua" ) 
 } 
 </pre> 

 Save the following to external-static.lua: 
 <pre> 
  local filename = lighty.env["physical.path"] 
  local stat = lighty.stat( filename ) 
  if not stat then 
    local static_name = string.match( filename, "static/([^/]+)$" ) 
    lighty.header["Location"] = "http://<new-location-with-big-pipes>/" .. static_name 
    return 302 
  end  
 </pre> 


 h1. Sample Files