AbsoLUAtion » History » Revision 24
« Previous |
Revision 24/55
(diff)
| Next »
laph, 2010-09-15 19:34
AbsoLUAtion - The powerful combo of Lighttpd + Lua¶
- Table of contents
- AbsoLUAtion - The powerful combo of Lighttpd + Lua
- What's needed for this to work?
- How to get it up and running
- Links
- Code-Snippets
- Fight DDoS
- Do basic HTTP-Auth against a MySQL DB/Table
- Small Helpers
- Mod_Security
- other solutions
We want to build a central resource for Lighttpd + Lua as this is one of the biggest advantages Lighttpd has over other webservers. It's useful, handy, simple and sometimes a quite powerful combo which gives you some additional flexibility and offers you solutions to small and big problems other httpds can't solve!
Again we hope you - the users of lighty - support this page by contributing links, code-snippets or simply offer your lua scripts with small descriptions of what they do and how it helps lighty to do stuff you want it to do.
What's needed for this to work?¶
- A recent version of Lighttpd
- mod_magnet
- Lua (v5.0 or better v5.1, your distro´s should have this) http://www.lua.org
How to get it up and running¶
Links¶
Link collection (Description -> Link).
- darix is maintaining the cleanurl.lua at http://nordisch.org/
- Jippi is maintaining the bundle.lua at http://www.cakephp.nu/faster-page-loads-bundle-your-css-and-javascript-lighttpd-mod_magnet-lua
- http://www.sitepoint.com/blogs/2007/04/10/faster-page-loads-bundle-your-css-and-javascript/
- Google CodeSearch is great for looking Lua examples. http://www.google.com/codesearch start your queries with "lang:lua"
- WP-Supercache http://tempe.st/2008/04/lighttpd-and-wp-supercache-now-you-can/
- WP-MultiUser http://www.bisente.com/blog/2007/04/08/lighttpd-wordpressmu-english/
- Dynamically generate thumbnails and cache them http://www.xarg.org/2010/04/dynamic-thumbnail-generation-on-the-static-server/
- LUA Script performing similar job to Mod_Security http://www.whmcr.com/2009/06/lighttpd-mod_security-via-mod_magnet/
- Drupal/OpenAtrium simple cleanurl solution http://sudhaker.com/web-development/drupal/drupal-clean-url-lighttpd.html
Dead links? You don´t like to be listed here? Please remove it. Thanks!
Code-Snippets¶
The is_file/is_dir dilemma
Known from Apache´s .htaccess:
Hint: see url.rewrite-[repeat-]if-not-file for the !-f part
RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteRule ^(.*)$ index.php?q=$1 [L,QSA]
As lighttpd doesn´t provide this is_file/is_dir check out of the box, again mod_magnet comes into play.
I took the example for drupal from darix site (http://nordisch.org/2007/2/6/drupal-on-lighttpd-with-clean-urls):
Lets assume drupal is already installed under http://example.com/drupal/ you now add the magnet part to it.
$HTTP["url"] =~ "^/drupal" { # we only need index.php here. index-file.names = ( "index.php" ) # for clean urls magnet.attract-physical-path-to = ( "/etc/lighttpd/drupal.lua" ) }
The drupal.lua:
-- little helper function function file_exists(path) local attr = lighty.stat(path) if (attr) then return true else return false end end function removePrefix(str, prefix) return str:sub(1,#prefix+1) == prefix.."/" and str:sub(#prefix+2) end -- prefix without the trailing slash local prefix = '/drupal' -- the magic ;) if (not file_exists(lighty.env["physical.path"])) then -- file still missing. pass it to the fastcgi backend request_uri = removePrefix(lighty.env["uri.path"], prefix) if request_uri then lighty.env["uri.path"] = prefix .. "/index.php" local uriquery = lighty.env["uri.query"] or "" lighty.env["uri.query"] = uriquery .. (uriquery ~= "" and "&" or "") .. "q=" .. request_uri lighty.env["physical.rel-path"] = lighty.env["uri.path"] lighty.env["request.orig-uri"] = lighty.env["request.uri"] lighty.env["physical.path"] = lighty.env["physical.doc-root"] .. lighty.env["physical.rel-path"] end end -- fallthrough will put it back into the lighty request loop -- that means we get the 304 handling for free. ;)
Overwrite default mime-type/content-type
Add "magnet.attract-physical-path-to = ( "/path-to/change-ctype.lua" )" to lighttpd.conf and save the following as "change-ctype.lua"
if (string.match(lighty.env["physical.rel-path"], ".swf")) then lighty.header["Content-Type"] = "text/html"
Sending text-files as HTML
This is a bit simplistic, but it illustrates the idea: Take a text-file and cover it in a "< pre >" tag.
Config-file
magnet.attract-physical-path-to = (server.docroot + "/readme.lua")
readme.lua
lighty.content = { "<pre>", { filename = "/README" }, "</pre>" } lighty.header["Content-Type"] = "text/html" return 200
Simple maintenance script
You need three files, maint.up, maint.down and maint.html.
maint.html holds a simple html-page of what you want to display to your users while in maintenance-mode.
Add "magnet.attract-physical-path-to = ( "/path-to-your/maint.lua" )" to your lighttpd.conf, best is global section or within a host-section of your config, e.g. a board/forum/wiki you know a maintenance-mode is needed from time to time. If you want to switch to maintenance-mode, just copy maint.down to maint.lua in your "/path-to-your/" location, and lighty will display your maint.html to all users - without restarting anything - this can be done on-the-fly. Work is done and all is up again? Copy maint.up to maint.lua in your "/path-to-your/" location. Whats maint.up doing? Nothing, just going on with normal file serving :-)
maint.up - all is up, user will see normal pages
-- This is empty, nothing to do.
maint.down - lighty will show the maintenance page -> maint.html
-- lighty.header["X-Maintenance-Mode"] = "1" -- uncomment the above if you want to add the header lighty.content = { { filename = "/path-to-your/maint.html" } } lighty.header["Content-Type"] = "text/html" return 503 -- or return 200 if you want
mod_flv_streaming
Config-file
magnet.attract-physical-path-to = (server.docroot + "/flv-streaming.lua")
flv-streaming.lua
if (lighty.env["uri.query"]) then -- split the query-string get = {} for k, v in string.gmatch(lighty.env["uri.query"], "(%w+)=(%w+)") do get[k] = v end header="" if (get and get["start"]) then start = tonumber(get["start"]) else start=0 end -- send te FLV header only when seeking + a seek into the file if (start ~= nil and start > 0) then header="FLV\1\1\0\0\0\9\0\0\0\9" end lighty.content = { header , { filename = lighty.env["physical.path"], offset = start } } lighty.header["Content-Type"] = "video/x-flv" return 200 end
You can also use a backend like php to use your own authorization or stuff like mod_secdl. Just activate x-rewrite in the backend configuration and use a header like
header("X-Rewrite-URI: flvstreaming?start=" . $start . "&path=" . $path);
The request is restarted and in the lua, you can catch the non-existing uri with the following code (wrap it between the example below)::
if (string.find(lighty.env["uri.path"],"/flvstreaming") then <flv streaming lua code> end
In the future, there will be a new magnet for response headers, maybe you can give your own headers like::
header("X-StreamMyFlv: $path");
to lua and use the header data as parameter for the streaming.
selecting a random file from a directory
Say, you want to send a random file (ad-content) from a directory.
To simplify the code and to improve the performance we define:
- all images have the same format (e.g. image/png)
- all images use increasing numbers starting from 1
- a special index-file names the highest number
Config
server.modules += ( "mod_magnet" ) magnet.attract-physical-path-to = ("random.lua")
random.lua
dir = lighty.env["physical.path"] f = assert(io.open(dir .. "/index", "r")) maxndx = f:read("*all") f:close() ndx = math.random(maxndx) lighty.content = { { filename = dir .. "/" .. ndx }} lighty.header["Content-Type"] = "image/png" return 200
denying illegal character sequences in the URL
Instead of implementing mod_security, you might just want to apply filters on the content and deny special sequences that look like SQL injection.
A common injection is using UNION to extend a query with another SELECT query.
if (string.find(lighty.env["request.uri"], "UNION%s")) then return 400 end
Traffic Quotas
If you only allow your virtual hosts a certain amount for traffic each month and want to disable them if the traffic is reached, perhaps this helps:
host_blacklist = { ["www.example.org"] = 0 } if (host_blacklist[lighty.request["Host"]]) then return 404 end
Just add the hosts you want to blacklist into the blacklist table in the shown way.
Complex rewrites
If you want to implement caching on your document-root and only want to regenerate content if the requested file doesn't exist, you can attract the physical.path:
magnet.attract-physical-path-to = ( server.document-root + "/rewrite.lua" )
rewrite.lua
attr = lighty.stat(lighty.env["physical.path"]) if (not attr) then -- we couldn't stat() the file for some reason -- let the backend generate it lighty.env["uri.path"] = "/dispatch.fcgi" lighty.env["physical.rel-path"] = lighty.env["uri.path"] lighty.env["physical.path"] = lighty.env["physical.doc-root"] .. lighty.env["physical.rel-path"] end
Extension rewrites
If you want to hide your file extensions (like .php) you can attract the physical.path:
magnet.attract-physical-path-to = ( server.document-root + "/rewrite.lua" )
rewrite.lua
attr = lighty.stat(lighty.env["physical.path"] .. ".php") if (attr) then lighty.env["uri.path"] = lighty.env["uri.path"] .. ".php" lighty.env["physical.rel-path"] = lighty.env["uri.path"] lighty.env["physical.path"] = lighty.env["physical.doc-root"] .. lighty.env["physical.rel-path"] end
User tracking
... or how to store data globally in the script-context:
Each script has its own script-context. When the script is started it only contains the lua-functions and the special lighty.* name-space. If you want to save data between script runs, you can use the global-script context:
if (nil == _G["usertrack"]) then _G["usertrack"] = {} end if (nil == _G["usertrack"][lighty.request["Cookie"]]) then _G["usertrack"][lighty.request["Cookie"]] else _G["usertrack"][lighty.request["Cookie"]] = _G["usertrack"][lighty.request["Cookie"]] + 1 end print _G["usertrack"][lighty.request["Cookie"]]
The global-context is per script. If you update the script without restarting the server, the context will still be maintained.
WordpressMU
wpmu.lua
if (not lighty.stat(lighty.env["physical.path"])) then if (string.match(lighty.env["uri.path"], "^(/?[^/]*/)files/$")) then lighty.env["physical.rel-path"] = "index.php" else n, a = string.match(lighty.env["uri.path"], "^(/?[^/]*/)files/(.+)") if a then lighty.env["physical.rel-path"] = "wp-content/blogs.php" lighty.env["uri.query"] = "file=" .. a else n, a = string.match(lighty.env["uri.path"], "^(/[^/]*)/(wp-.*)") if a then lighty.env["physical.rel-path"] = a; else n, a = string.match(lighty.env["uri.path"], "^(/[^/]*)/(.*\.php)$") if a then lighty.env["physical.rel-path"] = a else lighty.env["physical.rel-path"] = "index.php" end end end end lighty.env["physical.path"] = lighty.env["physical.doc-root"] .. "/".. lighty.env["physical.rel-path"] end
Fight DDoS¶
If your Server is under high load because of someone is flooding you with requests, a little bit lua might help you. ;) In our case we've got a lot of requests without a User-Agent in the request header.
if ( lighty.request["User-Agent"]== nil ) then file = io.open ("ips.txt","a") file:write(lighty.env["request.remote-ip"]) file:write("\n") file:close() return 200 end
The field request.remote-ip is available since Lighttpd 1.4.23. The file ips.txt must be writeable by the lighttpd user (www-data). The bad guys in the ips.txt file can be dropped into the firewall with a little shell script.
Do basic HTTP-Auth against a MySQL DB/Table¶
This script works for me, doing HTTP-Auth against a MySQL Table with Lua:
-- vim: set ts=4 sw=4 sts=4 noai noet: --[[ MySQL Auth Lua Script for Lighttpd How to use: 1) add this to lighttpd.conf magnet.attract-raw-url-to = ( "/path/to/script/mysql_auth.lua" ) 2) Configure Database Access 3) Create this Table and fill it with users: CREATE TABLE IF NOT EXISTS `users` ( `fdMe` varchar(127) NOT NULL, `username` varchar(127) NOT NULL, `pass` varchar(127) NOT NULL, `realm` varchar(127) DEFAULT NULL, PRIMARY KEY (`fdMe`), KEY `username` (`username`), KEY `password` (`pass`), KEY `realm` (`realm`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='Users'; The column fdMe may/should be changed to a autoincrementing unsigned int or whatever your primary key is. If realm is NULL, a user has access to all realms! If you want to change this, change the SQL-Statement in checkAuthMySQL() ]] --[[ Config Variables ]] local dbConf = {} dbConf.database = '' dbConf.username = '' dbConf.password = '' dbConf.hostname = '' --[[ Requires ]] -- Debian package: liblua5.1-socket2 -- required for Base64 De-/encoding. See: http://w3.impa.br/~diego/software/luasocket/home.html require("mime") -- Debian package: liblua5.1-sql-mysql-2 -- Lua Mysql Driver require("luasql.mysql") --[[ Function to send HTTP-Auth request ]] function doAuth() lighty.header["WWW-Authenticate"] = string.format('Basic realm="%s"', lighty.env["uri.authority"]) return 401 end --[[ Function to check Auth Creds against MySQL Database ]] function checkAuthMySQL(user,pass) local MySQL = luasql.mysql() local con = MySQL:connect( dbConf.database ,dbConf.username ,dbConf.password ,dbConf.hostname ) local res = con:execute(string.format([[ SELECT * FROM `users` WHERE `username` = '%s' AND `pass` = '%s' AND (`realm` = '%s' OR `realm` IS NULL) ]], user, pass, lighty.env["uri.authority"]) ) local row = res:fetch ({}, "a") -- print(type(row)) -- close everything res:close() con:close() MySQL:close() if (not row) then return false else return true end end -- MAIN --[[ Check for Authorization Header and force Basic Auth if not set. ]] if (not lighty.request.Authorization) then return doAuth() end --[[ Header found: check string for "Basic" and base64 encoded username & password - upb = User Password Base64 encoded ]] _, _, upb = string.find(lighty.request.Authorization, "^Basic%s+(.+)$") if (not upb) then return doAuth() end up = mime.unb64(upb) -- Base64 Decode _, _, username, password = string.find(up, "^(.+):(.+)$") -- split by ":" to get username and password supplied if (not checkAuthMySQL(username, password)) then return doAuth() end -- return nothing to proceed normal operation returnKnown Problems:
- This Script is blocking!!! Lighttpd will hang, if there are MySQL connection problems.
- Additionally, a whole new MySQL Connection is created with every request! So you shouldn't use this on High-Traffic Sites.
- Passwords are stored plain in MySQL - well, easy to fix. Look for MySQL's PASSWORD function....
Small Helpers¶
...
Mod_Security¶
Apache has mod_security available as a WAF (web application firewall) however this isn't available for other webservers. I've written a quick and dirty script to perform a similar task to mod_security using mod_magnet
http://www.whmcr.com/2009/06/lighttpd-mod_security-via-mod_magnet/
other solutions¶
external-static
I´ve seen this nice solution somewhere where they host some files locally on their machines. If popularity gets to high, files are too big or for whatever reasons the files are moved to i think it was amazon´s S3 or akamai for faster serving or to cope with high traffic. You still can use your hostname, urls, collect stats from your logs - your users are just redirected with a 302 to the files they ask for.
2008-11-17: Found the source: http://blog.innerfence.com/2008/05/31/presto-move-content-to-s3-with-no-code-changes/
Request -> check for local copy -> 302 (if not stored locally) -> let users download from a big pipe
Add the following to your lighttpd.conf:
$HTTP["url"] =~ "^/static/[^/]+[.]gif([?].*)?$" { #match the files you want this to work for magnet.attract-physical-path-to = ( "/path-to-your/external-static.lua" ) }
Save the following to external-static.lua:
local filename = lighty.env["physical.path"] local stat = lighty.stat( filename ) if not stat then local static_name = string.match( filename, "static/([^/]+)$" ) lighty.header["Location"] = "http://<new-location-with-big-pipes>/" .. static_name return 302 end
Updated by laph over 14 years ago · 55 revisions