Lighttpd

http://pixel.global-banlist.de/

see the mod_magnet articles there, especially http://pixel.global-banlist.de/2006/10/6/dr-magneto-vs-mr-404-handler .

this bug is a wont fix

I looked at the mod_magnet solution but that seemed to have a high performance hit. So I modified a copy of mod_rewrite to support a new pair of options: url.rewrite-repeat-NF and url.rewrite-once-NF. These only apply the rewrite if the file corresponding to the URI does NOT exist in the Filestore (NF = Not Filestore).

See here for more information: http://lmmrtech.com/lighttpd_mod_rewrite_with_file_exists_check

-- lee

so your site is doing more than 4000-5000r/s?

thats the value i reached with a running a mod_magnet script.

i forgot to mention. the normal speed of the box is not much higher.

Not even remotely close to that! However I do hope that the load will grow dramatically and I want to be prepared. The mod_magnet+lua approach is clearly very flexible but it does involve a high performance hit according to the article mentioned above (http://pixel.global-banlist.de/2006/10/6/dr-magneto-vs-mr-404-handler). Since I don't need the extra flexibility and want to maximize the performance I think my patch the way to go.

-- lee

Please remove this Spam! And by the way, is there something new for this topic?

i still dont think we need that patch. mod_magnet can easily handle that for you.

mod_magnet cannot handle it easily, maybe it can handle it, but i don't know how :(
The problem is, that i can check if file exists with mod_magnet, but lighttpd cannot parse the lighttpd rewrite rules AFTER mod_magnet. So i check if file exists. If it exists, I return a 200 and provide the static page. But if it does not exists i dont know what to do, that lighttpd uses then the rewrite rules and send the php file to the fastcgi backend. There is no possibility to use lua between rewrite rules and fastcgi backend.

there is. load mod_rewrite before mod_magnet.

but I want only use the mod_rewrite IF the file does not exist :(

-> mod_magnet before mod_rewrite, and in the lua file two conditions
1. file exists -> return 200
2. file does not exist -> rewrite and parse php

why should we maintain 2 solutions if mod_magnet can do that and even more?

so is mod_rewrite being deprecated then? why keep it at all?

mod_magnet is a much more robust solution. it's like why have simple-vhost and evhost then? one is quick and dirty, the other is much more feature-rich.

i don't really want to learn lua or try to debug/figure out if it is meeting my needs 100% with a basic -f / -d apache emulation. there has been a lot of demand for this, and it seems like a couple lua scripts made but specific for people's installs. seems like if a few lines of code can be thrown in to mod_rewrite that get triggered only if the file does not exist, why not? it won't add much overhead unless that condition is true...

I agree, I don't want to take the performance hit of dropping into mod_magnet.
I'd rather just use re-write rules directly. I know mod magnet is feature rich, and flexible but why force us to take a performance hit just to use one of its features?

-- tech

seriously: please prove that mod_magnet is a performance problem for you.

I recommend the patch made by lmmrtech to be merged. It shouldn't make lighttpd too much bulkier.

It is a better option than embedding another interpreted language. I think 90% of the CMS's out there that need clean urls only need to check for file or directory existance, and do not need the extra fancy overhead of LUA.

http://lmmrtech.com/lighttpd_mod_rewrite_with_file_exists_check

Yes please integrate this in mod_rewrite - preferably also in the 1.4.x branch. May be, that LUA is a great language and mod_magnet is also really cool. But I (and I'm sure many others out there) don't need either to fulfill the (and my client's) demands.

Integration of this would make transition from Apache much easier.

mod_rewrite is voodoo, but anybody switching has this voodoo already worked out. It's annoying to have 13 lines of boilerplate lua code to get effect of a simple built-in flag!

-- kL

And what exactly stops you for using rewrite-once line like this:

"^/files.*\.(jpg|jpeg|gif|png|swf|avi|mpg|mpeg|mp3|flv|ico|css)$" => "$0"

with the file extensions you need? It means rewrite the file to itself, exit rewrite.

Works just fine under medium load (single cpu box, joomla, wordpress, simple machines forums, ~80req/s) - no mentionable performance issues.

-- Maciej Lisiewski

that doesn't check if the file exists or not.

that just does a rewrite based on the incoming URI. nothing more.

this is for a filesystem-level check if a file exists or not... two totally different things.

Replying to mike503:

that doesn't check if the file exists or not.

that just does a rewrite based on the incoming URI. nothing more.

this is for a filesystem-level check if a file exists or not... two totally different things.

That's right, but in 99% of cases 'if file exists' check is used to exclude some parts of the website from rewrite rule. The remaining 1% in most cases could be handled by setting 404 file.

I do know that neither of that solutions really provides feature that this issue was opened for, but it really narrows down the list of people who actually need it, thus priority trivial.

No, a mod_rewrite file check would provide the ideal solution.

Your solution is a poor one, and not future-proof. It requires you to define all the assets ahead of time that you want parsed normally. How do you know of all the file extensions that will be there?

Sure, you've taken care of the basics and most common, but if you allow user uploads or have anything else not included in that line, it will not process it correctly.

Simply put, it's not the solution to this problem and is the common misconception with all rewrite rules I keep seeing for lighty, nginx, and other webservers. People think that they need to call out the jpg/jpeg/gif/css/blah/blah separately and then they're covered, and most of the time it's a "solution" that does not truly work - your solution again does not check the filesystem at all; you can still get 404 errors on a .jpg file, because the rewrite rule you've made doesn't care if the file exists. It might "work fine" for you but it's not the solution that people are asking for, and some day if you happen to upload a file that is not matched in your regexp, you'll have that issue too.

This ticket is for a check if the file exists, the webserver needs to check the filesystem for it first. Period. Please stop changing the status.

This seems like a pretty basic feature which could be implemented (and has) with almost no performance hit in C, or done with mod_magnet as another commenter mentioned could handle mod_rewrite's job anyway. There are reasons for mod_rewrite's existance.

I can't believe this is such a debate for such a simple feature. Could we meet in the middle and at least add the patch as a compile-time option with configure?

-- pcloches

The 'if file exists' is needed. I agree with kL that I don't want to touch mod_magnet or lua code (plus have it as a system dependency just for a one liner), I just want add one simple line in my main config file. I want to serve up a thumbnail statically unless it doesn't exist, whereupon it gets passed to Apache which creates it. This command would allow me to configure lighthttpd in an easily maintainable way in seconds. The above alternative solutions do not fulfill my needs. I concur with mike503's arguments and I would like to see the merging in of the lmmrtech patch.

Phillip.

-- philliptemple

I fully agree with Philip and others. The solution provided by Lee is nice and extremely easy to use compared to magnet and LUA. People that won't use it won't have any noticeable performance loss when it's in, so what exactly is so bad about adding it?
For now I've recompiled the ubuntu package to include the modified mod_rewrite.c, but that'll be annoying to do everytime a new version gets released.

-- lucas.aerbeydt

(void)plugins_call_handle_docroot(srv, con);

I will never commit a module which does such things. It is ugly, and you have no idea what you are doing.

Btw: make diffs with the -u option

Well, to be perfectly honest I indeed have no idea how I'm supposed to handle that. I haven't had time to look through the full lighttpd source yet.

I know the response of plugins_call_handle_docroot should be checked instead of thrown away. Could you give me a brief explanation of the response codes? (HANDLER_GO_ON, HANDLER_FINISHED, HANDLER_COMEBACK, HANDLER_WAIT_FOR_EVENT, HANDLER_ERROR)

The diff is now also replaced by a -u version.

-- lucas.aerbeydt

Do you really think the patch is easy to fix with a little bit error handling?

I think you didn't understand why i said it is crap. I said it is crap as it is not usable for anything. It is just heavily broken. And if it works for you you are just lucky that your config isn't a little bit more complex.

Ok then, guess I'll have to wait until I have the time to go through the full source (or someone else jumps in with more lighttpd source knowledge).

So I guess it's the path forming that's broken then for advanced configs? (Since the rest of the code is quite trivial)

It'd definitely be nice to atleast know in which direction to look for solutions. (E.g. an example of a config that wouldn't work with the current code would be great)

-- lucas.aerbeydt

Yes, the problem is the physical path. mod_rewrite is way earlier than the path handling, and there are many things that can happen to the physical path (think of mod_alias).

I see. Just checked some more source, and basically the problem can be solved by doing it the magnet way I guess? The NF rules would be stored seperately and processed by a p>handle_physical function while the current mod_rewrite functionality would just stay as-is. Or do you see any other problems with that solution?

-- lucas.aerbeydt

Could you explain why the second part of the condition is there in response.c?

if (con->mode == DIRECT && con->physical.path->used == 0) {

It ofcourse prevents me from processing a comeback after rewriting with -NF. I have the new code written and working fine, but the repeat-NF only works when I remove

 && con->physical.path->used == 0

for obvious reasons.

(That was me by the way)

My guess is that it's there so it can skip the url forming if the comeback comes from a later part. In that case it could be easily removed or replace by some flag which can be reset by the physical handlers. It could ofcourse have another reason too, which is why I'm asking here.

-- lucas.aerbeydt

buffer_reset is your friend.

Thanks, I was looking at it in a too complicated way :)

The result:

Diff: http://lucasaerbeydt.com/lighty-mod_rewrite/mod_rewrite.diff

Source: http://lucasaerbeydt.com/lighty-mod_rewrite/mod_rewrite.c

-- lucas.aerbeydt

Parson my noobishness, but how do you compile it? I've looked on google and pretty much everything else and I don't see anything on how to coompile lighttpd mods.. I installed 1.4.19 via apt-get Ubuntu, and just want to get mod_rewrite working a bit better.

In ubuntu it is pretty easy (these are instructions for Hardy Heron):

sudo apt-get install build-essential libmysqlclient15-dev libpcre3-dev \ 
zlib1g-dev dpkg-dev libssl-dev libbz2-dev debhelper cdbs libattr1-dev \ 
libgamin-dev libldap2-dev libfcgi-dev libgdbm-dev libmemcache-dev \ 
liblua5.1-0-dev dpatch patchutils pkg-config uuid-dev libsqlite3-dev \
libxml2-dev libkrb5-dev fakeroot
apt-get source lighttpd
wget http://lucasaerbeydt.com/lighty-mod_rewrite/mod_rewrite.c
wget http://lucasaerbeydt.com/lighty-mod_rewrite/changelog
mv mod_rewrite.c lighttpd-1.4.19/src
mv changelog lighttpd-1.4.19/debian
dpkg-buildpackage -rfakeroot
cd ..
sudo dpkg -i lighttpd_1.4.19-0ubuntu4_i386.deb 

Please note that this new mod_rewrite is not tested enough to completely rely on in a production environment, but it shouldn't give any problems.

-- lucas.aerbeydt

Thanks a bunch, I didn't realize I had to recompile all of lighttpd, but it worked like a charm! This should definitely replace the current mod_rewrite, works beautifully
Thanks again!

Replying to lucas.aerbeydt@gmail.com:

Thanks, I was looking at it in a too complicated way :)

The result:

Diff: http://lucasaerbeydt.com/lighty-mod_rewrite/mod_rewrite.diff

Source: http://lucasaerbeydt.com/lighty-mod_rewrite/mod_rewrite.c

Ok, i had a look at it now - and i think this solution is better than using mod_magnet; there is only one small thing missing: set "con->mode = DIRECT;" in the HANDLER_COMEBACK case of NF matching (or check if con>mode == DIRECT before matching), otherwise it may conflict with other modules which already startet handling the request like mod_fastcgi.

Apart from that you should be ok using that patch.

Files have been updated to include the con->mode check.

-- lucas.aerbeydt

Fantastic, I look forward to this being included in the standard distribution. I will be using it extensively. Thanks stbuehler and lucas.

Phillip.

-- Phillip

Great! Thanks everyone involved for this patch. Only because of this feature we are now going to use lighty in a current project (would have been apache otherwise). I think this is far more straightforward than taking the plunge into mod_magnet.

Cheers,
Dan

-- DanB

We also need this function on a production server and I want to lobby here to have the feature added to Lighty's standard distribution.

Thanks.

-- H.

Any chance to have this patch finally taken into mainstream? This is really usefull features that's really missing from lighttpd, especially in so many cases, such as installing Wordpress with rewrite.

mod_magnet is in the mainstream for quite a while.

does not seem to be in trunk (e.g. 1.5). Any reason?