Feature #985

Request: mod_rewrite check if file exists (like mod_rewrite in apache)

Added by Anonymous about 7 years ago. Updated almost 4 years ago.

Status:FixedStart date:
Priority:NormalDue date:
Assignee:jan% Done:

100%

Category:mod_rewrite
Target version:1.4.24
Missing in 1.5.x:

Description

Hello,

can you please implement the function, that mod_rewrite first checks if a files exists and then rewrites the url? for example i use the following syntax:

"/(.+)/?$" => "index.php?name=$1",

Because i have urls like http://somewhere.us/my-page/ and using a CMS (Like Mambo, Wordpress etc). Now i can not use CSS oder JS, or favicon cause Lighttpd also rewrites the url http://somewhere.us/favicon.ico. I had to add all the static things in my Config, which really sucks!
Please do that, that would be a huge improvement!

-- ice

Associated revisions

Revision 2647
Added by stbuehler over 4 years ago

mod_rewrite: add url.rewrite-[repeat-]if-not-file to rewrite if file doesn't exist or is not a regular file (fixes #985, thx lucas aerbeydt)

History

#1 Updated by Anonymous about 7 years ago

http://pixel.global-banlist.de/

see the mod_magnet articles there, especially http://pixel.global-banlist.de/2006/10/6/dr-magneto-vs-mr-404-handler .

this bug is a wont fix

#2 Updated by Anonymous about 7 years ago

I looked at the mod_magnet solution but that seemed to have a high performance hit. So I modified a copy of mod_rewrite to support a new pair of options: url.rewrite-repeat-NF and url.rewrite-once-NF. These only apply the rewrite if the file corresponding to the URI does NOT exist in the Filestore (NF = Not Filestore).

See here for more information: http://lmmrtech.com/lighttpd_mod_rewrite_with_file_exists_check

-- lee

#3 Updated by Anonymous about 7 years ago

so your site is doing more than 4000-5000r/s?

thats the value i reached with a running a mod_magnet script.

#4 Updated by darix about 7 years ago

i forgot to mention. the normal speed of the box is not much higher.

#5 Updated by Anonymous about 7 years ago

Not even remotely close to that! However I do hope that the load will grow dramatically and I want to be prepared. The mod_magnet+lua approach is clearly very flexible but it does involve a high performance hit according to the article mentioned above (http://pixel.global-banlist.de/2006/10/6/dr-magneto-vs-mr-404-handler). Since I don't need the extra flexibility and want to maximize the performance I think my patch the way to go.

-- lee

#6 Updated by Anonymous about 7 years ago

Please remove this Spam! And by the way, is there something new for this topic?

#7 Updated by darix about 7 years ago

i still dont think we need that patch. mod_magnet can easily handle that for you.

#8 Updated by Anonymous about 7 years ago

mod_magnet cannot handle it easily, maybe it can handle it, but i don't know how :(
The problem is, that i can check if file exists with mod_magnet, but lighttpd cannot parse the lighttpd rewrite rules AFTER mod_magnet. So i check if file exists. If it exists, I return a 200 and provide the static page. But if it does not exists i dont know what to do, that lighttpd uses then the rewrite rules and send the php file to the fastcgi backend. There is no possibility to use lua between rewrite rules and fastcgi backend.

#9 Updated by darix about 7 years ago

there is. load mod_rewrite before mod_magnet.

#10 Updated by Anonymous about 7 years ago

but I want only use the mod_rewrite IF the file does not exist :(

-> mod_magnet before mod_rewrite, and in the lua file two conditions
1. file exists -> return 200
2. file does not exist -> rewrite and parse php

#11 Updated by jan about 7 years ago

  • Status changed from New to Fixed
  • Resolution set to fixed

use mod_magnet. We will provide more examples later.

#12 Updated by mike503 almost 7 years ago

  • Status changed from Fixed to Need Feedback
  • Resolution deleted (fixed)

is there a reason why this can't be put into lighttpd?
http://lmmrtech.com/lighttpd_mod_rewrite_with_file_exists_check

seems like he did all the heavy lifting already. i know he posted it above, but it seems like this could be a great fix without requiring magnet.

#13 Updated by darix almost 7 years ago

why should we maintain 2 solutions if mod_magnet can do that and even more?

#14 Updated by mike503 almost 7 years ago

so is mod_rewrite being deprecated then? why keep it at all?

mod_magnet is a much more robust solution. it's like why have simple-vhost and evhost then? one is quick and dirty, the other is much more feature-rich.

i don't really want to learn lua or try to debug/figure out if it is meeting my needs 100% with a basic -f / -d apache emulation. there has been a lot of demand for this, and it seems like a couple lua scripts made but specific for people's installs. seems like if a few lines of code can be thrown in to mod_rewrite that get triggered only if the file does not exist, why not? it won't add much overhead unless that condition is true...

#15 Updated by Anonymous almost 7 years ago

I agree, I don't want to take the performance hit of dropping into mod_magnet.
I'd rather just use re-write rules directly. I know mod magnet is feature rich, and flexible but why force us to take a performance hit just to use one of its features?

-- tech

#16 Updated by darix almost 7 years ago

seriously: please prove that mod_magnet is a performance problem for you.

#17 Updated by Anonymous over 6 years ago

I recommend the patch made by lmmrtech to be merged. It shouldn't make lighttpd too much bulkier.

It is a better option than embedding another interpreted language. I think 90% of the CMS's out there that need clean urls only need to check for file or directory existance, and do not need the extra fancy overhead of LUA.

http://lmmrtech.com/lighttpd_mod_rewrite_with_file_exists_check

#18 Updated by Anonymous over 6 years ago

Yes please integrate this in mod_rewrite - preferably also in the 1.4.x branch. May be, that LUA is a great language and mod_magnet is also really cool. But I (and I'm sure many others out there) don't need either to fulfill the (and my client's) demands.

#19 Updated by Anonymous almost 6 years ago

Integration of this would make transition from Apache much easier.

mod_rewrite is voodoo, but anybody switching has this voodoo already worked out. It's annoying to have 13 lines of boilerplate lua code to get effect of a simple built-in flag!

-- kL

#20 Updated by Anonymous almost 6 years ago

And what exactly stops you for using rewrite-once line like this:

"^/files.*\.(jpg|jpeg|gif|png|swf|avi|mpg|mpeg|mp3|flv|ico|css)$" => "$0"

with the file extensions you need? It means rewrite the file to itself, exit rewrite.

Works just fine under medium load (single cpu box, joomla, wordpress, simple machines forums, ~80req/s) - no mentionable performance issues.

-- Maciej Lisiewski

#21 Updated by mike503 almost 6 years ago

that doesn't check if the file exists or not.

that just does a rewrite based on the incoming URI. nothing more.

this is for a filesystem-level check if a file exists or not... two totally different things.

#22 Updated by Anonymous almost 6 years ago

Replying to mike503:

that doesn't check if the file exists or not.

that just does a rewrite based on the incoming URI. nothing more.

this is for a filesystem-level check if a file exists or not... two totally different things.

That's right, but in 99% of cases 'if file exists' check is used to exclude some parts of the website from rewrite rule. The remaining 1% in most cases could be handled by setting 404 file.

I do know that neither of that solutions really provides feature that this issue was opened for, but it really narrows down the list of people who actually need it, thus priority trivial.

#23 Updated by mike503 almost 6 years ago

No, a mod_rewrite file check would provide the ideal solution.

Your solution is a poor one, and not future-proof. It requires you to define all the assets ahead of time that you want parsed normally. How do you know of all the file extensions that will be there?

Sure, you've taken care of the basics and most common, but if you allow user uploads or have anything else not included in that line, it will not process it correctly.

Simply put, it's not the solution to this problem and is the common misconception with all rewrite rules I keep seeing for lighty, nginx, and other webservers. People think that they need to call out the jpg/jpeg/gif/css/blah/blah separately and then they're covered, and most of the time it's a "solution" that does not truly work - your solution again does not check the filesystem at all; you can still get 404 errors on a .jpg file, because the rewrite rule you've made doesn't care if the file exists. It might "work fine" for you but it's not the solution that people are asking for, and some day if you happen to upload a file that is not matched in your regexp, you'll have that issue too.

This ticket is for a check if the file exists, the webserver needs to check the filesystem for it first. Period. Please stop changing the status.

#24 Updated by Anonymous almost 6 years ago

This seems like a pretty basic feature which could be implemented (and has) with almost no performance hit in C, or done with mod_magnet as another commenter mentioned could handle mod_rewrite's job anyway. There are reasons for mod_rewrite's existance.

I can't believe this is such a debate for such a simple feature. Could we meet in the middle and at least add the patch as a compile-time option with configure?

-- pcloches

#25 Updated by Anonymous over 5 years ago

The 'if file exists' is needed. I agree with kL that I don't want to touch mod_magnet or lua code (plus have it as a system dependency just for a one liner), I just want add one simple line in my main config file. I want to serve up a thumbnail statically unless it doesn't exist, whereupon it gets passed to Apache which creates it. This command would allow me to configure lighthttpd in an easily maintainable way in seconds. The above alternative solutions do not fulfill my needs. I concur with mike503's arguments and I would like to see the merging in of the lmmrtech patch.

Phillip.

-- philliptemple

#26 Updated by Anonymous over 5 years ago

I fully agree with Philip and others. The solution provided by Lee is nice and extremely easy to use compared to magnet and LUA. People that won't use it won't have any noticeable performance loss when it's in, so what exactly is so bad about adding it?
For now I've recompiled the ubuntu package to include the modified mod_rewrite.c, but that'll be annoying to do everytime a new version gets released.

-- lucas.aerbeydt

#27 Updated by stbuehler over 5 years ago

  • Status changed from Need Feedback to Fixed
  • Resolution set to wontfix

The proposed patch is crap (did you even read the code if you tell us it is "nice"?). We will never use it upstream.

Perhaps we will include the feature if someone has a sane patch, but i wouldn't wait for that. Just use mod_magnet, and don't tell me it is a performance hit unless proving it is for you (the main performance hit will be the stat() call for the requested file, not lua).

And as i don't need more comments saying "but i really need this feature" (as i don't care), i just close this bug again.

#28 Updated by Anonymous over 5 years ago

  • Status changed from Fixed to Need Feedback
  • Resolution deleted (wontfix)

The "nice" was referring to ease of use. I don't want to force my users to learn the LUA syntax for something as simple as this. Performancewise it probably doesn't make a huge difference indeed, for me it really is the simplicity of this solution that counts.

I've just checked the code and indeed it was a bit messy (though I wouldn't call it "crap"). I've cleaned it up a bit and uploaded it. It's made for the 1.4.19 release, but I could rewrite for SVN version if you want (if that one is different, haven't checked).

Diff: http://lucasaerbeydt.com/lighty-mod_rewrite/diff

Source: http://lucasaerbeydt.com/lighty-mod_rewrite/mod_rewrite.c

I've reopened it but left it at lowest priority and trivial severity. Let me know if you want something improved/changed in the code and I'll work on it.

-- lucas.aerbeydt

#29 Updated by stbuehler over 5 years ago


(void)plugins_call_handle_docroot(srv, con);

I will never commit a module which does such things. It is ugly, and you have no idea what you are doing.

Btw: make diffs with the -u option

#30 Updated by Anonymous over 5 years ago

Well, to be perfectly honest I indeed have no idea how I'm supposed to handle that. I haven't had time to look through the full lighttpd source yet.

I know the response of plugins_call_handle_docroot should be checked instead of thrown away. Could you give me a brief explanation of the response codes? (HANDLER_GO_ON, HANDLER_FINISHED, HANDLER_COMEBACK, HANDLER_WAIT_FOR_EVENT, HANDLER_ERROR)

The diff is now also replaced by a -u version.

-- lucas.aerbeydt

#31 Updated by stbuehler over 5 years ago

Do you really think the patch is easy to fix with a little bit error handling?

I think you didn't understand why i said it is crap. I said it is crap as it is not usable for anything. It is just heavily broken. And if it works for you you are just lucky that your config isn't a little bit more complex.

#32 Updated by Anonymous over 5 years ago

Ok then, guess I'll have to wait until I have the time to go through the full source (or someone else jumps in with more lighttpd source knowledge).

So I guess it's the path forming that's broken then for advanced configs? (Since the rest of the code is quite trivial)

It'd definitely be nice to atleast know in which direction to look for solutions. (E.g. an example of a config that wouldn't work with the current code would be great)

-- lucas.aerbeydt

#33 Updated by stbuehler over 5 years ago

Yes, the problem is the physical path. mod_rewrite is way earlier than the path handling, and there are many things that can happen to the physical path (think of mod_alias).

#34 Updated by Anonymous over 5 years ago

I see. Just checked some more source, and basically the problem can be solved by doing it the magnet way I guess? The NF rules would be stored seperately and processed by a p>handle_physical function while the current mod_rewrite functionality would just stay as-is. Or do you see any other problems with that solution?

-- lucas.aerbeydt

#35 Updated by Anonymous over 5 years ago

Could you explain why the second part of the condition is there in response.c?


if (con->mode == DIRECT && con->physical.path->used == 0) {

It ofcourse prevents me from processing a comeback after rewriting with -NF. I have the new code written and working fine, but the repeat-NF only works when I remove


 && con->physical.path->used == 0

for obvious reasons.

#36 Updated by Anonymous over 5 years ago

(That was me by the way)

My guess is that it's there so it can skip the url forming if the comeback comes from a later part. In that case it could be easily removed or replace by some flag which can be reset by the physical handlers. It could ofcourse have another reason too, which is why I'm asking here.

-- lucas.aerbeydt

#37 Updated by stbuehler over 5 years ago

buffer_reset is your friend.

#38 Updated by Anonymous over 5 years ago

Thanks, I was looking at it in a too complicated way :)

The result:

Diff: http://lucasaerbeydt.com/lighty-mod_rewrite/mod_rewrite.diff

Source: http://lucasaerbeydt.com/lighty-mod_rewrite/mod_rewrite.c

-- lucas.aerbeydt

#39 Updated by Anonymous over 5 years ago

Parson my noobishness, but how do you compile it? I've looked on google and pretty much everything else and I don't see anything on how to coompile lighttpd mods.. I installed 1.4.19 via apt-get Ubuntu, and just want to get mod_rewrite working a bit better.

#40 Updated by Anonymous over 5 years ago

In ubuntu it is pretty easy (these are instructions for Hardy Heron):


sudo apt-get install build-essential libmysqlclient15-dev libpcre3-dev \ 
zlib1g-dev dpkg-dev libssl-dev libbz2-dev debhelper cdbs libattr1-dev \ 
libgamin-dev libldap2-dev libfcgi-dev libgdbm-dev libmemcache-dev \ 
liblua5.1-0-dev dpatch patchutils pkg-config uuid-dev libsqlite3-dev \
libxml2-dev libkrb5-dev fakeroot
apt-get source lighttpd
wget http://lucasaerbeydt.com/lighty-mod_rewrite/mod_rewrite.c
wget http://lucasaerbeydt.com/lighty-mod_rewrite/changelog
mv mod_rewrite.c lighttpd-1.4.19/src
mv changelog lighttpd-1.4.19/debian
dpkg-buildpackage -rfakeroot
cd ..
sudo dpkg -i lighttpd_1.4.19-0ubuntu4_i386.deb 

Please note that this new mod_rewrite is not tested enough to completely rely on in a production environment, but it shouldn't give any problems.

-- lucas.aerbeydt

#41 Updated by Anonymous over 5 years ago

Thanks a bunch, I didn't realize I had to recompile all of lighttpd, but it worked like a charm! This should definitely replace the current mod_rewrite, works beautifully
Thanks again!

#42 Updated by stbuehler over 5 years ago

Replying to :

Thanks, I was looking at it in a too complicated way :)

The result:

Diff: http://lucasaerbeydt.com/lighty-mod_rewrite/mod_rewrite.diff

Source: http://lucasaerbeydt.com/lighty-mod_rewrite/mod_rewrite.c

Ok, i had a look at it now - and i think this solution is better than using mod_magnet; there is only one small thing missing: set "con->mode = DIRECT;" in the HANDLER_COMEBACK case of NF matching (or check if con>mode == DIRECT before matching), otherwise it may conflict with other modules which already startet handling the request like mod_fastcgi.

Apart from that you should be ok using that patch.

#43 Updated by Anonymous over 5 years ago

Files have been updated to include the con->mode check.

-- lucas.aerbeydt

#44 Updated by Anonymous over 5 years ago

Fantastic, I look forward to this being included in the standard distribution. I will be using it extensively. Thanks stbuehler and lucas.

Phillip.

-- Phillip

#45 Updated by Anonymous over 5 years ago

Great! Thanks everyone involved for this patch. Only because of this feature we are now going to use lighty in a current project (would have been apache otherwise). I think this is far more straightforward than taking the plunge into mod_magnet.

Cheers,
Dan

-- DanB

#46 Updated by Anonymous over 5 years ago

We also need this function on a production server and I want to lobby here to have the feature added to Lighty's standard distribution.

Thanks.

-- H.

#47 Updated by icy about 5 years ago

  • Target version changed from 1.4.21 to 1.4.22
  • Patch available set to Yes

#48 Updated by stbuehler about 5 years ago

  • Target version changed from 1.4.22 to 1.4.23

#49 Updated by stbuehler almost 5 years ago

  • Target version changed from 1.4.23 to 1.4.24

#50 Updated by ono almost 5 years ago

Any chance to have this patch finally taken into mainstream? This is really usefull features that's really missing from lighttpd, especially in so many cases, such as installing Wordpress with rewrite.

#51 Updated by darix almost 5 years ago

mod_magnet is in the mainstream for quite a while.

#52 Updated by stbuehler over 4 years ago

  • Status changed from Need Feedback to Fixed
  • % Done changed from 0 to 100

Applied in changeset r2647.

#53 Updated by lifeofguenter almost 4 years ago

does not seem to be in trunk (e.g. 1.5). Any reason?

Also available in: Atom