Project

General

Profile

[Solved] regression in regexp vars v1.4.40

Added by carpii over 2 years ago

I've been having a lot of trouble with redirect rules after upgrading to 1.4.45

It seems that the substitution vars are always empty, or in some cases / or // (but never what I expect them to be).

Even the basic documented examples are not working for me...

# based on first example on https://redmine.lighttpd.net/projects/1/wiki/HowToRedirectHttpToHttps (added /foo to prove the rule is matching)
$HTTP["scheme"] == "http" {
    $HTTP["host"] =~ ".*" {
        url.redirect = (".*" => "https://%0$0/foo")
    }
}

+Testing with lighttpd 1.4.40 -1.4.45

[carpii@imac:~]$ curl -v http://msolo.carpnet.local/bar
HTTP/1.1 301 Moved Permanently
Location: https:///bar/foo             <------
Content-Length: 0
Date: Mon, 20 Feb 2017 20:27:13 GMT
Server: lighttpd

Rolled back to lighttpd 1.4.39, same config as above

[carpii@imac:~]$ curl -v http://msolo.carpnet.local/bar
HTTP/1.1 301 Moved Permanently
Location: https://msolo.carpnet.local/bar/foo     <------
Content-Length: 0
Date: Mon, 20 Feb 2017 20:29:33 GMT
Server: lighttpd

Im aware of a redirect regression in mod_cgi here... https://redmine.lighttpd.net/issues/2793
But it doesn't seem as if this is entirely related

[Edit: Works in 1.4.39, broken in 1.4.40]


Replies (15)

RE: regression in regexp vars v1.4.40 - Added by gstrauss over 2 years ago

Sounds like a bug. Thanks for the thorough report. I'll have a chance to look at this more closely on Wed.

RE: regression in regexp vars v1.4.40 - Added by gstrauss over 2 years ago

So far, I have been unable to reproduce what you are seeing. I tried a simple lighttpd.conf:

server.bind = "localhost" 
server.port = 8080
server.document-root = "/tmp" 
server.modules += ( "mod_redirect")
$HTTP["host"] =~ ".*" {
    url.redirect = (".*" => "https://%0$0/foo")
}

with curl -v http://localhost:8080/bar with result:
< HTTP/1.1 301 Moved Permanently
< Location: https://localhost:8080/bar/foo
< ...

carpii, please check that your curl command is sending a "Host" request header, or else %0 will match the empty string "", which may be what you're seeing. Also, would you please provide your lighttpd.conf (output of lighttpd -p -f /etc/lighttpd/lighttpd.conf) ?

RE: regression in regexp vars v1.4.40 - Added by gstrauss over 2 years ago

A have a hunch this is DNS-related.

Try this with more recent lighttpd to force curl to send 'Host' request header (and you can change that value to verify what is returned)

[carpii@imac:~]$ curl -v -H "Host: msolo.carpnet.local" http://msolo.carpnet.local/bar

RE: regression in regexp vars v1.4.40 - Added by carpii over 2 years ago

Thanks, I suspect you are right about it being Host header related.
My dev box does serve various subdomains from .carpnet.local but I assumed curl sent a Host header automatically

I will dig into this fully in the next day or two, and report back

RE: regression in regexp vars v1.4.40 - Added by gstrauss over 2 years ago

$HTTP["scheme"] == "http" {
    $HTTP["host"] =~ ".+" {
        url.redirect = (".*" => "https://%0$0/foo")
    }
    else {
        url.redirect = (".*" => "https://msolo.carpnet.local$0/foo")
    }
}

You can also do something similar with a short snippet of lua code and mod_magnet.

RE: regression in regexp vars v1.4.40 - Added by carpii over 2 years ago

Thanks for looking into this. I'm gradually homing in on the issue I think
I can confirm my curl requests were sending a Host header, and Ive confirmed lighty is receiving this header it in all cases (I added it to access log format).

What I overlooked, is the fact Im actually accessing lighttpd via haproxy
Haproxy does little more than provide https termination infront of lighty, and also appends a X-Forwarded-Proto header ("http" or "https").
Now, If I curl lighttpd directly (I have it running on port 8000), then all appears to be working, and this corresponds with what you're seeing.

In order for lighty to pick up the X-Forwarded-Proto header from haproxy, and use $HTTP["scheme"] based rules, I am using mod_extforward.
This appears to be where the issue is.

Whenever a request goes through haproxy, and lighty is using mod_extforward, then the regexp vars appear to be getting mangled somehow (even when the Host header is present).
I should have realised this was related a lot sooner, as I tidied up the mod_extforward wiki page a few days ago.

There was some sort of bugfix to mod_extforward in v1.4.40 and I think this has caused the regression.
I cant find the actual revision at the moment, I'll keep looking..

RE: regression in regexp vars v1.4.40 - Added by carpii over 2 years ago

Found it.. I think it may be related to this

https://redmine.lighttpd.net/issues/1499

RE: regression in regexp vars v1.4.40 - Added by carpii over 2 years ago

I've simplified my haproxy and lighty configs as much as I can to demonstrate the problem...

https://gist.github.com/carpii/63b98744216e05c5768e388f48a0a33e

Hitting haproxy, lighttpd 1.4.45, using mod_extforward

[carpii@imac:~/lighty]$ curl -v -H "Host: msolo.carpnet.local" http://192.168.1.79/bar
*   Trying 192.168.1.79...
* TCP_NODELAY set
* Connected to 192.168.1.79 (192.168.1.79) port 80 (#0)
> GET /bar HTTP/1.1
> Host: msolo.carpnet.local
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Location: https:///bar/foo1
< Content-Length: 0
< Date: Thu, 23 Feb 2017 12:38:54 GMT
< Server: lighttpd
<
* Curl_http_done: called premature == 0
* Connection #0 to host 192.168.1.79 left intact

Hitting haproxy, lighttpd 1.4.45, mod_extforward not loaded

[carpii@imac:~/lighty]$ curl -v -H "Host: msolo.carpnet.local" http://192.168.1.79/bar
*   Trying 192.168.1.79...
* TCP_NODELAY set
* Connected to 192.168.1.79 (192.168.1.79) port 80 (#0)
> GET /bar HTTP/1.1
> Host: msolo.carpnet.local
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Location: https://msolo.carpnet.local/bar/foo1
< Content-Length: 0
< Date: Thu, 23 Feb 2017 12:44:02 GMT
< Server: lighttpd
<
* Curl_http_done: called premature == 0
* Connection #0 to host 192.168.1.79 left intact

Hitting lighttpd directly, lighttpd 1.4.45, mod_extforward still configured but not relevant

[carpii@imac:~/lighty]$ curl -v -H "Host: msolo.carpnet.local" http://192.168.1.79:8000/bar
*   Trying 192.168.1.79...
* TCP_NODELAY set
* Connected to 192.168.1.79 (192.168.1.79) port 8000 (#0)
> GET /bar HTTP/1.1
> Host: msolo.carpnet.local
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Location: https://msolo.carpnet.local/bar/foo1
< Content-Length: 0
< Date: Thu, 23 Feb 2017 12:38:59 GMT
< Server: lighttpd
<
* Curl_http_done: called premature == 0
* Connection #0 to host 192.168.1.79 left intact

RE: regression in regexp vars v1.4.40 - Added by gstrauss over 2 years ago

For your use case, please try putting mod_extforward in server.modules before mod_rewrite and mod_redirect.

RE: regression in regexp vars v1.4.40 - Added by gstrauss over 2 years ago

Able to reproduce error with mod_extforward. Still looking into why.

module load order of mod_extforward with mod_redirect does not make a difference, though mod_extforward order with mod_rewrite can make a difference, but not in this case.

RE: regression in regexp vars v1.4.40 - Added by gstrauss over 2 years ago

mod_extforward is the only module currently using config_cond_cache_reset_item() and the only consumer of the extra layer of cached condition result in caches[dc->context_ndx].local_result. Removing this apparent over-optimization (part of ad65603e) fixes the problem; the bug is not with mod_extforward, but in configfile-glue.c. I'll look to see if I can fix the additional result caching, or else will remove it.

Please file a bug at https://redmine.lighttpd.net/projects/lighttpd/issues/new

Aside: in the future, please include your lighttpd.conf when reporting issues, as you can see its absence resulted in some days of delay before identifying the problem.

RE: regression in regexp vars v1.4.40 - Added by gstrauss over 2 years ago

--- a/src/configfile-glue.c
+++ b/src/configfile-glue.c
@@ -603,8 +603,14 @@ static void config_cond_clear_node(server *srv, connection *con, data_config *dc
        if (con->cond_cache[dc->context_ndx].result != COND_RESULT_UNSET) {
                size_t i;

-               con->cond_cache[dc->context_ndx].patterncount = 0;
-               con->cond_cache[dc->context_ndx].comp_value = NULL;
+               switch (con->cond_cache[dc->context_ndx].local_result) {
+               case COND_RESULT_TRUE:
+               case COND_RESULT_FALSE:
+                       break;
+               default:
+                       con->cond_cache[dc->context_ndx].patterncount = 0;
+                       con->cond_cache[dc->context_ndx].comp_value = NULL;
+               }
                con->cond_cache[dc->context_ndx].result = COND_RESULT_UNSET;

                for (i = 0; i < dc->children.used; ++i) {

RE: regression in regexp vars v1.4.40 - Added by carpii over 2 years ago

Thankyou! Will test this patch shortly

I've created a bug report at https://redmine.lighttpd.net/issues/2794

RE: regression in regexp vars v1.4.40 - Added by carpii over 2 years ago

Confirmed as working, thankyou.

I've sent another PayPal donation to support lighty :-)

Aside: in the future, please include your lighttpd.conf when reporting issues, as you can see its absence resulted in some days of delay before identifying the problem.

No probs

RE: [Solved] regression in regexp vars v1.4.40 - Added by gstrauss over 2 years ago

Thank you very kindly for your support!

    (1-15/15)