Lighttpd

Sounds like a bug. Thanks for the thorough report. I'll have a chance to look at this more closely on Wed.

So far, I have been unable to reproduce what you are seeing. I tried a simple lighttpd.conf:

server.bind = "localhost" 
server.port = 8080
server.document-root = "/tmp" 
server.modules += ( "mod_redirect")
$HTTP["host"] =~ ".*" {
    url.redirect = (".*" => "https://%0$0/foo")
}

< HTTP/1.1 301 Moved Permanently
< Location: https://localhost:8080/bar/foo
< ...

@carpii, please check that your curl command is sending a "Host" request header, or else %0 will match the empty string "", which may be what you're seeing. Also, would you please provide your lighttpd.conf (output of lighttpd -p -f /etc/lighttpd/lighttpd.conf) ?

A have a hunch this is DNS-related.

Try this with more recent lighttpd to force curl to send 'Host' request header (and you can change that value to verify what is returned)

Thanks, I suspect you are right about it being Host header related.
My dev box does serve various subdomains from .carpnet.local but I assumed curl sent a Host header automatically

I will dig into this fully in the next day or two, and report back

$HTTP["scheme"] == "http" {
    $HTTP["host"] =~ ".+" {
        url.redirect = (".*" => "https://%0$0/foo")
    }
    else {
        url.redirect = (".*" => "https://msolo.carpnet.local$0/foo")
    }
}

You can also do something similar with a short snippet of lua code and mod_magnet.

Thanks for looking into this. I'm gradually homing in on the issue I think
I can confirm my curl requests were sending a Host header, and Ive confirmed lighty is receiving this header it in all cases (I added it to access log format).

What I overlooked, is the fact Im actually accessing lighttpd via haproxy
Haproxy does little more than provide https termination infront of lighty, and also appends a X-Forwarded-Proto header ("http" or "https").
Now, If I curl lighttpd directly (I have it running on port 8000), then all appears to be working, and this corresponds with what you're seeing.

In order for lighty to pick up the X-Forwarded-Proto header from haproxy, and use $HTTP["scheme"] based rules, I am using mod_extforward.
This appears to be where the issue is.

Whenever a request goes through haproxy, and lighty is using mod_extforward, then the regexp vars appear to be getting mangled somehow (even when the Host header is present).
I should have realised this was related a lot sooner, as I tidied up the mod_extforward wiki page a few days ago.

There was some sort of bugfix to mod_extforward in v1.4.40 and I think this has caused the regression.
I cant find the actual revision at the moment, I'll keep looking..

Found it.. I think it may be related to this

https://redmine.lighttpd.net/issues/1499

I've simplified my haproxy and lighty configs as much as I can to demonstrate the problem...

https://gist.github.com/carpii/63b98744216e05c5768e388f48a0a33e

Hitting haproxy, lighttpd 1.4.45, using mod_extforward

[carpii@imac:~/lighty]$ curl -v -H "Host: msolo.carpnet.local" http://192.168.1.79/bar
*   Trying 192.168.1.79...
* TCP_NODELAY set
* Connected to 192.168.1.79 (192.168.1.79) port 80 (#0)
> GET /bar HTTP/1.1
> Host: msolo.carpnet.local
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Location: https:///bar/foo1
< Content-Length: 0
< Date: Thu, 23 Feb 2017 12:38:54 GMT
< Server: lighttpd
<
* Curl_http_done: called premature == 0
* Connection #0 to host 192.168.1.79 left intact

Hitting haproxy, lighttpd 1.4.45, mod_extforward not loaded

[carpii@imac:~/lighty]$ curl -v -H "Host: msolo.carpnet.local" http://192.168.1.79/bar
*   Trying 192.168.1.79...
* TCP_NODELAY set
* Connected to 192.168.1.79 (192.168.1.79) port 80 (#0)
> GET /bar HTTP/1.1
> Host: msolo.carpnet.local
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Location: https://msolo.carpnet.local/bar/foo1
< Content-Length: 0
< Date: Thu, 23 Feb 2017 12:44:02 GMT
< Server: lighttpd
<
* Curl_http_done: called premature == 0
* Connection #0 to host 192.168.1.79 left intact

Hitting lighttpd directly, lighttpd 1.4.45, mod_extforward still configured but not relevant

[carpii@imac:~/lighty]$ curl -v -H "Host: msolo.carpnet.local" http://192.168.1.79:8000/bar
*   Trying 192.168.1.79...
* TCP_NODELAY set
* Connected to 192.168.1.79 (192.168.1.79) port 8000 (#0)
> GET /bar HTTP/1.1
> Host: msolo.carpnet.local
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Location: https://msolo.carpnet.local/bar/foo1
< Content-Length: 0
< Date: Thu, 23 Feb 2017 12:38:59 GMT
< Server: lighttpd
<
* Curl_http_done: called premature == 0
* Connection #0 to host 192.168.1.79 left intact

For your use case, please try putting mod_extforward in server.modules before mod_rewrite and mod_redirect.

Able to reproduce error with mod_extforward. Still looking into why.

module load order of mod_extforward with mod_redirect does not make a difference, though mod_extforward order with mod_rewrite can make a difference, but not in this case.

mod_extforward is the only module currently using config_cond_cache_reset_item() and the only consumer of the extra layer of cached condition result in caches[dc->context_ndx].local_result. Removing this apparent over-optimization (part of ad65603e) fixes the problem; the bug is not with mod_extforward, but in configfile-glue.c. I'll look to see if I can fix the additional result caching, or else will remove it.

Please file a bug at https://redmine.lighttpd.net/projects/lighttpd/issues/new

Aside: in the future, please include your lighttpd.conf when reporting issues, as you can see its absence resulted in some days of delay before identifying the problem.

--- a/src/configfile-glue.c
+++ b/src/configfile-glue.c
@@ -603,8 +603,14 @@ static void config_cond_clear_node(server *srv, connection *con, data_config *dc
        if (con->cond_cache[dc->context_ndx].result != COND_RESULT_UNSET) {
                size_t i;

-               con->cond_cache[dc->context_ndx].patterncount = 0;
-               con->cond_cache[dc->context_ndx].comp_value = NULL;
+               switch (con->cond_cache[dc->context_ndx].local_result) {
+               case COND_RESULT_TRUE:
+               case COND_RESULT_FALSE:
+                       break;
+               default:
+                       con->cond_cache[dc->context_ndx].patterncount = 0;
+                       con->cond_cache[dc->context_ndx].comp_value = NULL;
+               }
                con->cond_cache[dc->context_ndx].result = COND_RESULT_UNSET;

                for (i = 0; i < dc->children.used; ++i) {

Thankyou! Will test this patch shortly

I've created a bug report at https://redmine.lighttpd.net/issues/2794

Confirmed as working, thankyou.

I've sent another PayPal donation to support lighty :-)

Aside: in the future, please include your lighttpd.conf when reporting issues, as you can see its absence resulted in some days of delay before identifying the problem.

No probs

Thank you very kindly for your support!