Project

General

Profile

[Solved] Change header value in proxied requests

Added by the_error 12 months ago

lighttpd/1.4.67

Hi, after I did spent a lot of time, I've come here, just to confirm my concerns - it's probably not possible to manipulate of custom header value of proxied requests, right?

My use-case is, that I'm trying to use lighttpd as reverse proxy for WebDav server. It's all working nice, until you will try to move some file. In header (Destination) it will use HTTPS, however in real it suppose to be HTTP and it fails with this:

So, client generates this kind of request

curl -v -k -X MOVE --header 'Destination:https://dav.server.com/dav/7minuta.png' 'https://dav.server.com/dav/7minut.png'

but it ends with this:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<title>502 Bad Gateway</title>
</head><body>
<h1>Bad Gateway</h1>
<p>Destination URI refers to different scheme or port (https://hostname:443)
(want: http://hostname:5005)</p>
</body></html>

however almost identical, but with http in Destination will pass.

curl -v -k -X MOVE --header 'Destination:http://dav.server.com/dav/7minuta.png' 'https://dav.server.com/dav/7minut.png'

So, I need to override Destination header in order to make it work. In case of Apache, it's not big deal:

location / {
    # to avoid 502 Bad Gateway:
    # http://vanderwijk.info/Members/ivo/articles/ComplexSVNSetupFix
    set $destination $http_destination;

    if ($destination ~* ^https(.+)$) {
         set $destination http$1;
    }

    proxy_set_header Destination $destination;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $remote_addr;

    proxy_pass http://svn;
}

but my conclusion is, that in case of lighttpd this is not possible. Am I right? It was part of 1.5 and proxy_core, however looks it's depricated. :/

Thank you.


Replies (19)

RE: Change header value in proxied requests - Added by gstrauss 12 months ago

it's probably not possible to manipulate of custom header value of proxied requests, right?

100% wrong.

but my conclusion is, that in case of lighttpd this is not possible. Am I right?

No. You are not right. Did you bother trying to read lighttpd mod_proxy documentation?

There are multiple solutions, readily available and already described in the documentation.

mod_proxy proxy.header = ("https-remap" => "enable")
mod_magnet (arbitrary header customization)
mod_webdav lighttpd WebDAV implementation

RE: [Solved] Change header value in proxied requests - Added by the_error 12 months ago

Sure I did.

mod_proxy proxy.header https-remap is not working this way. It's just replacing host value and some other stuff, if I get it right (and even so, I've tried also this option, but it makes no difference), but maybe I just didn't get it right?
mod_magned actually may work, but I'm bit sceptical to learn to write some come as I'm afraid it will end as setenv.set-request-header, what have any affect as then it end as "not allowed" request, because it servers as normal web server. But, if you are sure, it will work, I'll give it try.

And about mod_webdav - it suppose to help me how? I dont' need to run another WebDav server. I need to gat access to one already running on local network via proxy running on router. I can't see in docs anyway of forwarding request using it to some another server.

RE: [Solved] Change header value in proxied requests - Added by the_error 12 months ago

actually, if mod_proxy proxy header can change some specific header, can you please show me how?

RE: [Solved] Change header value in proxied requests - Added by gstrauss 12 months ago

Did you bother trying to read lighttpd mod_proxy documentation?

        #"https-remap" => "enable",
            # For https requests from client, map https:// to http://
            # when map-host-request matches URI in request, and map http://
            # to https:// when map-host-response matches URI in response.

What do you think the sentence in the description means? It is a compound sentence. Did you read the whole sentence? What part(s) were unclear? What did you try? How to get support

RE: [Solved] Change header value in proxied requests - Added by gstrauss 12 months ago

can you please show me how?

Let me be blunt: no. Learn how to help yourself after you have been given some guidance.

Read the mod_proxy documentation. The answers are there. If you do not understand something specific in the documentation, ask a specific question and reference the specific documentation. Not only does it demonstrate that you are making an effort, but you are also much more likely to solve your issue or issues. Also, it might point to an area in the documentation that can be clarified for the benefit of others.

You shared this error.

<p>Destination URI refers to different scheme or port (https://hostname:443)
(want: http://hostname:5005)</p>

Read the mod_proxy documentation and try to solve that error.

RE: [Solved] Change header value in proxied requests - Added by the_error 12 months ago

Well, I've spend whole day of trying, the doc I've read 100 times complete and yes, this looks like it should help, but doesn't as it obviously manage another thing. This was actually first thing I've give a try. Please, do not say, that I didn't read or tried as I yesterday didn't nothing else, than trying and googling. It's quite common problem for WebDav behind proxy, but didn't find any solution for lighttpd. Only Apache and Nginx - all by rewriting the header, but it's not really clear how to achieve this on lighttpd. Based on description it maybe should work - but it doesn't. And I'm trying to figure out why, as I'm out of ideas.

That's why I've ask how to do, as your arrogant answer helps nobody and you are blaming me for something, what is NOT true. Actually, I'm not sure, if you anyhow read the whole context by yourself. The error there is because of invalid scheme in header, but I can't figure out, how that scheme override in that header. And surprisingly, https-remap just don't work, even that it looks like it should. So, let's assume I do have it just wrong, so how then it should looks like?

The only possible tricky part from my point could be SSL redirect, but that should not anyhow affect termination and proxying proces, or can it?

$HTTP["host"] == "dav.server.com" {

   $HTTP["scheme"] == "http" {
      $HTTP["url"] !~ "^/(.well-known|wtf)" {
         $HTTP["host"] =~ ".*" {
            url.redirect = (".*" => "https://%0$0")
         }
      }
   }

   server.document-root = "/www/dav/" 
   alias.url = ( "/" => "/www/dav/" )
   accesslog.filename = "/var/log/lighttpd/access-dav.log" 

#   proxy.debug = 1
#   debug.log-response-header    = "enable" 
#   debug.log-request-header-on-error = "enable" 
#   debug.log-condition-handling = "enable" 

# Fix stream chaching
   server.stream-response-body = 2

#   ssl.engine = "enable" #enabled elswhere
   ssl.pemfile = "/root/.acme.sh/dav.server.com/fullchain.cer" 
   ssl.ca-file = "/root/.acme.sh/da.server.com/ca.cer" 

   $HTTP["url"] !~ "^/(.well-known|wtf)" {

      proxy.server  = ( "" => (
         ( "host" => "192.168.1.3", "port" => 5005 )
      ))
      proxy.header = (
          "upgrade" => "enable",
        "https-remap" => "enable" 
      )
      proxy.forwarded = ( "for"          => 1,
                        "proto"        => 1,
                        "host"        => 1,
                        "by"          => 1
      )
   }

}

RE: [Solved] Change header value in proxied requests - Added by the_error 12 months ago

Hmm, and looks actually, that I'm not alone, who things the same, that it's not possible. Could even mod_magnet work?

https://serverfault.com/questions/608257/rewrite-location-response-header-in-lighttpd

RE: [Solved] Change header value in proxied requests - Added by the_error 12 months ago

But on other hand, it's bit different case. It's about response, and I do need request, but kind of similar case. Question is, how. I'm out of ideas.

RE: [Solved] Change header value in proxied requests - Added by the_error 12 months ago

And yet one my experiment, what failed as well:

$HTTP["Destination"] =~ ^https://(.+) {
    setenv.set-request-header += ("Destination" => "http://$1")
}

No matter where it was - before or after proxy, it won't work as well, it just returned not allowed request (i think it's not even pass it to WebDav, just tried to handle it's own, probably. Also didn't work, when I placed proxy inside this condition.

RE: [Solved] Change header value in proxied requests - Added by the_error 12 months ago

I mean

$REQUEST_HEADER["Destination"] =~ "^https://(.+)" {
    setenv.set-request-header += ("Destination" => "http://$1")
    proxy.server  = ( "" => (
         ( "host" => "192.168.1.3", "port" => 5005 )
    ))
}

I've copy it from some notes, where syntax were wrong. Anyway, this "workaround" ends with

< HTTP/1.1 405 Method Not Allowed
< Date: Wed, 26 Apr 2023 08:53:46 GMT
< Server: Apache
< Allow: GET,POST,OPTIONS,HEAD
< Content-Length: 221
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>405 Method Not Allowed</title>
</head><body>
<h1>Method Not Allowed</h1>
<p>The requested method MOVE is not allowed for this URL.</p>
</body></html>

RE: [Solved] Change header value in proxied requests - Added by gstrauss 12 months ago

Only in the most recent comments do you share your configuration, which you so obviously need help with. I can not fathom why you omitted it for so long since that is precisely the thing for which you are asking help.

With that information, I can see that you did not take my earlier guidance. I asked some very specific questions above in https://redmine.lighttpd.net/boards/2/topics/11003?r=11007#message-11007 It would appear that you are reading faster than you are comprehending. Try reading my prior comment again. Instead of reading the documentation 100 times and complaining, try reading the documentation more slowly and focus on the place to which I have pointed you.

RE: [Solved] Change header value in proxied requests - Added by gstrauss 12 months ago

curl -v -k -X MOVE --header 'Destination:https://dav.server.com/dav/7minuta.png' 'https://dav.server.com/dav/7minut.png'

Is your WebDAV client sending something similar, or is your WebDAV client more advanced and know that an absolute-URI is needed only when multiple independent WebDAV servers are involved (and which is not supported by all webdav servers). When the WebDAV server is the same server, MOVE can use the path-absolute part of the URI in Destination, e.g.
curl -v -k -X MOVE --header 'Destination:/dav/7minuta.png' 'https://dav.server.com/dav/7minut.png'

RE: [Solved] Change header value in proxied requests - Added by gstrauss 12 months ago

Your syntax for mod_setenv is incorrect and will never work. mod_setenv does not perform regex substitution "http://$1" (invalid)

Where in the documentation here did you see that syntax for mod_setenv? If you saw it on this site, I'd like to fix the doc.

RE: [Solved] Change header value in proxied requests - Added by the_error 12 months ago

Can't say, actually. I'm using ForkLift now as for testing it, but the issue related to Destination header is quite common case and you can find this all around, when googling for WebDav behind proxy. Ex.
https://stackoverflow.com/questions/1110710/webdav-behind-a-reverse-proxy
https://www.dimoulis.net/posts/webdav-behind-reverse-proxy/

If the https-remap is related (and need) to match of "map-host-response" and "map-host-request" what the hell value should I place there then. I'm really lost about that authority and host header. I thought it's related only to some host name change or so, I really don't know. I do missing some example of usage in doc for this. The description do not reveals mystery about for me. :)

RE: [Solved] Change header value in proxied requests - Added by gstrauss 12 months ago

If the https-remap is related (and need) to match of "map-host-response" and "map-host-request" what the hell value should I place there then. I'm really lost about that authority and host header. I thought it's related only to some host name change or so, I really don't know. I do missing some example of usage in doc for this. The description do not reveals mystery about for me. :)

You're getting warmer. You are starting to actually read and reference the documentation. This is a key life-lesson that will hopefully help you in the future, too.

Yes, you need to understand "map-host-request" better than you do now. What does the mod_proxy documentation say about "map-host-request"? What advice did I give above about asking specific questions?

I repeat again the error that you provided.

<p>Destination URI refers to different scheme or port (https://hostname:443)
(want: http://hostname:5005)</p>

RE: [Solved] Change header value in proxied requests - Added by the_error 12 months ago

those last posts were just "walking around problem" I've already tried and was, obviously wrong. :)

As I'm desperate, I've tried several other option mentioned in docs, but no luck as well. Are those actually needed, in order to make https-remap working, or it suppose to work it's own without them? As I do not need to remap anything, actually. Just those schema in Destination in header.

proxy.server  = ( "" => (
         ( "host" => "192.168.1.3", "port" => 5005 )
      ))
      proxy.header = (
          "upgrade" => "enable",
    "https-remap" => "enable",
        "map-host-request" => (
        "" => "-" 
    ),
    "map-host-response" => (
        "-" => "" 
    )
      )

      proxy.forwarded = ( "for"          => 1,
                        "proto"        => 1,
                        "host"        => 1,
                        "by"          => 1,
                        #"remote_user" => 1
      )

RE: [Solved] Change header value in proxied requests - Added by gstrauss 12 months ago

Are those actually needed, in order to make https-remap working, or it suppose to work it's own without them?

        #"https-remap" => "enable",
            # For https requests from client, map https:// to http://
            # when map-host-request matches URI in request, and map http://
            # to https:// when map-host-response matches URI in response.

The sentence reads: x happens when y matches URI ...
You seem to be having a lot of trouble reading and understanding this compound sentence. How would you recommend rewriting it to make it clearer?

As I do not need to remap anything, actually. Just those schema in Destination in header.

<p>Destination URI refers to different scheme or port (https://hostname:443)
(want: http://hostname:5005)</p>

Most people should be able to look at the above and see that the difference is more than "http" and "https".
You seem to need a lot of help with noticing glaringly obvious differences.

Your backend WebDAV server may require that you specify host mappings explicitly for hostname and port. This is generic:

proxy.header = (
  "upgrade" => "enable",
  "https-remap" => "enable",
  "map-host-request" => ("-" => "-"),
  "map-host-response" => ("-" => "-")
)

(You should not need "upgrade" => "enable" for WebDAV, but I included it here since it was in the config you shared)

RE: [Solved] Change header value in proxied requests - Added by the_error 12 months ago

OMG, it's working. OK, now I have to know WHY!

#"map-host-request" => (
     #"-" => "...",#replace provided given Host request authority

#"map-host-response" => (
     #"-" => "...",#replace authority used in backend request

Is the dash kind of always match "alias"? For me the most "strange" is actually the "..." as I do not have idea, what I want/need to place there and the dash really didn't come to my mind. Do I understand, correctly, that

"map-host-request" => (
     "-" => "192.168.1.3:5005")

"map-host-response" => (
     "-" => "dav.server.com:443" 
)

will actually work the "same" way?

RE: [Solved] Change header value in proxied requests - Added by gstrauss 12 months ago

The syntax in "map-host-request" and "map-host-response" was my own creation back in 2016 to provide this functionality in #152.
"..." means fill in with something. Most people are not confused by the meaning of the ellipsis. https://en.wikipedia.org/wiki/Ellipsis
"" is not a valid host name, and on the right side of "..." => "", the "" is treated as a "do not match and do not continue trying to match"
"-" is not a valid host name, and on the left side of "-" => "..." in "map-host-request" means match anything and replace with "..."
"-" on the left side of "map-host-response" means use the host that matched in "map-host-request", if there was a match in "map-host-request", for matching bidirectional mapping. For arbitrary rewrite of request or response headers, use mod_magnet.
"-" on the right side means use the original host in the request.

The following using explicit matches should work for you similarly to the generic example above.

"https-remap" => "enable",
"map-host-request"  => ("dav.server.com"   => "192.168.1.3:5005"),
"map-host-response" => ("192.168.1.3:5005" => "dav.server.com"  )

WebDAV can point to different servers (though that is typical seen only for very large installations with multiple different servers), and proxy.header applies to all use of mod_proxy, not only WebDAV. For both reasons and more, proxy.header mapping syntax must handle many different scenarios, not just your specific WebDAV use case.


Your WebDAV clients are poorly written. RFC 4918 was published in 2007 and allows path-absolute form in Destination, which would not have needed any mapping.

    (1-19/19)