Project

General

Profile

[Solved] Reverse proxy with Jenkins

Added by hectorvido over 1 year ago

Hi guys,

I am trying to configure Lighttpd + Jenkins, is working fine but Jenkins keeps showing a message telling me there is a problem in the reverse proxy configuration as you can take more details in this page.
This is a Debian 11 with Lighttpd 1.4.59.

I tested with an Nginx and this configuration and Jenkins stopped to complain, then I analyze the headers of head request and noticed that Lighttpd is sending the same Nginx headers and some more.

Lighttpd

GET / HTTP/1.1
Host: jenkins.example.com
user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
accept-language: en-US,en;q=0.5
accept-encoding: gzip, deflate, br
dnt: 1
upgrade-insecure-requests: 1
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: none
sec-fetch-user: ?1
te: trailers
X-Host: jenkins.example.com
X-Forwarded-Host: jenkins.example.com
Forwarded: for=191.193.46.68;proto=https
X-Forwarded-For: 191.193.46.68
X-Forwarded-Proto: https
Connection: close

Nginx

GET /manage HTTP/1.1
Connection: close
Host: jenkins.example.com
X-Forwarded-For: 191.193.46.68
X-Forwarded-Proto: https
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Upgrade-Insecure-Requests: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?

I started to think that maybe header X-Forwarded-Host or the extra ones in Lighttpd could be causing this issue. The I tried to remove the headers but it did not work, the headers don't change at all.
Here is my proxy configuration:

server.modules   += ( "mod_proxy", "mod_setenv" )

$HTTP["host"] == "jenkins.example.com" {
        proxy.balance = "hash" 
        proxy.server  = ( "" => ( ( "host" => "127.0.0.1", "port" => "8080" ) ) )
        proxy.forwarded = ( "for" => 1, "proto" => 1, "host" => 0, "by" => 0, "remote_user" => 0 )
        setenv.set-request-header = ( "X-Host" => "", "X-Forwarded-Host" => "" )
}

Is there a way to remove these headers?

Thanks in advance!

Best,


Replies (5)

RE: Reverse proxy with Jenkins - Added by gstrauss over 1 year ago

Is there a way to remove these headers?

No, not currently. However, you also have not established that this is a problem. You're guessing.

Also, the information in those headers is consistent and seemingly correct for
https://www.jenkins.io/doc/book/system-administration/reverse-proxy-configuration-troubleshooting/
As that page does not mention the Forwarded header, have you tried commenting out proxy.forwarded from your lighttpd config?

but Jenkins keeps showing a message telling me there is a problem in the reverse proxy configuration

How about sharing more concrete log messages?

This should be removed from your lighttpd.conf for jenkins.example.com:
setenv.set-request-header = ( "X-Host" => "", "X-Forwarded-Host" => "" )
Those headers are generated by lighttpd mod_proxy.

RE: Reverse proxy with Jenkins - Added by gstrauss over 1 year ago

If jenkins is not properly using the headers added by lighttpd mod_proxy, then you might try adding a configuration using proxy.header
proxy.header = ("map-host-response" => ("-" => "-"), "https-remap" => "enable")
If Jenkins uses websockets then you need to enable that in proxy.header by adding , "upgrade" => "enable" inside the above parens for proxy.header

RE: Reverse proxy with Jenkins - Added by hectorvido over 1 year ago

Hi gstrauss,

I was trying to remove the headers because they are the extra ones in comparison with ones from nginx, this is why I was using these config below:

setenv.set-request-header = ( "X-Host" => "", "X-Forwarded-Host" => "" )

What I understood of mod setenv page, particularly this phrase: "Set a blank value to remove request header or remove response header."

My line of thought was the emulation of nginx headers because they are working

I added this configuration proxy.header = ("map-host-response" => ("-" => "-"), "https-remap" => "enable") with and without the "upgrade" => "enable" but the result is the same.

I cannot find logs about this proxy behavior in Jenkins, but there is curl with a "Referrer Page" information we use to test, and maybe this can give some light. It should return a 200, but in Lighttpd is returning a 404. The output is weird here in the post, so I will attach these logs as files.

Thanks,

RE: Reverse proxy with Jenkins - Added by gstrauss over 1 year ago

It looks like jenkins is encoding https:// in the URI with "%2F%2F", and lighttpd default normalization may normalize it to a single "/"
Try this:

server.http-parseopts += ( "url-path-2f-decode" => "disable" )
server.modules += ( "mod_proxy" )
$HTTP["host"] == "jenkins.example.com" {
        proxy.balance = "hash" 
        proxy.server  = ( "" => ( ( "host" => "127.0.0.1", "port" => "8080" ) ) )
        #proxy.forwarded = ( "for" => 1, "proto" => 1, "host" => 0, "by" => 0, "remote_user" => 0 )
}

RE: Reverse proxy with Jenkins - Added by hectorvido over 1 year ago

gstrauss, you are a genius, it worked!

That url encoding in the curl response was invisible to my eyes.

I never paid attention to these configurations before, but there is a clear explanation in my default config file from Debian (I changed the configuration in this file):

# strict parsing and normalization of URL for consistency and security
# https://redmine.lighttpd.net/projects/lighttpd/wiki/Server_http-parseoptsDetails
# (might need to explicitly set "url-path-2f-decode" = "disable" 
#  if a specific application is encoding URLs inside url-path)
server.http-parseopts = (
  "header-strict"           => "enable",# default
  "host-strict"             => "enable",# default
  "host-normalize"          => "enable",# default
  "url-normalize-unreserved"=> "enable",# recommended highly
  "url-normalize-required"  => "enable",# recommended
  "url-ctrls-reject"        => "enable",# recommended
  "url-path-2f-decode"      => "disable",# recommended highly (unless breaks app, ex: Jenkins)
 #"url-path-2f-reject"      => "enable",
  "url-path-dotseg-remove"  => "enable",# recommended highly (unless breaks app)
 #"url-path-dotseg-reject"  => "enable",
 #"url-query-20-plus"       => "enable",# consistency in query string
)

I will try to add the lighttpd configuration in Jenkins examples page.

Thank you!

Best,

    (1-5/5)