Lighttpd

The perfect solution would be something like dynamic reverse proxying, where the proxied host is derived from the URL that is called; so, if I call http://lighttpd.ip/machine/host1, it will proxy ws://host1:5000; if I call http://lighttpd.ip/machine/host2, it will proxy ws://host2:5000; etc.

Something like that is possible, but not precisely what you have requested.

While I do not know your reasoning behind why you have chosen to statically route to specific machines, I would suggest re-mapping that to lighttpd mod_proxy load balancing options mod_proxy proxy.balance to see if one of those options meets your needs. The lighttpd proxy.balance = "hash" option will balance based on hash of the URL, and "sticky" will balance based on the remote IP address (of the client). If lighttpd tries to balance to a host that is down, lighttpd will try again on another host if the connect() fails.

Regarding adding and removing machine: for a high availability configuration, when you remove a machine, another machine should could pick up the IP address of the removed machine to avoid excess downtime.

If you want to add and remove machines dynamically to connection pools, then I suggest putting something like HAProxy in front of lighttpd.

Hi gstrauss,

thanks for your response!
To shed some light what on what I am trying to do. This is not about load balancing.
I basically have a test pool of different IoT devices in an isolated lab network, running a websocket application. This fleet of different devices is available to developers for testing etc. only via the machine lightttpd is running on as a jumphost (IT won't allow routing between the isolated lab network and the office network, but reverse proxying is OK, whatever...).
When developers want to test something, they want to test it on one specific device (specific model, FW version etc.), so I that's why I need a static setup of the proxy hosts here. So this is not really about high availability or performance, it is just supposed to make a lot of different devices available to developers they otherwise cannot access.

So, currently I generate the lighttpd.conf (or better, an include-snippet) with all the proxy sections in a script whenenver there is a change, and then restart lighttpd. Not a big deal, works fine, but it seems kind of clumsy, and I was wondering if there is a better, more elegant way to do this without having to continously modify the config.

BTW, I first tried this with nginx and could not get the websocket proxying working at all. Seems the nginx version I have available (the jumphost machine is hosted by IT and I am limited in what I am allowed to install there...) does not support this. With lighttpd it worked out-of-the box, so that was the way to go for me.

Anyway, many thanks for your pointers. I will look into the balancing options to see if I can "creatively" use those to achieve what I need...

By design as a security choice, lighttpd does not make outbound connections (after potentially doing optional DNS lookups only at lighttpd startup to resolve DNS names in lighttpd.conf). lighttpd will run perfectly well in a network where IT configures the network to block or filter outbound connections. Therefore, lighttpd is not designed for that perfect solution you have in mind to use lighttpd to work around your IT network routing restrictions. (Yes, I know that DNS resolution is not necessarily an outbound connection, but it might be and lighttpd does not have insight into the network topology.)

Sounds like your IT dept will allow an application-level proxy (e.g. lighttpd). If you have more control inside the development lab, you might consider lighttpd on the jump host to reverse proxy all traffic to another jump box inside the development lab network which runs another instance of lighttpd, or might run HAProxy, to route based on the URL. You may have more leeway on how to manage your service on the second jump box inside the development lab network.

Hi gstrauss,

thanks for taking the time to explain. I was considering the approach with a second jumphost in the lab net, but figured that maybe I can do without that. But seems like that will be the way to go. In the lab network, I have more leeway of what I can and can't do.

Thanks again!

For kicks, consider this kludge:
Preconfigure lighttpd with lots of /2 /3 ... /100 (or whatever labels you like)

  $HTTP["url"] =~ "^/2/1/" {
    proxy.header = (
      "map-urlpath" => ( "/2/1" => "/" ),
      "upgrade" => "enable" 
    )
    proxy.server = ( "" => (( "host" => "192.168.111.2", "port" => "5000" )))
  }
  else $HTTP["url"] =~ "^/3/1/" {
    proxy.header = (
      "map-urlpath" => ( "/3/1" => "/" ),
      "upgrade" => "enable" 
    )
    proxy.server = ( "" => (( "host" => "192.168.111.3", "port" => "5000" )))
  }
  else $HTTP["url"] =~ "^/4/1/" {
    proxy.header = (
      "map-urlpath" => ( "/4/1" => "/" ),
      "upgrade" => "enable" 
    )
    proxy.server = ( "" => (( "host" => "192.168.111.4", "port" => "5000" )))
  }

and then use lighttpd mod_magnet to run a custom lua script in which you rewrite /machine2 to /2 (to match the above mapping). lighttpd mod_magnet dynamically reloads your custom lua script when the modification time on the file changes. That might give you the "dynamic" updating you seek, as long as you have previously allocated the IP mappings to point to IP addresses of devices inside your development lab. Use lighttpd mod_magnet to rewrite the URL and then lighttpd mod_proxy will proxy back to the chosen IP based on preconfigured prefix you mapped to IPs.

This might be another way to write the above (untested, both above and below):

  proxy.header = (
    "upgrade" => "enable",
    "map-urlpath" => (
       "/2/1" => "/",
       "/3/1" => "/",
       "/4/1" => "/",
    ),
  )
  proxy.server = (
    "/2/1" => (( "host" => "192.168.111.2", "port" => "5000" )),
    "/3/1" => (( "host" => "192.168.111.3", "port" => "5000" )),
    "/4/1" => (( "host" => "192.168.111.4", "port" => "5000" )),
  )