Lighttpd

Thanks for testing it out. Looks like a logic bug in that part of the config parsing and validation. I'll push a patch shortly.

However, the value of wstunnel.origins = ( "http://example.com" ) must not contain an empty string and probably should contain a URI if that is what is sent in the Origin: header.

(Documentation for this new module will be added soonish)

Pulled your changes and now the "wstunnel.origins" error is fixed.

Anyway I'm still not able to setup a working configuration, so I'll wait for sample config files.

Maybe you can post what configs you tried? (For VNC as well)

@gstrauss: thanks for your time.

I'll try to briefly explain what I'm trying to do.

I'm integrating noVNC (http://novnc.com) into an existing web interface for an embedded system.
Actually users can access the existing VNC server on port 5900, but I'd like to let them access it through noVNC: no VNC client, only browser, access control, ...

First step was to install websockify (a WebSocket<-->TCP bridge/proxy, https://github.com/novnc/websockify) on the host and configure it to proxy port 5901 to 127.0.0.1:5900.
noVNC opens a websocket connection to ws://hostname:5901/websockify and everything works.

Next step now is to avoid the extra 5900/5901 ports and let everything work on HTTP port 80.
I don't know if WebSocket<-->TCP bridging could be done by lighttpd itself; this should be great because it will let me remove the extra websockify server.

AFAIK a websocket connection starts as an HTTP one, then it requires an "upgrade" to a websocket.
What I'd like to do, if I well understood how it works, is to let lighttpd serve all of the port 80 traffic and "proxy" requests to /websockify to websockify proxy server.

This is an excerpt of my current config:

server.port = 80

server.modules = (
    "mod_rewrite",
    "mod_redirect",
    "mod_openssl",
    "mod_wstunnel",
    "mod_fastcgi" 
)

fastcgi.debug = 1
fastcgi.server = (
    ".php" => ((                                      
         "bin-path" => "/web/bin/php-cgi",
         "socket" => "/web/var/tmp/php.socket",
         "max-procs" => 1,
         "bin-environment" => (
           "PHP_FCGI_CHILDREN" => "5",
           "PHP_FCGI_MAX_REQUESTS" => "1000" 
         ),
         "broken-scriptfilename" => "enable" 
    ))
)
url.rewrite-if-not-file = (
    "^/$" => "$0",
    "^/(?!.+\.php)[^\?]+(\?.*)?" => "app.php/$1$2",
)

wstunnel.origins = ( "http://hostname" )
wstunnel.server = (
    "/websockify" => ((
        "host" => "127.0.0.1",
        "port" => "5901",  # proxy to websockify bridge
        "proto" => "websocket",
        "type" => "binary" 
    ))
)
wstunnel.debug = 4

This is lighttpd error log when noVNC client connects (unsuccessfully):

2017-09-27 13:48:09: (lighttpd-1.4.46-git/src/gw_backend.c.995) proc: tcp:127.0.0.1:5901 0 0 0 0
2017-09-27 13:48:10: (lighttpd-1.4.46-git/src/gw_backend.c.995) proc: tcp:127.0.0.1:5901 0 0 0 0
2017-09-27 13:48:11: (lighttpd-1.4.46-git/src/gw_backend.c.995) proc: tcp:127.0.0.1:5901 0 0 0 0
2017-09-27 13:48:12: (lighttpd-1.4.46-git/src/gw_backend.c.933) gw - found a host 127.0.0.1 5901
2017-09-27 13:48:12: (lighttpd-1.4.46-git/src/mod_wstunnel.c.452) http://hostname matches allowed origin: http://hostname
2017-09-27 13:48:12: (lighttpd-1.4.46-git/src/mod_wstunnel.c.514) WebSocket Version = 13
2017-09-27 13:48:12: (lighttpd-1.4.46-git/src/mod_wstunnel.c.539) will recv text data from backend
2017-09-27 13:48:12: (lighttpd-1.4.46-git/src/gw_backend.c.972) connect delayed; will continue later: tcp:127.0.0.1:5901
2017-09-27 13:48:12: (lighttpd-1.4.46-git/src/gw_backend.c.234) got proc: pid: 0 socket: tcp:127.0.0.1:5901 load: 1
2017-09-27 13:48:12: (lighttpd-1.4.46-git/src/gw_backend.c.995) proc: tcp:127.0.0.1:5901 0 0 1 0
2017-09-27 13:48:13: (lighttpd-1.4.46-git/src/gw_backend.c.995) proc: tcp:127.0.0.1:5901 0 0 1 0
... last line repeated ...

and this is websockify log:

# ./websockify 5901 127.0.0.1:5900
Waiting for connections on :5901
  0: got client connection from 127.0.0.1
  0: forking handler process
  0: ignoring empty handshake
  0: No connection after handshake
  0: handler exit

It seems that the connection won't go after the Upgrade request...

If you are using Docs_ModWStunnel, then you should not use the websockify layer.
If you are using Docs_ModProxy, with proxy.header = ( "upgrade" => "enable" ) then you do need to use the websockify layer.
(You need to be using lighttpd git master for either of the above, as these are experimental additions that will be part of lighttpd 1.4.46)

(I updated those docs this past weekend. Please have a look.)

Still not able to make it work, sources updated to latest commit (a156fdbc7bb00f1ead9df41038062efd7b829105).

If you are using Docs_ModWStunnel, then you should not use the websockify layer.

Great, that's my use case.
All HTTP requests must be managed by PHP/fastcgi except the ones to /websockify URL that must be bridged/proxied to local VNC 5900 port, without the websockify proxy layer.

I'm actually getting a segmentation fault with this config; I don't know what's wrong with it (anyway it shouldn't lead to a segfault):

$HTTP["url"] =~ "^/websockify" {
    wstunnel.debug = 65535
    wstunnel.frame-type = "binary" 
    wstunnel.server = (
        "" => ((
            "host" => "127.0.0.1",
            "port" => "5900", 
            #"proto" => "websocket",
            #"type" => "binary" 
        ))
    )
}

fastcgi.server = (
    ".php" => ((                                      
         "bin-path" => "/web/bin/php-cgi",
         "socket" => "/web/var/tmp/php.socket",
         "max-procs" => 1,
         "bin-environment" => (
           "PHP_FCGI_CHILDREN" => "5",
           "PHP_FCGI_MAX_REQUESTS" => "1000" 
         ),
         "broken-scriptfilename" => "enable" 
    ))
)

url.rewrite-if-not-file = (
    "^/$" => "$0",
    "^/(?!.+\.php)[^\?]+(\?.*)?" => "app.php/$1$2",
)

Sorry, I don't have gdb or valgrind on target machine (it's an embedded system) so I tried to debug the segfault point using... ehm, printf() ;)
This is what I got (filename:functionName:line):

mod_wstunnel.c:mod_wstunnel_check_extension():577
gw_backend.c:gw_check_extension():2187
gw_backend.c:gw_check_extension():2216
gw_backend.c:gw_check_extension():2223
*** segmentation fault because exts==NULL ***

Hope it could shed some light ;)

Thanks. I'll check why the array is NULL in a follow-up, because it should exist since these arrays should be created at startup when reading config.

diff --git a/src/gw_backend.c b/src/gw_backend.c
index 2cf63cc..2ed122d 100644
--- a/src/gw_backend.c
+++ b/src/gw_backend.c
@@ -2220,7 +2220,7 @@ handler_t gw_check_extension(server *srv, connection *con, gw_plugin_data *p, in
             exts = p->conf.exts_resp;
         }

-        if (0 == exts->used) continue;
+        if (NULL == exts || 0 == exts->used) continue;

         /* gw.map-extensions maps extensions to existing gw.server entries
          *

I applied your patch and it fixes the SEGFAULT, but noVNC still not working.
This is what I see now in error log, from client connection to failure:

2017-09-29 13:53:41: (/src/lighttpd-1.4.46-git/src/gw_backend.c.933) gw - found a host  0
2017-09-29 13:53:41: (/src/lighttpd-1.4.46-git/src/gw_backend.c.234) got proc: pid: 4293 socket: unix:/web/var/tmp/php.socket-0 load: 1
2017-09-29 13:53:43: (/src/lighttpd-1.4.46-git/src/gw_backend.c.308) released proc: pid: 4293 socket: unix:/web/var/tmp/php.socket-0 load: 0
2017-09-29 13:53:44: (/src/lighttpd-1.4.46-git/src/gw_backend.c.933) gw - found a host 127.0.0.1 5900
2017-09-29 13:53:44: (/src/lighttpd-1.4.46-git/src/mod_wstunnel.c.428) allowed origins not specified
2017-09-29 13:53:44: (/src/lighttpd-1.4.46-git/src/mod_wstunnel.c.514) WebSocket Version = 13
2017-09-29 13:53:44: (/src/lighttpd-1.4.46-git/src/mod_wstunnel.c.532) will recv binary data from backend
2017-09-29 13:53:44: (/src/lighttpd-1.4.46-git/src/gw_backend.c.972) connect delayed; will continue later: tcp:127.0.0.1:5900
2017-09-29 13:53:44: (/src/lighttpd-1.4.46-git/src/gw_backend.c.234) got proc: pid: 0 socket: tcp:127.0.0.1:5900 load: 2
2017-09-29 13:53:44: (/src/lighttpd-1.4.46-git/src/mod_wstunnel.c.842) send handshake response
2017-09-29 13:53:45: (/src/lighttpd-1.4.46-git/src/mod_wstunnel.c.341) recv data from backend ( fd = 15 ), size = 0x0c
2017-09-29 13:53:45: (/src/lighttpd-1.4.46-git/src/mod_wstunnel.c.1083) type = binary
2017-09-29 13:53:45: (/src/lighttpd-1.4.46-git/src/mod_wstunnel.c.1100) payload size = 0x0c
2017-09-29 13:53:45: (/src/lighttpd-1.4.46-git/src/mod_wstunnel.c.1128) send data to client ( fd = 14 ), frame size = 0x0f
2017-09-29 13:53:48: (/src/lighttpd-1.4.46-git/src/mod_wstunnel.c.600) timeout client ( fd = 17 )
2017-09-29 13:53:48: (/src/lighttpd-1.4.46-git/src/mod_wstunnel.c.1096) type = close
2017-09-29 13:53:48: (/src/lighttpd-1.4.46-git/src/mod_wstunnel.c.1100) payload size = 0x00
2017-09-29 13:53:48: (/src/lighttpd-1.4.46-git/src/mod_wstunnel.c.1128) send data to client ( fd = 17 ), frame size = 0x03
2017-09-29 13:53:48: (/src/lighttpd-1.4.46-git/src/gw_backend.c.308) released proc: pid: 0 socket: tcp:127.0.0.1:5900 load: 1
2017-09-29 13:54:04: (/src/lighttpd-1.4.46-git/src/mod_wstunnel.c.1087) type = ping
2017-09-29 13:54:04: (/src/lighttpd-1.4.46-git/src/mod_wstunnel.c.1100) payload size = 0x04
2017-09-29 13:54:04: (/src/lighttpd-1.4.46-git/src/mod_wstunnel.c.1128) send data to client ( fd = 14 ), frame size = 0x07
2017-09-29 13:54:25: (/src/lighttpd-1.4.46-git/src/mod_wstunnel.c.1087) type = ping
2017-09-29 13:54:25: (/src/lighttpd-1.4.46-git/src/mod_wstunnel.c.1100) payload size = 0x04
2017-09-29 13:54:25: (/src/lighttpd-1.4.46-git/src/mod_wstunnel.c.1128) send data to client ( fd = 14 ), frame size = 0x07

2017-09-29 13:53:48: (/src/lighttpd-1.4.46-git/src/mod_wstunnel.c.600) timeout client ( fd = 17 )

The default configuration is server.max-read-idle = 60. Did you change this to 3 seconds?
Please see the comments at the top of mod_wstunnel.c. It includes such things as;
" * - directive websocket.timeout should be replaced with server.max-read-idle"
You probably read documentation elsewhere for wstunnel.timeout, before I wrote Docs_ModWStunnel

Did you share your whole lighttpd.conf? (lighttpd -p -f /etc/lighttpd/lighttpd.conf (and replace any sensitive data with XXXXX))
Did you share your VNC configuration?

Sorry for the late reply, but I'm still not receiving redmine notifications (now I've tried switching to another mailbox...).

I've set server.max-read-idle = 3 but still can't bridge noVNC to local VNC server.

Attached here you'll find my lighttpd.conf (edited).
As for the VNC server I have no config file (it's an embedded system with static config).
It listens for connections on port 5900 and is protected with plain VNC password; it actually works fine with any (external) VNC client I've tried and with websockify.

Thanks for trying out the new feature (mod_wstunnel) and reporting your issues.

There were some things that needed to be fixed in the new mod_wstunnel for VNC to work. I'll push the updates tomorrow.

I installed x11vnc and novnc on my machine and ran x11vnc -localhost -display :0
novnc installed into /usr/share/novnc and I pointed my webserver document root there.
novnc defaults to using the /websockify path, so I used the following in my lighttpd.conf

server.document-root = "/usr/share/novnc" 
server.indexfiles = ("index.html")
server.modules += ( "mod_wstunnel" )
$HTTP["url"] =~ "^/websockify" {
    wstunnel.server = ( "" => ( ( "host" => "127.0.0.1", "port" => "5900" ) ) )
    # (optional, but recommended settings with noVNC)
    wstunnel.frame-type = "binary" 
    server.stream-request-body  = 2
    server.stream-response-body = 2
}
# (minimal mimetype.assign for use with noVNC)
mimetype.assign           = (
  ".html"         =>      "text/html",
  ".css"          =>      "text/css",
  # make the default mime type application/octet-stream
  ""              =>      "application/octet-stream",
)

One more thing: I recommend leaving server.max-read-idle alone and not setting it to 3 seconds unless you test that setting out and that behavior is what you really want.

Great, thanks for working on it.
Will look forward for your commits.

PS: still not receiving redmine notifications, don't know what to do; I'm forced to manually poll this page.
Is there any admin I could contact?

@nicorac: you probably need to click "watch" at the top of this page to subscribe to the topic. AFAICT (looking at the database table) you didn't subscribe so far.

@stbuehler: thanks for your help.
I double checked it and it's set, as you can see from the attached screens.
I also tried changing email adddress from @gmail.com to @yahoo.com.
Feel free to PM me so we won't clutter this thread.

FYI: 513e407b was pushed yesterday. Please test your noVNC setup with lighttpd again.

Commit 513e407b worked at first shot, thank you very much!

Next step now is try to make it work with https.

I've added these config lines to lighttpd.conf:

$SERVER["socket"] == "0.0.0.0:443" {
    ssl.pemfile = "/etc/mycert.pem" 
    ssl.engine = "enable" 
}

My PHP application works correctly so I suppose that https: is up and running.
But noVnc does not connect on 443; debugging it with Firefox debugger I see that the websocket connection stops with HTTP error 101: switching protocols.

NOTE: the issue is not a self-signed certificate (see here https://github.com/novnc/websockify/wiki/Encrypted-Connections) because I've already accepted the self-signed certificate in https: and, also, Firefox debugger shows that the websocket connection was accepted but not "upgraded".

I'm attaching the updated configuration file.
Is there something wrong/missing?

PS: config file actually does not have these parameters

server.stream-request-body = 2
server.stream-response-body = 2

Slow but steady. Thanks for your persistence. I don't have time this evening to look further, but the next thing that I would like to try would be to set up SSL and to try with mod_proxy (and websockify running on port 6080, the default, and x11vnc running on port 5900, the default). I had tested that without SSL (and it worked) in lighttpd git master before focusing on the mod_wstunnel commits above. I would be interested if this config works with SSL, to narrow down whether this is something in mod_wstunnel, or something deeper.

server.modules += ( "mod_proxy" )
$HTTP["url"] =~ "^/websockify" {
    proxy.server = ( "" => ( ( "host" => "127.0.0.1", "port" => "6080" ) ) )
    proxy.header = ( "upgrade" => "enable" )
}

Slow but steady. Thanks for your persistence.

Thank you for your active help.

I've tried the suggested mod_proxy configuration and it works perfectly.

It only have one downside: Firefox won't allow wss:// connections to hostname:6080 because of the self-signed certificate I'm using that has to be accepted as an exception by the browser.
Firefox, like others, stores its exceptions using the hostname:port tuple so hostname:443 (already accepted at first web application usage) and hostname:6080 are two different sites.
As a workaround it's possible to open https://hostname:6080/websockify URL in the browser and accept the proposed self-signed certificate, but this is cumbersome and difficult for a mid-low end user.

• it works on the same 443 port as https://, so the certificate has already been accepted (if needed)
• it does not require the extra websockify layer

PS: @stbuehler: I'm actually receiving the notifications for this issue, thanks for fixing it.

Useful info. mod_proxy and mod_wstunnel share quite a bit of underlying infrastructure, so this helps me in where to look next: mod_wstunnel, but alas, this will have to wait until after the workday.

Don't worry, take your time ;)

In my testing, first I had to accept a self-signed certificate permanently in Firefox, since temporarily accepting the certificate for the https URL was not sufficient for use by wss scheme in Firefox.

Then, when I connected to lighttpd server via port 8443 (testing port running SSL), both mod_wstunnel and mod_proxy received a disconnect from client after RFB version number was sent. The reason eludes me at the moment, and a quick look in noVNC console debugging did not help (I appended "?logging=debug" to the noVNC URL)

Did the above mod_proxy configuration work for you even when client connected via https?

Ugh! What a pain to track down. Turns out the bug emanates from a long time ago: 3888c103 from 31 Oct 2005, over 10 years before I added the ability for lighttpd to perform bidirectional input and output. The historical commit became stale (and harmful) with the design change to bidirectional input and output.

Here is a quick-and-dirty patch, though I plan to try to find a more appropriate place for this code, or to determine that it would be better to remove it. I am leaning towards removing it. (The line numbers in the patch are not exact to current lighttpd git master since they are against a subsequent patch that I have not yet pushed.)

--- a/src/mod_openssl.c
+++ b/src/mod_openssl.c
@@ -1193,17 +1193,9 @@ static int
 connection_write_cq_ssl (server *srv, connection *con,
                          chunkqueue *cq, off_t max_bytes)
 {
-    /* the remote side closed the connection before without shutdown request
-     * - IE
-     * - wget
-     * if keep-alive is disabled */
     handler_ctx *hctx = con->plugin_ctx[plugin_data_singleton->id];
     SSL *ssl = hctx->ssl;

-    if (con->keep_alive == 0) {
-        SSL_set_shutdown(ssl, SSL_RECEIVED_SHUTDOWN);
-    }
-
     chunkqueue_remove_finished_chunks(cq);

     while (max_bytes > 0 && NULL != cq->first) {

Just pulled commit fa1eef00 and I've seen that the patch seems to be unrequired on that commit.
Built and tested it without patch: it works, no issues, great!

I plan to try to find a more appropriate place for this code, or to determine that it would be better to remove it

I can't understand the underlying issues it could involve, anyway I can go on with my application testing and will check this issue for final fix.
Thanks again.

PS: just a small quirk, server.upload-dirs config param does not support variable expansion anymore; I had to change

var.home_dir = "/web" 
server.upload-dirs = ( home_dir + "/var/tmp" )

server.upload-dirs = ( "/web/var/tmp" )

Thanks for flagging that. It's a regression. Code was added in bd77abe0 to more strictly check the contents of arrays and give better error messages when parsing the config file. Unfortunately, when combining items, the config parser (not modified in that commit) reuses a piece of data which makes array_is_vlist() fail for items expected to be value lists (as opposed to key => value lists) This likely affects all code using array_is_vlist(). I have to dig into the parser.

This should fix the regression.

--- a/src/configparser.y
+++ b/src/configparser.y
@@ -50,7 +50,9 @@ static data_unset *configparser_get_variable(config_t *ctx, const buffer *key) {
     array_print(dc->value, 0);
 #endif
     if (NULL != (du = array_get_element_klen(dc->value, CONST_BUF_LEN(key)))) {
-      return du->copy(du);
+      du = du->copy(du);
+      buffer_reset(du->key);
+      return du;
     }
   }
   return NULL;