Project

General

Profile

Actions

Bug #3240

closed

URL after hashmark invisible in lua

Added by flynn 2 months ago. Updated 2 months ago.

Status:
Invalid
Priority:
Normal
Category:
mod_magnet
Target version:
-
ASK QUESTIONS IN Forums:
No

Description

If a URL contains a hashmark, e.g. http://localhost/bla#blabla&test=1, everything after the hashmark # is not visible in any lua lighty variable.

I would expect it to see in at least in lighty.r.req_attr["request.uri"] and lighty.r.req_attr["request.orig-uri"], but both variables have the value /bla, but I expect bla#blabla&test=1.
I iterated over all variables in all lighty.r.* tables/userdata, none of them contain the full uri with hashmark.

If I replace the hashmark by a question sign, e.g http://localhost/bla?blabla&test=1, both variables have the expected value: /bla?blabla&test=1

Actions #1

Updated by gstrauss 2 months ago · Edited

  • Status changed from New to Invalid
  • Target version deleted (1.4.74)

Removing the fragment is intentional behavior in lighttpd (and I believe also in many other web servers).

The # fragment in a URI may be relevant to the client, but is not part of the target URI. lighttpd removes the fragment, if present, during initial request header processing and normalization. This happens to all requests parsed by lighttpd, before any module sees the parsed and normalized URL. (Not removing the fragment could possibly lead to security issues during regex matching, depending on how a lighttpd user wrote the regexes in lighttpd.conf.)

https://www.rfc-editor.org/rfc/rfc9110#name-determining-the-target-reso

A URI reference is resolved to its absolute form in order to obtain the "target URI". The target URI excludes the reference's fragment component, if any, since fragment identifiers are reserved for client-side processing ([URI], Section 3.5).

Actions #2

Updated by flynn 2 months ago · Edited

This means that there is no possibility to access/read the URL after the hashmark inside mod_magnet?

Background: I want to authenticate against authentik using openid and the callback of authentik looks like this:

/callback/openid#access_token=...&id_token=...&...

Actions #3

Updated by gstrauss 2 months ago

lighty.r.req_attr["request.orig-uri"]

Actions #4

Updated by gstrauss 2 months ago · Edited

lighty.r.req_attr["request.orig-uri"]
Note that is the original request sent by client and remains the same (unchanged) by rewrites or other internal redirects.

Note: I tested this with HTTP/1.0. I'll see what curl sends for HTTP/2.0 in a sec.

Actions #5

Updated by flynn 2 months ago

As I wrote in my initial request, I already tried lighty.r.req_attr["request.orig-uri"] with no success: wget -SqO- 'http://localhost/bla#blabla&test=1'

2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_header: key Host, value: localhost
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_header: key User-Agent, value: Wget/1.21.4
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_header: key Accept, value: */*
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_header: key Accept-Encoding, value: identity
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_header: key Connection, value: Keep-Alive
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key physical.path, value: /var/www/bla
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key physical.rel-path, value: /bla
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key physical.doc-root, value: /var/www/
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key physical.basedir, value: /var/www/
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key uri.path, value: /bla
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key uri.path-raw, value: /bla
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key uri.scheme, value: http
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key uri.authority, value: localhost
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key uri.query, value: nil
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key request.method, value: GET
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key request.uri, value: /bla
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key request.orig-uri, value: /bla
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key request.path-info, value: nil
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key request.remote-ip, value: ::1
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key request.remote-addr, value: ::1
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key request.remote-port, value: 45630
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key request.server-addr, value: ::1
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key request.server-port, value: 80
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key request.protocol, value: HTTP/1.1
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key request.server-name, value: localhost
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_attr: key request.stage, value: handle-req
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_item: key http_status, value: 0
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_item: key req_header_len, value: 127
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_item: key resp_header_len, value: 0
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_item: key bytes_in, value: 127
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_item: key bytes_out, value: 0
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_item: key stream_id, value: 0
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_item: key req_count, value: 1
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) r.req_item: key keep_alive, value: 1
2024-02-14 07:16:41: (mod_magnet.c.2945) (lua-print) lighty.r.req_item: key start_time, value: 1707891401

Configuration: magnet.attract-physical-path-to = ( "/etc/lighttpd/lua/debug.lua" )

Actions #6

Updated by gstrauss 2 months ago

I do not think curl or wget are sending the fragment in the request. lighttpd is not receiving the fragment.

Actions #7

Updated by gstrauss 2 months ago · Edited

I do not think curl or wget are sending the fragment in the request. lighttpd is not receiving the fragment.
==> confirmed with strace: curl and wget strip the fragment before sending the request.

lighttpd will include the fragment in lighty.r.req_attr["request.orig-uri"] if received in the request, but I want to emphasize that sending fragment to the server is fragile at best, and many clients (or intermediary proxies) will strip the fragment when sending the request.

Actions #8

Updated by gstrauss 2 months ago

Background: I want to authenticate against authentik using openid and the callback of authentik looks like this:

/callback/openid#access_token=...&id_token=...&...

I am not familiar with authentik.
You may have to rewrite the fragment into the query string or path-info if you're going to forward it to a server, or you need to find a tool which does not strip the fragment from the URI before sending the request.

Actions #9

Updated by flynn 2 months ago

I used wget as client only for this ticket, orignally I found this issue in the browser (firefox), which has the same problem.

But: thank you a lot for pointing me to the client issue. I'll try to change the way how authentik generates the redirect url...

Actions #10

Updated by gstrauss 2 months ago

Actions #11

Updated by gstrauss 2 months ago

To be more explicit: for security reasons, you might consider sending the access_token, id_token, and additional info in a POST body rather than including sensitive info in a URI, which may go through proxies and may be logged, including the sensitive information.

Actions #12

Updated by flynn 2 months ago

I suspect that authentik uses the hashmark in the GET request to prevent the logging of sensitive data such as tokens.

I could change the request type of the redirect uri to POST, which solves the problem.

Actions #13

Updated by flynn 2 months ago · Edited

Note: I tried to read the POST-body in lua with lighty.r.req_body.*, but it failed with

unable to collect request body (handler already set); (perhaps load mod_magnet earlier in server.modules, before mod_fastcgi; or require r.req_env['REMOTE_USER'] before attempting r.req_body.collect?)

Neither loading mod_magnet before mod_fastcgi nor require r.req_env['REMOTE_USER'] helped.

The fix was to exclude this url from url.rewrite. Maybe this can be added to the error message: the url must not be rewritten ??

Actions #14

Updated by gstrauss 2 months ago · Edited

The error message includes the module that was already set to handle the request, in your case mod_fastcgi.

Are you using magnet.attract-raw-url-to to collect the request body? That should work as long as mod_magnet is listed before mod_fastcgi in server.modules. I have a hunch that you are using mod_fastcgi with "check-local" => "disable", and so mod_fastcgi is grabbing the request in plugin_call_handle_uri_clean(), and you are using mod_magnet magnet.attract-physical-path-to which runs later in request processing in plugin_call_handle_physical().

I'll also guess that modifying your URL rewrite rules helped since the url.rewrite rule result was a path that is handled by mod_fastcgi in your config.

Actions #15

Updated by flynn 2 months ago

I changed the loading order of modules and verified with lighttpd -p that mod_magnet is listed before mode_fastcgi, but it still failed with the same error message.

Even if mod_magnet ist the first module, it fails. Only when I disable the url-rewriting, I can read the body. The whole configuration is inside a $HTTP["url"] conditional, maybe that is the problem ...

Actions #16

Updated by gstrauss 2 months ago

You must run the script in an earlier lighttpd request hook if something like mod_fastcgi is handling the request sooner.
magnet.attract-raw-url-to = ( "/etc/lighttpd/lua/debug.lua" )

Maybe try debug.log-request-handling = "enable" or debug.log-condition-handling = "enable" to get some insight?

Actions #17

Updated by gstrauss 2 months ago

--- a/src/mod_magnet.c
+++ b/src/mod_magnet.c
@@ -2510,6 +2510,7 @@ static int magnet_reqbody(lua_State *L) {
             else {
                 log_error(r->conf.errh, __FILE__, __LINE__,
                   "unable to collect request body (handler already set); " 
+                  "(prefer to collect in magnet.attract-raw-url-to config) " 
                   "(perhaps load mod_magnet earlier in server.modules, " 
                   "before mod_%s; or require r.req_env['REMOTE_USER'] before " 
                   "attempting r.req_body.collect?)", r->handler_module->name);
Actions #18

Updated by gstrauss 2 months ago

@flynn did you get it working by following my instructions?

Actions #19

Updated by flynn 2 months ago

As suspected I'm using magnet.attract-physical-path-to with "check-local" => "disable" in the fastcgi-section.

Changing magnet.attract-raw-url-to works and solves the problem.

Using debug.log-request-handling = "enable" and debug.log-condition-handling = "enable" the output is quite large and not easy to diff between different configurations. While magnet.attract-raw-url-to works I did not inevestigate this ouput further here.

Actions

Also available in: Atom