[Solved] reading completed POST req_body in mod_magnet context

Hello!

From here https://stackoverflow.com/a/60853401
and here https://redmine.lighttpd.net/projects/lighttpd/wiki/Mod_magnet#Overview
"Please note: mod_magnet is aimed at request manipulation such as url-rewriting and is not meant to be a general replacement for your entire scripting environment."
and here: src/mod_magnet.c.2426,2436 https://github.com/lighttpd/lighttpd1.4/blob/6c6c619fac05583e4d768e5545ff6cb8484cebe1/src/mod_magnet.c#L2436
(and one more place on redmine, but i lost the source) ...

i understand that mod_magnet finds lighttpd in a request-routing phase and does not provide lua with a complete POST req_body.

i'd like lighttpd to respond to a request by processing the POST req_body and handling it as can be done with lighty.r.req_attr["uri.query"] & lighty.c.urldec_query. An https POST has the advantage of not sending the "payload" until a formal encrypted channel is made for the communication, which is (arguably) better than including the query params in the request uri. But the core reason is the tiny memory footprint, speed, and tooling that the lua lighty.* object and library functions provides: https://redmine.lighttpd.net/projects/lighttpd/wiki/Mod_magnet

So what i'm asking is how can i hand-off the request (i guess to prevent blocking incoming requests) from this routing phase to another lighttpd instance(s) that can react with POST req_body processing included/available. Is this something better suited for gw_backend (perhaps with a lighttpd instance on a unix-domain socket)? Maybe i'm overthinking this and what i mean is mod_proxy...? Not sure what tradeoffs are between mod_cgi, mod_fastcgi, mod_scgi, mod_proxy handing requests off to another lighttpd instance(s), so maybe you could direct regarding all this?

Using lighttpd-1.4.71, the following illustrates how i got here:

# /var/www/lighttpd.conf
server.modules            = ( "mod_magnet" )
server.pid-file           = "/var/www/lighttpd.pid" 
server.errorlog           = "/var/www/error.log" 
server.document-root      = "/var/www/docroot/" 
server.port               = 80
magnet.attract-raw-url-to = ( "/var/www/route.lua" )

-- /var/www/route.lua
local r = lighty.r
local reqa = r.req_attr
local requesturi = reqa["request.uri"]:gsub("/+$", "")
if requesturi:match('^/myapiendpoint/') then
    local collected = r.req_body.collect
    print( string.format('ROUTE: %q', collected) )
    local handler = dofile '/var/www/apiendpoints/index.lua'
    return handler( lighty ) or 400
end
return 0

-- /var/www/apiendpoints/index.lua
local function requesthandler( lighty )
    local r = lighty.r
    local reqa = r.req_attr
    if 'POST' ~= reqa['request.method'] then return 405 end
    local reqbody = r.req_body
    local gotit = reqbody.get
    local len = reqbody.len -- likely set from content-length header in original request
    print( string.format('API: %q %q', gotit, len) )
    -- if this were to have a collected POST body
    -- do something constructive with the payload
    -- and send dynamic response body
    return 200
end
return requesthandler

#!/bin/bash
curl localhost:80 --data "variable=value" 
tail /var/www/error.log
2023-06-28 16:52:03: (mod_magnet.c.2436) unable to collect request body (handler already set); (perhaps load mod_magnet earlier in server.modules, before mod_magnet; or require r.req_env['REMOTE_USER'] before attempting r.req_body.collect?)
2023-06-28 16:52:03: (mod_magnet.c.2922) (lua-print) ROUTE: false
2023-06-28 16:52:03: (mod_magnet.c.2922) (lua-print) API: nil 14

OH, important question (difficult to keep everything succinct):
Seeing how much of the request handling in lua mod_magnet can take the stead of lighttpd configuration (lighttpd.conf), is there any such interface/way for sending the request onto mod_fastcgi/proxy from lua (without having it to be explicitly defined in lighttpd.conf)?

Replies (18)

RE: reading completed POST req_body in mod_magnet context - Added by gstrauss about 1 year ago

Please carefully read the documentation for lighty.r.req_body.collect on mod_magnet
You are not checking and reacting to the return value from lighty.r.req_body.collect.
You are not using lighty.r.req_body.collect properly. Please read the doc more carefully.

For restarting a request, search for lighty.RESTART_REQUEST on mod_magnet. However, lighty.RESTART_REQUEST was never intended for use with lighty.r.req_body.collect, and may or may not work for you, especially if you make any modifications. YMMV.

RE: reading completed POST req_body in mod_magnet context - Added by spillkiss about 1 year ago

Well at least i was on the right track.

But, i've seem to have stumbled on unexpected functionality of HTTP/2... with SSL:

FILLED post body: curl -iv http://example.com --data "tested".
nil post body: curl -iv https://example.com --data "tested".
FILLED post body: curl -iv --http1.1 https://example.com --data "tested".

Then if i use these flags: server.feature-flags = ( "server.h2proto" => "disable", # post body over ssl? "server.h2c" => "enable" )

FILLED post body: curl -iv https://example.com --data "tested".
FILLED post body: curl -iv --http2 http://example.com --data "tested".

RE: reading completed POST req_body in mod_magnet context - Added by gstrauss about 1 year ago

But, i've seem to have stumbled on unexpected functionality of HTTP/2... with SSL:

Odds are better that you have still failed to read the documentation carefully. You have not shared any modifications to your code, so I can only assume you failed to modify your code properly, or are logging trace with behavior I expect to see, but you apparently do not and you have not explained why you think the trace is incorrect.

RE: reading completed POST req_body in mod_magnet context - Added by spillkiss about 1 year ago

Did test this what felt like thoroughly, but user error is the likely theme and i agree. i may be able to look more deeply into this in a few days and report back.

i'll explain what i understand and is working:

req_body.collect has a return value of false (0)
the script must return 0
iterations may be tracked with an env variable (also header like https://redmine.lighttpd.net/projects/lighttpd/wiki/ModMagnetExamples#lua-mod_uploadprogress)
if req_body.collect returns false on a subsequent load, the script must return 0 until 1 is the return value
when the return value will be 1, req_body.get will have full request (with headers); sanity can be checked comparing req_body.len to what follows the first double-line

With curl --http2 and a valid cert from let's encrypt, the iterations will keep going unless i disable h2 or force http1.1.

i did have some limits in my scripts to (GET|POST|HEAD|PRI|OPTIONS), but lifting these yielded the same error from my comment above. My tests made a mess of confs and scripts, so next i'll try set up a cleaner limited space and report back.

RE: reading completed POST req_body in mod_magnet context - Added by gstrauss about 1 year ago

I think there is a continued failure in basic reading comprehension, and subsequently a failure for you to VALIDATE YOUR ASSUMPTIONS.

mod_magnet documents:
lighty.r.req_header[]: HTTP request headers
and
lighty.r.req_body.*: HTTP request body attributes and accessors (since 1.4.65)

It is completely illogical for you to somehow arrive at the assumption:

req_body.get will have full request (with headers); sanity can be checked comparing req_body.len to what follows the first double-line

What did you read in what documentation that led you to such bogus conclusions? Both are incorrectly stated.
Request headers are the request headers.
Request body is the request body.
A general HTTP request consists of headers followed by an (optional) body.

Did you write some simple code to show you what is received? That is basic debugging when TESTING YOUR ASSUMPTIONS.

iterations may be tracked with an env variable (also header like https://redmine.lighttpd.net/projects/lighttpd/wiki/ModMagnetExamples#lua-mod_uploadprogress)

That is unrelated to everything discussed thus far. As you have stated, you have a mess of code which you have not methodically debugged.

RE: reading completed POST req_body in mod_magnet context - Added by spillkiss about 1 year ago

Yeah you're right, i'm confusing what i thought were headers in a properly formatted POST body.

Sorry for the pulling teeth, here's a more complete report (just tested):

# /var/www/modmagnet/lighttpd.conf
server.modules            = ( "mod_openssl", "mod_magnet" )
server.pid-file           = "/var/www/modmagnet/lighttpd.pid" 
server.document-root      = "/var/www/modmagnet/" 
server.errorlog           = "/var/www/modmagnet/error.log" 
server.port               = 443
ssl.engine                = "enable" 
ssl.pemfile = "/var/www/cert/example.com/fullchain.pem" 
ssl.privkey = "/var/www/cert/example.com/privkey.pem" 
ssl.openssl.ssl-conf-cmd = (
  "Options" => "-ServerPreference",
)
$SERVER["socket"] == ":80" { ssl.engine = "disable" }
$SERVER["socket"] == "[::]:80" { ssl.engine = "disable" }
$SERVER["socket"] == "[::]:443" { ssl.engine = "enable" }
magnet.attract-raw-url-to = ( "/var/www/modmagnet/route.lua" )

-- /var/www/modmagnet/route.lua
if lighty.r.req_attr['request.uri']:match('^/apiendpoint/') then
    local handler = dofile '/var/www/modmagnet/sub/index.lua'
    return handler(lighty) or 400
end
return 200

-- /var/www/modmagnet/sub/index.lua
local function maybecalledcollect(lighty)
    local collect = lighty.r.req_body.collect
    print(string.format('COLLECT: %s', collect))
    if not collect then return 0 end
    return nil
end
local function requesthandler(lighty)
    local reqb = lighty.r.req_body
    print(string.format('STATS: %s, %s, %s', reqb.len, reqb.bytes_in, reqb.bytes_out))
    local phase = maybecalledcollect(lighty)
    if 0 == phase then return 0 end
    local result = reqb.get
    print(string.format('RESULT: %s', result))
    return 200
end
return requesthandler

#!/bin/bash
curl -vi https://example.com/apiendpoint/ --data 'lets see how this goes'
*   Trying 0001:0002::0003:0004:443...
* TCP_NODELAY set
* Connected to example.com (0001:0002::0003:0004) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=example.com
*  start date: Apr 18 03:48:26 2023 GMT
*  expire date: Jul 17 03:48:25 2023 GMT
*  subjectAltName: host "example.com" matched cert's "example.com" 
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55ab3e011320)
> POST /apiendpoint/ HTTP/2
> Host: example.com
> user-agent: curl/7.68.0
> accept: */*
> content-length: 22
> content-type: application/x-www-form-urlencoded
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 8)!
* We are completely uploaded and fine

# /var/www/modmagnet/error.log
2023-06-30 21:52:41: (server.c.1909) server started (lighttpd/1.4.71)
2023-06-30 21:53:32: (mod_magnet.c.2922) (lua-print) STATS: 22, 0, 0
2023-06-30 21:53:32: (mod_magnet.c.2922) (lua-print) COLLECT: false
2023-06-30 21:53:32: (mod_magnet.c.2922) (lua-print) STATS: 22, 0, 0
2023-06-30 21:53:32: (mod_magnet.c.2922) (lua-print) COLLECT: false
[ ... this will repeat on until the process is killed ]

#!/bin/bash
curl --http1.1 -vi https://example.com/apiendpoint/ --data 'lets see how this goes'
*   Trying 0001:0002::0003:0004:443...
* TCP_NODELAY set
* Connected to example.com (0001:0002::0003:0004) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=example.com
*  start date: Apr 18 03:48:26 2023 GMT
*  expire date: Jul 17 03:48:25 2023 GMT
*  subjectAltName: host "example.com" matched cert's "example.com" 
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> POST /apiendpoint/ HTTP/1.1
> Host: example.com
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Length: 22
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 22 out of 22 bytes
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Length: 0
Content-Length: 0
< Date: Fri, 30 Jun 2023 21:58:23 GMT
Date: Fri, 30 Jun 2023 21:58:23 GMT
< Server: lighttpd/1.4.71
Server: lighttpd/1.4.71

#/var/www/modmagnet/error.log
2023-06-30 21:58:07: (server.c.1909) server started (lighttpd/1.4.71)
2023-06-30 21:58:23: (mod_magnet.c.2922) (lua-print) STATS: 22, 0, 0
2023-06-30 21:58:23: (mod_magnet.c.2922) (lua-print) COLLECT: false
2023-06-30 21:58:23: (mod_magnet.c.2922) (lua-print) STATS: 22, 22, 0
2023-06-30 21:58:23: (mod_magnet.c.2922) (lua-print) COLLECT: true
2023-06-30 21:58:23: (mod_magnet.c.2922) (lua-print) RESULT: POST /apiendpoint/ HTTP/1.1\r\nHost: example.com\r\nUser-Agent: curl/7.68.0\r\nAccept: */*\r\nContent-Length: 22\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\nlets see how this goes

RE: reading completed POST req_body in mod_magnet context - Added by gstrauss about 1 year ago

Yeah you're right, i'm confusing what i thought were headers in a properly formatted POST body.

That makes no sense to me. An HTTP POST request with Content-Type: application/x-www-form-urlencoded is "formatted" (sic) as application/x-www-form-urlencoded. I don't know what you mean by headers in the body. HTTP request headers are distinct from the (optional) HTTP request body. If you are using a different Content-Type in your HTTP POST request, then the HTTP request body should be "formatted" as the Content-Type you intend.

In any case, thank you for showing some code. It appears that there is a regression somewhere in lighttpd 1.4.70 where lighty.r.req_body.collect results in spinning inside lighttpd rather than making progress. I'll track down that bug and will post a patch, hopefully later this weekend. The bug is likely a code (over)simplification I made when further isolating h2 flow control from h1 flow control.

Aside: Your code local handler = dofile '/var/www/modmagnet/sub/index.lua' is much less efficient than allowing lighttpd to monitor the script. On every single request that is routed here in your code, the script is loaded from disk and compiled by lua. If you included that code in lua rather than using dofile, it would not be reloaded on every request. Now, lighttpd mod_magnet monitors the target magnet.attract-raw-url-to = ( "/var/www/modmagnet/route.lua" ) and if route.lua changes, lighttpd will reload the script automatically. Therefore, you can either keep all the code in route.lua, or touch route.lua whenever a different included file is changed, rather than calling dofile upon every request to that route.

RE: reading completed POST req_body in mod_magnet context - Added by gstrauss about 1 year ago

There is a code change in lighttpd 1.4.70 in h2.c which sets initial r->state = CON_STATE_HANDLE_REQUEST, and this is fine for most usage which handle request body even if r->state is not CON_STATE_READ_POST. However, mod_magnet checks r->state = CON_STATE_READ_POST for lighty.r.req_body.collect, and that is the error you are seeing when accessed with HTTP/2.

mod_webdav also checks r->state = CON_STATE_READ_POST and is likely affected by this bug when used with HTTP/2.

--- a/src/h2.c
+++ b/src/h2.c
@@ -1569,12 +1569,16 @@ h2_recv_headers (connection * const con, uint8_t * const s, uint32_t flen)
          *      GOAWAY, e.g. might avoid allocating (request_st *r) */
         r = h2_init_stream(h2r, con);
         r->x.h2.id = id;
-        r->x.h2.state = (s[4] & H2_FLAG_END_STREAM)
-          ? H2_STATE_HALF_CLOSED_REMOTE
-          : H2_STATE_OPEN;
-        r->state = (0 == r->reqbody_length)
-          ? CON_STATE_HANDLE_REQUEST
-          : CON_STATE_READ_POST;
+        if (s[4] & H2_FLAG_END_STREAM) {
+            r->x.h2.state = H2_STATE_HALF_CLOSED_REMOTE;
+            r->state = CON_STATE_HANDLE_REQUEST;
+            r->reqbody_length = 0;
+        }
+        else {
+            r->x.h2.state = H2_STATE_OPEN;
+            r->state = CON_STATE_READ_POST;
+            r->reqbody_length = -1;
+        }
         /* Note: timestamps here are updated only after receipt of entire header
          * (HEADERS frame might have been sent in multiple packets
          *  and CONTINUATION frames may have been sent in multiple packets)
@@ -1659,15 +1663,6 @@ h2_recv_headers (connection * const con, uint8_t * const s, uint32_t flen)

     if (!h2c->sent_goaway) {
         h2c->h2_cid = id;
-        if (!light_btst(r->rqst_htags, HTTP_HEADER_CONTENT_LENGTH))
-            r->reqbody_length = (s[4] & H2_FLAG_END_STREAM) ? 0 : -1;
-      #if 0
-        else if (r->reqbody_length > 0 && (s[4] & H2_FLAG_END_STREAM)) {
-            /*(handled in connection_handle_read_post_state())*/
-            /* XXX: TODO if (r->conf.log_request_header_on_error) */
-            r->http_status = 400; /* Bad Request */
-        }
-      #endif
         /*(lighttpd.conf config conditions not yet applied to request,
          * but do not increase window size if BUFMIN set in global config)*/
         if (r->reqbody_length /*(see h2_init_con() for session window)*/
--- a/src/request.c
+++ b/src/request.c
@@ -464,7 +464,8 @@ static int http_request_parse_single_header(request_st * const restrict r, const
             off_t clen = (off_t)li_restricted_strtoint64(v, vlen, &err);
             if (err == v+vlen) {
                 /* (set only if not set to -1 by Transfer-Encoding: chunked) */
-                if (0 == r->reqbody_length) r->reqbody_length = clen;
+                if (r->http_version > HTTP_VERSION_1_1 || 0 == r->reqbody_length)
+                    r->reqbody_length = clen;
             }
             else {
                 return http_request_header_line_invalid(r, 400, "invalid Content-Length header -> 400");

RE: reading completed POST req_body in mod_magnet context - Added by spillkiss about 1 year ago

dofile '/var/www/modmagnet/sub/index.lua' is much less efficient

This is not my production code, i put it together to illustrate what i was experiencing.

To maintain order in production i need to load multiple .lua files which i do with some caching (from your direction about a year ago, thanks!: https://redmine.lighttpd.net/boards/2/topics/quote/10558), and if i included anything that looks like that it might muddle things.

TBHWY, i would not have opened this ticket if i hadn't read this https://stackoverflow.com/a/60853401 and understood it as i did.

Since i've managed to make it onto the lighttpd reporting contributors hall of fame, i'd like to take a moment to give you (et al.) kudos on lighttpd. It's easily A+ software, FWIW coming from me. But i don't doubt there are many who benefit from this and who are similarly very thankful. Thank you

RE: reading completed POST req_body in mod_magnet context - Added by gstrauss about 1 year ago

Few take a moment to express that, so thank you.

Patch is now on lighttpd git master and will be part of lighttpd 1.4.72 (estimated release some time this summer).

RE: [Solved] reading completed POST req_body in mod_magnet context - Added by spillkiss about 1 year ago

thank you

You are most certainly welcome!
And i realized that the link on my last comment to your suggestion wasn't correct.
So in case anyone reading this would appreciate it: https://redmine.lighttpd.net/boards/2/topics/10548?r=10558#message-10558

Thanks for the patch ... but i might be missing something.
i trialed it out and i'm not getting the post body on h2.
Here's what i got:

#!/bin/bash
curl -vi https://example.com/apiendpoint/ --data 'take 2'
*   Trying 0001:0002::0003:0004:443...
* TCP_NODELAY set
* Connected to example.com (0001:0002::0003:0004) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=example.com
*  start date: Apr 18 03:48:26 2023 GMT
*  expire date: Jul 17 03:48:25 2023 GMT
*  subjectAltName: host "example.com" matched cert's "example.com" 
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55f3896d4320)
> POST /apiendpoint/ HTTP/2
> Host: example.com
> user-agent: curl/7.68.0
> accept: */*
> content-length: 6
> content-type: application/x-www-form-urlencoded
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 8)!
* We are completely uploaded and fine
< HTTP/2 200 
HTTP/2 200 
< content-length: 0
content-length: 0
< date: Fri, 07 Jul 2023 05:23:40 GMT
date: Fri, 07 Jul 2023 05:23:40 GMT
< server: lighttpd/1.4.72-devel-lighttpd-1.4.71-16-g4a499883
server: lighttpd/1.4.72-devel-lighttpd-1.4.71-16-g4a499883

< 
* Connection #0 to host example.com left intact

#!/bin/bash
curl -vi --http1.1 https://example.com/apiendpoint/ --data 'take 2'
*   Trying 0001:0002::0003:0004:443...
* TCP_NODELAY set
* Connected to example.com (0001:0002::0003:0004) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=example.com
*  start date: Apr 18 03:48:26 2023 GMT
*  expire date: Jul 17 03:48:25 2023 GMT
*  subjectAltName: host "example.com" matched cert's "example.com" 
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> POST /apiendpoint/ HTTP/1.1
> Host: example.com
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Length: 6
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 6 out of 6 bytes
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Length: 0
Content-Length: 0
< Date: Fri, 07 Jul 2023 05:24:02 GMT
Date: Fri, 07 Jul 2023 05:24:02 GMT
< Server: lighttpd/1.4.72-devel-lighttpd-1.4.71-16-g4a499883
Server: lighttpd/1.4.72-devel-lighttpd-1.4.71-16-g4a499883

#/var/www/modmagnet/error.log
2023-07-07 05:23:26: (server.c.1923) server started (lighttpd/1.4.72-devel-lighttpd-1.4.71-16-g4a499883)
[ this is the 1st curl ... ]
2023-07-07 05:23:40: (mod_magnet.c.2922) (lua-print) STATS: 6, 0, 0
2023-07-07 05:23:40: (mod_magnet.c.2922) (lua-print) COLLECT: false
2023-07-07 05:23:40: (mod_magnet.c.2922) (lua-print) STATS: 6, 6, 0
2023-07-07 05:23:40: (mod_magnet.c.2922) (lua-print) COLLECT: true
2023-07-07 05:23:40: (mod_magnet.c.2922) (lua-print) RESULT:
[ this is the 2nd curl ... ]
2023-07-07 05:24:02: (mod_magnet.c.2922) (lua-print) STATS: 6, 0, 0
2023-07-07 05:24:02: (mod_magnet.c.2922) (lua-print) COLLECT: false
2023-07-07 05:24:02: (mod_magnet.c.2922) (lua-print) STATS: 6, 6, 0
2023-07-07 05:24:02: (mod_magnet.c.2922) (lua-print) COLLECT: true
2023-07-07 05:24:02: (mod_magnet.c.2922) (lua-print) RESULT: POST /apiendpoint/ HTTP/1.1\r\nHost: example.com\r\nUser-Agent: curl/7.68.0\r\nAccept: */*\r\nContent-Length: 6\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\ntake 2

RE: [Solved] reading completed POST req_body in mod_magnet context - Added by spillkiss about 1 year ago

Well, i accidentally spilled some info in there. i hope you'd be able to correct that. It's hard to be perfect these days.

RE: [Solved] reading completed POST req_body in mod_magnet context - Added by gstrauss about 1 year ago

What would you like corrected? (Hint at it; don't be explicit, since I'll have to remove that, too)
I changed two places to subject: CN=example.com

I do not have time to look at the moment, but I did test locally and have pushed changes to lighttpd git master along with others. Please test HEAD: https://git.lighttpd.net/lighttpd/lighttpd1.4.git/ and I'll try to test again tomorrow.

RE: [Solved] reading completed POST req_body in mod_magnet context - Added by spillkiss about 1 year ago

non-redacted c n

RE: [Solved] reading completed POST req_body in mod_magnet context - Added by gstrauss about 1 year ago

I changed two places to subject: CN=example.com

RE: [Solved] reading completed POST req_body in mod_magnet context - Added by spillkiss about 1 year ago

To me it looked like the github branch was the same as the self-hosted, but i trialed it as you suggested and i've the same result. Sorry for the headache(s).

RE: [Solved] reading completed POST req_body in mod_magnet context - Added by gstrauss about 1 year ago

That is a separate bug in copying the body from the lighttpd chunkqueue into a lua string.

I pushed a couple commits to my devel branch https://git.lighttpd.net/lighttpd/lighttpd1.4/src/branch/personal/gstrauss/master

Now I see why you thought lighty.r.req_body.get provided request headers and body, since that is what you saw with HTTP/1.1 request with a small request after lighty.r.req_body.collect. lighttpd simply adjusted the offset in the chunk and moved the chunk into r->reqbody_queue, which avoided copying the data. For HTTP/2, chunk containing the small request body was also moved and the data was at an offset, but the first byte of the chunk was presumably '\0'.

RE: [Solved] reading completed POST req_body in mod_magnet context - Added by spillkiss about 1 year ago

That branch corrects it on my end (thanks!). :two-thumbs-up!:

(1-18/18)

Project

General

Profile

Lighttpd