Project

General

Profile

[Solved] Build issue

Added by Agossi 2 months ago

Hi there,

I have issues with build starting from 1.4.61-64 after a cross compiling (successful build). GCC Version (GCC) 4.4.2
lighttpd starts also without any issue.

I get the following error in the WebBrowsers:
GET http://IP/index.php net::ERR_INVALID_CHUNKED_ENCODING 200 (OK)

I tried also to just get the page via wget and this was the result:

index.php                                             [ <=>                                                                                                          ] 127,98K  --.-KB/s    in 0,005s  

2022-03-11 13:38:06 (24,5 MB/s) - *Read error at byte 131056*  (Success).Retrying.

--2022-03-11 13:38:10--  (try: 5)  http://IP/index.php
Connecting to localhost (localhost)|127.0.0.1|:3128... connected.
Proxy request sent, awaiting response... 200 OK
*Length: unspecified [text/html]*
Saving to: ‘index.php’

index.php                                             [ <=>                                                                                                          ] 127,98K  --.-KB/s    in 0,006s  

2022-03-11 13:38:11 (22,4 MB/s) - *Read error at byte 131056* (Success).Retrying.

Any hint where I can look to fix this issue? Version 1.4.60 still works on the system.

Best Regards


Replies (23)

RE: Build issue - Added by gstrauss 2 months ago

insufficient information. please try to add more details.

RE: Build issue - Added by Agossi 2 months ago

Configure used:

./configure --with-openssl \
    --without-libev \
    --without-mysql \
    --without-pgsql \
    --without-dbi \
    --without-sasl \
    --without-ldap \
    --without-pam \
    --without-wolfssl \
    --with-pcre \
    --without-pcre2 \
    --without-brotli \
    --without-fam \
    --without-xxhash \
    --without-maxminddb \
    --host=arm-none-linux-gnueabi \
    CFLAGS='-std=c99'

Results In:

Plugins:

enabled: 
  mod_access
  mod_accesslog
  mod_ajp13
  mod_alias
  mod_auth
  mod_authn_file
  mod_cgi
  mod_deflate
  mod_dirlisting
  mod_evhost
  mod_expire
  mod_extforward
  mod_fastcgi
  mod_indexfile
  mod_openssl
  mod_proxy
  mod_redirect
  mod_rewrite
  mod_rrdtool
  mod_scgi
  mod_secdownload
  mod_setenv
  mod_simple_vhost
  mod_sockproxy
  mod_ssi
  mod_staticfile
  mod_status
  mod_userdir
  mod_usertrack
  mod_vhostdb
  mod_webdav
  mod_wstunnel
disabled: 
  mod_authn_dbi
  mod_authn_gssapi
  mod_authn_ldap
  mod_authn_pam
  mod_gnutls
  mod_magnet
  mod_maxminddb
  mod_mbedtls
  mod_nss
  mod_vhostdb_dbi
  mod_vhostdb_ldap
  mod_vhostdb_mysql
  mod_vhostdb_pgsql
  mod_wolfssl

Features:

enabled: 
  auth-crypt
  compress-deflate
  compress-gzip
  large-files
  network-ipv6
  network-openssl
  regex-conditionals
  stat-cache-inotify
disabled: 
  compress-brotli
  compress-bzip2
  dbi
  kerberos
  ldap
  lua
  maxminddb
  mysql
  network-gnutls
  network-mbedtls
  network-nss
  network-wolfssl
  pam
  postgresql
  webdav-locks
  webdav-properties

now calling the make lead to this issue:

 -Wshadow -pedantic -MT t_test_mod-stat_cache.o -MD -MP -MF .deps/t_test_mod-stat_cache.Tpo -c -o t_test_mod-stat_cache.o `test -f 'stat_cache.c' || echo './'`stat_cache.c
stat_cache.c: In function ‘fam_dir_monitor’:
stat_cache.c:723: error: ‘IN_EXCL_UNLINK’ undeclared (first use in this function)
stat_cache.c:723: error: (Each undeclared identifier is reported only once
stat_cache.c:723: error: for each function it appears in.)
make[3]: *** [Makefile:4181: t_test_mod-stat_cache.o] Error 1
make[2]: *** [Makefile:1768: all] Error 2
make[1]: *** [Makefile:470: all-recursive] Error 1
make: *** [Makefile:400: all] Error 2

which got fixed in stat_cache.c by:

#ifndef IN_EXCL_UNLINK
    /* supported in Kernel 3.8.6 but not in default includes set it hard*/
    #define IN_EXCL_UNLINK        0x04000000 
#endif 

CALLIN make again lead to next issue:

 -Wshadow -pedantic -MT ls-hpack/lighttpd-lshpack.o -MD -MP -MF ls-hpack/.deps/lighttpd-lshpack.Tpo -c -o ls-hpack/lighttpd-lshpack.o `test -f 'ls-hpack/lshpack.c' || echo './'`ls-hpack/lshpack.c
In file included from ls-hpack/lshpack.h:34,
                 from ls-hpack/lshpack.c:40:
ls-hpack/lsxpack_header.h:59: warning: type of bit-field ‘flags’ is a GCC extension
ls-hpack/lshpack.c:687: warning: unknown option after ‘#pragma GCC diagnostic’ kind
ls-hpack/lshpack.c: In function ‘lshpack_enc_huff_encode’:
ls-hpack/lshpack.c:704: error: #pragma GCC diagnostic not allowed inside functions
ls-hpack/lshpack.c:705: error: #pragma GCC diagnostic not allowed inside functions
ls-hpack/lshpack.c:706: error: #pragma GCC diagnostic not allowed inside functions
ls-hpack/lshpack.c:821: error: #pragma GCC diagnostic not allowed inside functions
ls-hpack/lshpack.c: At top level:
ls-hpack/lshpack.c:2062: warning: expected [error|warning|ignored] after ‘#pragma GCC diagnostic’
make[3]: *** [Makefile:2809: ls-hpack/lighttpd-lshpack.o] Error 1
make[2]: *** [Makefile:1768: all] Error 2
make[1]: *** [Makefile:470: all-recursive] Error 1
make: *** [Makefile:400: all] Error 2

This was "fixed" by adding && GNUC >= 5 to the occurences.

After these changes the lighttpd and modules got build, but lead to the above issue described.

Config of lighttpd used:

config {
    var.CWD                        = "/var/volatile/tmp" 
    var.PID                        = 7851
    server.modules                 = (
        "mod_rewrite",
        "mod_access",
        "mod_auth",
        "mod_setenv",
        "mod_fastcgi",
        "mod_expire",
        "mod_evasive",
        "mod_openssl",
    )
    server.document-root           = "/opt/web/" 
    server.errorlog                = "/var/log/lighttpd.error.log" 
    index-file.names               = ("index.php", "index.html", "index.htm", "default.htm")
    mimetype.assign                = (
        ".pdf"     => "application/pdf",
        ".sig"     => "application/pgp-signature",
        ".spl"     => "application/futuresplash",
        ".class"   => "application/octet-stream",
        ".ps"      => "application/postscript",
        ".torrent" => "application/x-bittorrent",
        ".dvi"     => "application/x-dvi",
        ".gz"      => "application/x-gzip",
        ".pac"     => "application/x-ns-proxy-autoconfig",
        ".swf"     => "application/x-shockwave-flash",
        ".tar.gz"  => "application/x-tgz",
        ".tgz"     => "application/x-tgz",
        ".tar"     => "application/x-tar",
        ".zip"     => "application/zip",
        ".mp3"     => "audio/mpeg",
        ".m3u"     => "audio/x-mpegurl",
        ".wma"     => "audio/x-ms-wma",
        ".wax"     => "audio/x-ms-wax",
        ".ogg"     => "application/ogg",
        ".wav"     => "audio/x-wav",
        ".gif"     => "image/gif",
        ".jpg"     => "image/jpeg",
        ".jpeg"    => "image/jpeg",
        ".png"     => "image/png",
        ".xbm"     => "image/x-xbitmap",
        ".xpm"     => "image/x-xpixmap",
        ".xwd"     => "image/x-xwindowdump",
        ".css"     => "text/css",
        ".html"    => "text/html",
        ".htm"     => "text/html",
        ".js"      => "text/javascript",
        ".asc"     => "text/plain",
        ".c"       => "text/plain",
        ".cpp"     => "text/plain",
        ".log"     => "text/plain",
        ".conf"    => "text/plain",
        ".text"    => "text/plain",
        ".txt"     => "text/plain",
        ".dtd"     => "text/xml",
        ".xml"     => "text/xml",
        ".mpeg"    => "video/mpeg",
        ".mpg"     => "video/mpeg",
        ".mov"     => "video/quicktime",
        ".qt"      => "video/quicktime",
        ".avi"     => "video/x-msvideo",
        ".asf"     => "video/x-ms-asf",
        ".asx"     => "video/x-ms-asf",
        ".wmv"     => "video/x-ms-wmv",
        ".bz2"     => "application/x-bzip",
        ".tbz"     => "application/x-bzip-compressed-tar",
        ".tar.bz2" => "application/x-bzip-compressed-tar" 
    )
    server.tag                     = "" 
    evasive.max-conns-per-ip       = 50
    evasive.http-status            = 304
    evasive.retry-after            = 10
    static-file.exclude-extensions = (".php", ".pl", ".fcgi")
    url.rewrite-if-not-file        = (
        "^/(.*)/$" => "/error404.php",
    )

    $HTTP["url"] =~ "\.pdf$" {
        # block 1
        server.range-requests = "disable" 

    } # end of $HTTP["url"] =~ "\.pdf$" 

    $SERVER["socket"] == "0.0.0.0:80" {
        # block 2

    } # end of $SERVER["socket"] == "0.0.0.0:80" 

    $SERVER["socket"] == "[::]:80" {
        # block 3
        server.use-ipv6 = "enable" 

    } # end of $SERVER["socket"] == "[::]:80" 

    $SERVER["socket"] =~ ":(80)" {
        # block 4
        fastcgi.debug               = 1
        server.stream-response-body = 2
        fastcgi.server              = (
            ".php" => (
                "localhost" => (
                    "socket"            => "/var/tmp/php-fastcgi.socket",
                    "bin-path"          => "/usr/bin/php-cgi",
                    "max-procs"         => 10,
                    "bin-environment"   => (
                        "PHP_FCGI_CHILDREN"     => "5",
                        "PHP_FCGI_MAX_REQUESTS" => "500",
                    ),
                    "allow-x-send-file" => "enable",
                ),
            ),
        )

    } # end of $SERVER["socket"] =~ ":(80)" 
}

Hope this info helps to dig inot the issue.

RE: Build issue - Added by gstrauss 2 months ago

Hope this info helps to dig inot the issue.

The config may be useful as prerequisite information for context, but is lacking any further description of the problem.

Request and response headers? A description of what index.php does?
Configuration: Debug Variables

FYI: #pragma GCC diagnostic push/pop was added in GCC 4.6, I believe.

RE: Build issue - Added by gstrauss 2 months ago

Connecting to localhost (localhost)|127.0.0.1|:3128... connected.
Proxy request sent, awaiting response... 200 OK
*Length: unspecified [text/html]*

Looks like your client is connecting through a proxy. Given the age of your compiler, I wouldn't be surprised to find out that the intermediate proxy is also ancient and buggy. Since length wasn't provided in the response, is wget sending an HTTP/1.1 request and the proxy downgrading the request to HTTP/1.0? Does the intermediate proxy not support Transfer-Encoding: chunked. Have you attempted the download with a modern version of curl? Modern curl also supports HTTP/2.

Does the problem occur with server.stream-response-body = 0? (It should not, since lighttpd will capture the complete response and then will send Content-Length with response)

RE: Build issue - Added by Agossi 2 months ago

Hi,

I have done some debug setup an the issue still occurs.
I have created a simple PHP File (index.php) which just shall print out 10.000 lines (attached).

Using "server.stream-response-body = 0" did not have any effect on the issue that the response is not completly returned.
I also disabled the proxy and now directly connect to the system, but still same results.

Using CURL:

Lighttpd 1.4.64:

curl "http://192.168.1.12/index.php" -o test_debugout_1.4.64.html  -v
*   Trying 192.168.1.12...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 192.168.1.12 (192.168.1.12) port 80 (#0)
> GET /index.php HTTP/1.1
> Host: 192.168.1.12
> User-Agent: curl/7.55.1
> Accept: */*
>
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0< HTTP/1.1 200 OK
< Set-Cookie: PHPSESSID=6a6b163i9vp6ovgn7edar0gpph2m8qgkq22rlfejmpl0ljet; path=/; HttpOnly; SameSite=Strict
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-type: text/html; charset=UTF-8
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15768000; includeSubdomains
< Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data:; object-src 'none'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: same-origin
< Transfer-Encoding: chunked
< Date: Mon, 14 Mar 2022 05:26:56 GMT
<
{ [10416 bytes data]
* Malformed encoding found in chunked-encoding
100 62490    0 62490    0     0  62490      0 --:--:-- --:--:-- --:--:--  105k
* Closing connection 0
curl: (56) Malformed encoding found in chunked-encoding

Lighttpd: 1.4.60:

curl "http://192.168.1.12/index.php" -o test_debugout_1.4.60.html  -v
*   Trying 192.168.1.12...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 192.168.1.12 (192.168.1.12) port 80 (#0)
> GET /index.php HTTP/1.1
> Host: 192.168.1.12
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Set-Cookie: PHPSESSID=3ts9va1pn6m2kcl6kv6dn97aftv63kbrjat09m24b2vaivl4; path=/; HttpOnly; SameSite=Strict
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-type: text/html; charset=UTF-8
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15768000; includeSubdomains
< Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data:; object-src 'none'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: same-origin
< Transfer-Encoding: chunked
< Date: Mon, 14 Mar 2022 05:24:34 GMT
<
{ [10416 bytes data]
100  301k    0  301k    0     0   301k      0 --:--:-- --:--:-- --:--:--  321k
* Connection #0 to host 192.168.1.12 left intact

So if I use the old lighttpd 1.4.60 everthing works fine with the same config.
After switching to lighttpd 1.4.61,62,63 or 64 I get the "Malformed encoding found in chunked-encoding" errors.

Also the responses in lighttpd 1.4.64 using curl stop at iteration

Zeile 9160 von 10000
Zeile 9161

with 1.4.60 no issue:

Zeile 9998 von 10000
Zeile 9999 von 10000
Zeile 10000 von 10000

Any hint is welcome or anything I have to look like in linux which lighttpd now depends on since lighttpd 1.4.61 which may not be supported by older distros?

lighttpd_chunked_encoding_issue.zip (23.2 KB) lighttpd_chunked_encoding_issue.zip lighttpd chunked encoding issue embedded old linux

RE: Build issue - Added by gstrauss 2 months ago

Thank you for trying to put together a test case.

I have tried to reproduce this using your test case, and accessing the server across a local network, but have not been able to trigger a failure. I tried different values of curl --limit-rate 100K and 10K to try to trigger the issue, too.

To try to narrow down if the issue is somewhere closer to lighttpd and the client, or closer to lighttpd and the backend, you might try running index.php using CGI instead of using mod_fastcgi, and see if that makes a difference.

Separately, since you noted that the issue does not occur with lighttpd 1.4.60, but does occur with lighttpd 1.4.61, if you have a chance, would you please try building and testing some commits between lighttpd 1.4.60 and lighttpd 1.4.61?

Does the problem occur with server.stream-response-body = 0? (It should not, since lighttpd will capture the complete response and then will send Content-Length with response)

Using "server.stream-response-body = 0" did not have any effect on the issue that the response is not completly returned.

So it still failed (sometimes) with server.stream-response-body = 0? Can you confirm those responses contained Content-Length response header, and did not contain Transfer-Encoding: chunked response header?

RE: Build issue - Added by Agossi 2 months ago

Hi there,

To try to narrow down if the issue is somewhere closer to lighttpd and the client, or closer to lighttpd and the backend, you might try running index.php using CGI instead of using mod_fastcgi, and see if that makes a difference.

=> without fastcgi it works. using the normal mod_cgi

So it still failed (sometimes) with server.stream-response-body = 0? Can you confirm those responses contained Content-Length response header, and did not contain Transfer-Encoding: chunked response header?

=> setting it to 0 does not seem to change anything in Transfer Encoding

mod_fastcgi and server.stream-response-body = 0:

curl -v "http://192.168.1.12/index-test.php" -o file.ttt
*   Trying 192.168.1.12...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 192.168.1.12 (192.168.1.12) port 80 (#0)
> GET /index-test.php HTTP/1.1
> Host: 192.168.1.12
> User-Agent: curl/7.55.1
> Accept: */*
>
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0< HTTP/1.1 200 OK
< Set-Cookie: PHPSESSID=SESSIONID; path=/; HttpOnly; SameSite=Strict
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-type: text/html; charset=UTF-8
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15768000; includeSubdomains
< Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data:; object-src 'none'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: same-origin
< Transfer-Encoding: chunked
< Date: Wed, 16 Mar 2022 04:11:04 GMT
<
{ [10416 bytes data]
* *Malformed encoding found in chunked-encoding
100 62490    0 62490    0     0  62490      0 --:--:-- --:--:-- --:--:--  126k
* Closing connection 0
curl: (56) Malformed encoding found in chunked-encoding*

mod_cgi and server.stream-response-body = 0:

curl -v "http://192.168.1.12/index-test.php" -o file.ttt
*   Trying 192.168.1.12...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 192.168.1.12 (192.168.1.12) port 80 (#0)
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0> GET /index-test.php HTTP/1.1
> Host: 192.168.1.12
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Set-Cookie: PHPSESSID=SESSIONID; path=/; HttpOnly; SameSite=Strict
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-type: text/html; charset=UTF-8
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15768000; includeSubdomains
< Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data:; object-src 'none'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: same-origin
< Transfer-Encoding: chunked
< Date: Wed, 16 Mar 2022 04:13:48 GMT
<
{ [2826 bytes data]
100  301k    0  301k    0     0   301k      0 --:--:--  0:00:01 --:--:--  264k
* Connection #0 to host 192.168.1.12 left intact

Separately, since you noted that the issue does not occur with lighttpd 1.4.60, but does occur with lighttpd 1.4.61, if you have a chance, would you please try building and testing some commits between lighttpd 1.4.60 and lighttpd 1.4.61?

=> I will try if I have some spare time, espacially in the mod_fastcgi section.

RE: Build issue - Added by Agossi 2 months ago

Update:

I have replaced the functions "void chunkqueue_mark_written(chunkqueue *cq, off_t len)" and "chunkqueue_steal_with_tempfiles" in chunk.c from the 1.4.60 into 1.4.64 src
and now I do not get any chunked encoding issue.

curl -v "http://192.168.1.12/index-test.php" -o file.ttt
*   Trying 192.168.1.12...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 192.168.1.12 (192.168.1.12) port 80 (#0)
> GET /index-test.php HTTP/1.1
> Host: 192.168.1.12
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Set-Cookie: PHPSESSID=pkso8m5e4oaohro29i06pbra1edajpipbsmiq7nrh7olses8; path=/; HttpOnly; SameSite=Strict
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-type: text/html; charset=UTF-8
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15768000; includeSubdomains
< Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data:; object-src 'none'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: same-origin
< Transfer-Encoding: chunked
< Date: Wed, 16 Mar 2022 05:25:39 GMT
<
{ [10416 bytes data]
100  301k    0  301k    0     0   301k      0 --:--:-- --:--:-- --:--:--  351k
* Connection #0 to host 192.168.1.12 left intact

RE: Build issue - Added by Agossi 2 months ago

Next Update:

If I manually modify the config.h file after running the configure command from:

/* Define to 1 if you have the `pwritev' function. */
#define HAVE_PWRITEV 1

to

/* Define to 1 if you have the `pwritev' function. */
#undef HAVE_PWRITEV

it also seem to work, but config.log shows that it should be supported by my system:

configure:16730: checking for pwritev
configure:16730: arm-linux-gnueabi-gcc -o conftest -std=c99 -I/sysroot-target/usr/include -I/usr/include -D_REENTRANT  -L/sysroot-target/usr/lib -L/opt/arm/arm-linux-gnueabi/usr/lib conftest.c -lrt  >&5
configure:16730: $? = 0
configure:16730: result: yes

RE: Build issue - Added by gstrauss 2 months ago

Thanks! I'm reviewing all the commits to chunk.c between lighttpd 1.4.60 and lighttpd 1.4.64. I am up to commit 3964a6ac, which uses pwritev and which is what it seems you have zeroed in on above.

RE: Build issue - Added by Agossi 2 months ago

Thanks! I'm reviewing all the commits to chunk.c between lighttpd 1.4.60 and lighttpd 1.4.64. I am up to commit 3964a6ac, which uses pwritev and which is what it seems you have zeroed in on above.

=> Yes, that is correct by removing the support define (#undef) it does not show the issue.

RE: Build issue - Added by gstrauss 2 months ago

This is not a fix, but please try lighttpd 1.4.64 with this patch (and without #undef HAVE_PWRITEV). The patch disables an optimization which tries to write existing data in the destination chunkqueue into temporary files along with data from the source chunkqueue, all in a single pwritev(). If this patch works for you, then it will help me to reproduce the bug. It's late and I'll have to pick this up again tomorrow.

--- a/src/chunk.c
+++ b/src/chunk.c
@@ -947,6 +947,9 @@ static ssize_t chunkqueue_append_cqmem_to_tempfile(chunkqueue * const restrict d

     off_t dlen = 0;
     chunk *c;
+  #if 1
+    c = dest->first;
+  #else
     for (c = dest->first; c && c->type == MEM_CHUNK; c = c->next) {
         const off_t clen = chunk_remaining_length(c);
         iov[iovcnt].iov_base = c->mem->ptr + c->offset;
@@ -956,6 +959,7 @@ static ssize_t chunkqueue_append_cqmem_to_tempfile(chunkqueue * const restrict d
         if (__builtin_expect( (iovcnt == sizeof(iov)/sizeof(*iov)), 0))
             break; /*(not expecting large number of MEM_CHUNK)*/
     }
+  #endif
     if (__builtin_expect( (c != NULL), 0) && dest->first->type == MEM_CHUNK) {
         /*(expecting only MEM_CHUNK if dest cq starts w/ MEM_CHUNK)*/
         /*(use less efficient fallback if that assumption does not hold true)*/

RE: Build issue - Added by gstrauss 2 months ago

Separately, you noted that the response still contained Transfer-Encoding: chunked, even with server.stream-response-body = 0. I am curious: Is your FastCGI producing a response containing Transfer-Encoding: chunked? That's not necessary with FastCGI, and also not recommended with FastCGI, as lighttpd has to decode the chunked encoding when framing for HTTP/2 (and for HTTP/1.0).

RE: Build issue - Added by Agossi 2 months ago

Hi,

I have tried the patch without the #undef but the result is still the "curl: (56) Malformed encoding found in chunked-encoding".

Separately, you noted that the response still contained Transfer-Encoding: chunked, even with server.stream-response-body = 0. I am curious: Is your FastCGI producing a response containing Transfer-Encoding: chunked? That's not necessary with FastCGI, and also not recommended with FastCGI, as lighttpd has to decode the chunked encoding when framing for HTTP/2 (and for HTTP/1.0).

=> good question I am just using php7-cgi

php-cgi -v
PHP 7.3.26 (cgi-fcgi) (built: Jan 19 2021 07:52:30)
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.26, Copyright (c) 1998-2018 Zend Technologies

with the following lighttpd config file:

    fastcgi.server             = ( ".php" =>
                               ( "localhost" =>
                                 (
                                   "socket" => "/tmp/php-fastcgi.socket",
                                   "bin-path" => "/usr/bin/php-cgi",
                                   "max-procs" => 10,
                                   "bin-environment" =>
                                   (
                                     "PHP_FCGI_CHILDREN" => "5",
                                     "PHP_FCGI_MAX_REQUESTS" => "500" 
                                   ),
                 "allow-x-send-file" => "enable" 
                                 )
                               )

.....

I have again checked my configs and found a later occurence of the server.stream-response-body set to 2 which has overwritten the set of 0.
Sorry I missed the end of the config files, my fault.

I also checked the history for this "server.stream-response-body = 2 " change and this was made because of limited space in the embedded system

I retried curl with the patched chunk.c and reponse-body now really at 0:

curl -v "http://192.168.1.12/index-test.php" -o file.ttt
*   Trying 192.168.1.12...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 192.168.1.12 (192.168.1.12) port 80 (#0)
> GET /index-test.php HTTP/1.1
> Host: 192.168.1.12
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Set-Cookie: PHPSESSID=1hnvmn3etckmh2ibt692nt36gtj0l7tuvgoec8c0prv0890a; path=/; HttpOnly; SameSite=Strict
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-type: text/html; charset=UTF-8
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15768000; includeSubdomains
< Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data:; object-src 'none'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: same-origin
< Accept-Ranges: bytes
< Content-Length: 309202
< Date: Wed, 16 Mar 2022 09:09:06 GMT
<
{ [2808 bytes data]
* transfer closed with 243674 bytes remaining to read
 21  301k   21 65528    0     0  65528      0  0:00:04 --:--:--  0:00:04  120k
* Closing connection 0
curl: (18) transfer closed with 243674 bytes remaining to read

and again without any patch but set to 0:

curl -v "http://192.168.1.12/index-test.php" -o file.ttt
*   Trying 192.168.1.12...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 192.168.1.12 (192.168.1.12) port 80 (#0)
> GET /index-test.php HTTP/1.1
> Host: 192.168.1.12
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Set-Cookie: PHPSESSID=d10p6hcfso0kn2ob0ohjdtvfp6td6g9rqibf9koq9re9gog0; path=/; HttpOnly; SameSite=Strict
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-type: text/html; charset=UTF-8
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15768000; includeSubdomains
< Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data:; object-src 'none'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: same-origin
< Accept-Ranges: bytes
< Content-Length: 309202
< Date: Wed, 16 Mar 2022 09:19:58 GMT
<
{ [2808 bytes data]
* transfer closed with 178146 bytes remaining to read
 42  301k   42  127k    0     0   127k      0  0:00:02 --:--:--  0:00:02  151k
* Closing connection 0
curl: (18) transfer closed with 178146 bytes remaining to read

RE: Build issue - Added by gstrauss 2 months ago

I also checked the history for this "server.stream-response-body = 2 " change and this was made because of limited space in the embedded system

It is fine to use server.stream-response-body = 2. I was asking to help me to reproduce the issue. I asked you to confirm since the results you initially shared (with Transfer-Encoding: chunked) were not expected with @server.stream-response-body = 0 (which should result in Content-Length)

I have tried the patch without the #undef but the result is still the "curl: (56) Malformed encoding found in chunked-encoding".

I have been staring at the code and do not yet see anything amiss. Would you please double check that you are using lighttpd 1.4.64 with no other changes besides the small patch above in https://redmine.lighttpd.net/boards/2/topics/10325?r=10337#message-10337 ? It's great that you've narrowed down the issue to how lighttpd is trying to optimize chunk.c writes using pwritev() while lighttpd is streaming the response, so we must be getting close.

RE: Build issue - Added by gstrauss 2 months ago

Can you grab an strace -s 1024 during a failure? I am curious what pwritev() is returning on your system right before a failure.

RE: Build issue - Added by gstrauss 2 months ago

Any hint is welcome or anything I have to look like in linux which lighttpd now depends on since lighttpd 1.4.61 which may not be supported by older distros?

As a sanity check, please test lighttpd with HAVE_PWRITEV defined but with a substitution to use lseek() and writev() instead of pwritev()

--- a/src/chunk.c
+++ b/src/chunk.c
@@ -988,7 +988,8 @@ static ssize_t chunkqueue_append_cqmem_to_tempfile(chunkqueue * const restrict d
     if (c->file.fd < 0) return -1;
   #endif
     /* coverity[negative_returns : FALSE] */
-    ssize_t wr = pwritev(c->file.fd, iov, (int)iovcnt, c->file.length);
+    if (-1 == lseek(c->file.fd, c->file.length, SEEK_SET)) return -1;
+    ssize_t wr = writev(c->file.fd, iov, (int)iovcnt);

     /*(memory use in chunkqueues is expected to be limited before spilling
      * to tempfiles, so common case will write entire iovec to tempfile,

RE: Build issue - Added by Agossi 2 months ago

As a sanity check, please test lighttpd with HAVE_PWRITEV defined but with a substitution to use lseek() and writev() instead of pwritev()

=> I kept the patch1 and the lseek , writev patch2 and now the result is:

curl -v "http://192.168.1.12/index-test.php" -o ttt.ttt
*   Trying 192.168.1.12...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 192.168.1.12 (192.168.1.12) port 80 (#0)
> GET /index-test.php HTTP/1.1
> Host: 192.168.1.12
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Set-Cookie: PHPSESSID=pbj4odtv74jdd1lev0j2p7m7m4fingj9ee9uradqv4c5m5le; path=/; HttpOnly; SameSite=Strict
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-type: text/html; charset=UTF-8
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15768000; includeSubdomains
< Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data:; object-src 'none'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: same-origin
< Transfer-Encoding: chunked
< Date: Thu, 17 Mar 2022 03:39:01 GMT
<
{ [10416 bytes data]
100  301k    0  301k    0     0   301k      0 --:--:-- --:--:-- --:--:--  410k
* Connection #0 to host 192.168.1.12 left intact

Can you grab an strace -s 1024 during a failure? I am curious what pwritev() is returning on your system right before a failure.

=> result of strace is attached with no patches added (failing orginal src).

Best regards

strace_fail_chunked.txt (10.9 KB) strace_fail_chunked.txt failed (orginal src)

RE: Build issue - Added by gstrauss 2 months ago

In the strace, I see the calls to pwritev are writing the number of bytes submitted. That is good. However, the 4th argument to pwritev is the file offset, and that looks obscenely large. Does an strace which succeeds look similar for pwritev?

As far as substituting lseek and writev for pwritev, if that works for you in lighttpd 1.4.64 without any other patches and with HAVE_PWRITEV defined, then that would suggest that pwritev is not operating correctly on your system/libc. What is the kernel? What is the version of libc? Is it glibc? musl? Something else? Maybe lighttpd build scripts are detecting pwritev() support in libc, but the kernel does not support it properly.

Are you building on the same system on which you are running lighttpd? Or are you cross-compiling on a different system?

If substituting lseek and writev for pwritev works (with no other changes), then this is not a bug in lighttpd, though I would appreciate help in modifying the lighttpd build to detect invalid pwritev handling by kernel and/or libc so that HAVE_PWRITEV could be disabled in the build.

Also, if pwritev() is not working properly on your system, then it would also be prudent to test if pwrite is working properly.

RE: Build issue - Added by gstrauss 2 months ago

Correction: in the strace, I see system calls to pwrite(), not to pwritev().

If you have ltrace on your system, you might check that lighttpd is calling pwritev() with expected arguments.
(BTW, it is possible that the libc implementation might use system call for pwrite() if iovcnt passed to pwritev() is 1)

RE: Build issue - Added by Agossi 2 months ago

Are you building on the same system on which you are running lighttpd? Or are you cross-compiling on a different system?

=> I am cross compiling the lighttpd in Ubutnu for the arm embedded.

If you have ltrace on your system, you might check that lighttpd is calling pwritev() with expected arguments.

=> Sorry I do not have ltrace on the system.

What is the kernel? What is the version of libc? Is it glibc? musl?

=>
Libc:

/lib/libc.so.6
GNU C Library (EGLIBC) stable release version 2.10.1
Compiled by GNU CC version 4.4.2.

Kernel:

3.8.6 armv6l GNU/Linux

I hope this helps.

RE: Build issue - Added by gstrauss 2 months ago

I hope this helps.

Well, I hope this helps you. My conclusion is that this is not a bug in lighttpd. When cross-compiling, you probably should disable pwrite and pwritev if those are causing you issues.

According to https://sourceware.org/glibc/wiki/Glibc%20Timeline, glibc 2.10.1 was released 2009.05.18. eglibc was based on the glibc 2.10 branch, so I presume it is about as ancient.

What kind of system that ancient is worth your time to maintain rather than to schedule hardware upgrade? (A ROCKPro64 w/ 4GB RAM for $80 is likely more powerful than your current hardware, and if not, then you should not be running such ancient software.)

Please note well that this has been a huge waste of my time -- for which I am not compensated in any way -- and is lost time that could have been spent on more productive development of lighttpd.


BTW, the man page for pwrite explains what we see in strace

   C library/kernel differences
       The raw preadv() and pwritev() system calls have call signatures that differ slightly from that of the corresponding GNU C library wrapper functions shown in the SYNOPSIS.  The final argument, offset, is unpacked  by  the
       wrapper functions into two arguments in the system calls:

           unsigned long pos_l, unsigned long pos

       These arguments contain, respectively, the low order and high order 32 bits of offset.

RE: Build issue - Added by Agossi 2 months ago

Okay, thank you for you help.

It is not that easy to just upgrade to a new system, but you are right, I shall consider an upgrade until HW is EOL.

Sorry that you have spent so much time in that, but I was not aware of the details.

So I will just unset the support for PWRITEV and use lighttpd.

Many thanks again for your time.

    (1-23/23)