[Solved] Build issue
Added by Agossi about 3 years ago
Hi there,
I have issues with build starting from 1.4.61-64 after a cross compiling (successful build). GCC Version (GCC) 4.4.2
lighttpd starts also without any issue.
I get the following error in the WebBrowsers:
GET http://IP/index.php net::ERR_INVALID_CHUNKED_ENCODING 200 (OK)
I tried also to just get the page via wget and this was the result:
index.php [ <=> ] 127,98K --.-KB/s in 0,005s 2022-03-11 13:38:06 (24,5 MB/s) - *Read error at byte 131056* (Success).Retrying. --2022-03-11 13:38:10-- (try: 5) http://IP/index.php Connecting to localhost (localhost)|127.0.0.1|:3128... connected. Proxy request sent, awaiting response... 200 OK *Length: unspecified [text/html]* Saving to: ‘index.php’ index.php [ <=> ] 127,98K --.-KB/s in 0,006s 2022-03-11 13:38:11 (22,4 MB/s) - *Read error at byte 131056* (Success).Retrying.
Any hint where I can look to fix this issue? Version 1.4.60 still works on the system.
Best Regards
Replies (23)
RE: Build issue - Added by gstrauss about 3 years ago
insufficient information. please try to add more details.
RE: Build issue - Added by Agossi about 3 years ago
Configure used:
./configure --with-openssl \
--without-libev \
--without-mysql \
--without-pgsql \
--without-dbi \
--without-sasl \
--without-ldap \
--without-pam \
--without-wolfssl \
--with-pcre \
--without-pcre2 \
--without-brotli \
--without-fam \
--without-xxhash \
--without-maxminddb \
--host=arm-none-linux-gnueabi \
CFLAGS='-std=c99'
Results In:
Plugins: enabled: mod_access mod_accesslog mod_ajp13 mod_alias mod_auth mod_authn_file mod_cgi mod_deflate mod_dirlisting mod_evhost mod_expire mod_extforward mod_fastcgi mod_indexfile mod_openssl mod_proxy mod_redirect mod_rewrite mod_rrdtool mod_scgi mod_secdownload mod_setenv mod_simple_vhost mod_sockproxy mod_ssi mod_staticfile mod_status mod_userdir mod_usertrack mod_vhostdb mod_webdav mod_wstunnel disabled: mod_authn_dbi mod_authn_gssapi mod_authn_ldap mod_authn_pam mod_gnutls mod_magnet mod_maxminddb mod_mbedtls mod_nss mod_vhostdb_dbi mod_vhostdb_ldap mod_vhostdb_mysql mod_vhostdb_pgsql mod_wolfssl Features: enabled: auth-crypt compress-deflate compress-gzip large-files network-ipv6 network-openssl regex-conditionals stat-cache-inotify disabled: compress-brotli compress-bzip2 dbi kerberos ldap lua maxminddb mysql network-gnutls network-mbedtls network-nss network-wolfssl pam postgresql webdav-locks webdav-properties
now calling the make lead to this issue:
-Wshadow -pedantic -MT t_test_mod-stat_cache.o -MD -MP -MF .deps/t_test_mod-stat_cache.Tpo -c -o t_test_mod-stat_cache.o `test -f 'stat_cache.c' || echo './'`stat_cache.c stat_cache.c: In function ‘fam_dir_monitor’: stat_cache.c:723: error: ‘IN_EXCL_UNLINK’ undeclared (first use in this function) stat_cache.c:723: error: (Each undeclared identifier is reported only once stat_cache.c:723: error: for each function it appears in.) make[3]: *** [Makefile:4181: t_test_mod-stat_cache.o] Error 1 make[2]: *** [Makefile:1768: all] Error 2 make[1]: *** [Makefile:470: all-recursive] Error 1 make: *** [Makefile:400: all] Error 2
which got fixed in stat_cache.c by:
#ifndef IN_EXCL_UNLINK
/* supported in Kernel 3.8.6 but not in default includes set it hard*/
#define IN_EXCL_UNLINK 0x04000000
#endif
CALLIN make again lead to next issue:
-Wshadow -pedantic -MT ls-hpack/lighttpd-lshpack.o -MD -MP -MF ls-hpack/.deps/lighttpd-lshpack.Tpo -c -o ls-hpack/lighttpd-lshpack.o `test -f 'ls-hpack/lshpack.c' || echo './'`ls-hpack/lshpack.c
In file included from ls-hpack/lshpack.h:34,
from ls-hpack/lshpack.c:40:
ls-hpack/lsxpack_header.h:59: warning: type of bit-field ‘flags’ is a GCC extension
ls-hpack/lshpack.c:687: warning: unknown option after ‘#pragma GCC diagnostic’ kind
ls-hpack/lshpack.c: In function ‘lshpack_enc_huff_encode’:
ls-hpack/lshpack.c:704: error: #pragma GCC diagnostic not allowed inside functions
ls-hpack/lshpack.c:705: error: #pragma GCC diagnostic not allowed inside functions
ls-hpack/lshpack.c:706: error: #pragma GCC diagnostic not allowed inside functions
ls-hpack/lshpack.c:821: error: #pragma GCC diagnostic not allowed inside functions
ls-hpack/lshpack.c: At top level:
ls-hpack/lshpack.c:2062: warning: expected [error|warning|ignored] after ‘#pragma GCC diagnostic’
make[3]: *** [Makefile:2809: ls-hpack/lighttpd-lshpack.o] Error 1
make[2]: *** [Makefile:1768: all] Error 2
make[1]: *** [Makefile:470: all-recursive] Error 1
make: *** [Makefile:400: all] Error 2
This was "fixed" by adding && GNUC >= 5 to the occurences.
After these changes the lighttpd and modules got build, but lead to the above issue described.
Config of lighttpd used:
config {
var.CWD = "/var/volatile/tmp"
var.PID = 7851
server.modules = (
"mod_rewrite",
"mod_access",
"mod_auth",
"mod_setenv",
"mod_fastcgi",
"mod_expire",
"mod_evasive",
"mod_openssl",
)
server.document-root = "/opt/web/"
server.errorlog = "/var/log/lighttpd.error.log"
index-file.names = ("index.php", "index.html", "index.htm", "default.htm")
mimetype.assign = (
".pdf" => "application/pdf",
".sig" => "application/pgp-signature",
".spl" => "application/futuresplash",
".class" => "application/octet-stream",
".ps" => "application/postscript",
".torrent" => "application/x-bittorrent",
".dvi" => "application/x-dvi",
".gz" => "application/x-gzip",
".pac" => "application/x-ns-proxy-autoconfig",
".swf" => "application/x-shockwave-flash",
".tar.gz" => "application/x-tgz",
".tgz" => "application/x-tgz",
".tar" => "application/x-tar",
".zip" => "application/zip",
".mp3" => "audio/mpeg",
".m3u" => "audio/x-mpegurl",
".wma" => "audio/x-ms-wma",
".wax" => "audio/x-ms-wax",
".ogg" => "application/ogg",
".wav" => "audio/x-wav",
".gif" => "image/gif",
".jpg" => "image/jpeg",
".jpeg" => "image/jpeg",
".png" => "image/png",
".xbm" => "image/x-xbitmap",
".xpm" => "image/x-xpixmap",
".xwd" => "image/x-xwindowdump",
".css" => "text/css",
".html" => "text/html",
".htm" => "text/html",
".js" => "text/javascript",
".asc" => "text/plain",
".c" => "text/plain",
".cpp" => "text/plain",
".log" => "text/plain",
".conf" => "text/plain",
".text" => "text/plain",
".txt" => "text/plain",
".dtd" => "text/xml",
".xml" => "text/xml",
".mpeg" => "video/mpeg",
".mpg" => "video/mpeg",
".mov" => "video/quicktime",
".qt" => "video/quicktime",
".avi" => "video/x-msvideo",
".asf" => "video/x-ms-asf",
".asx" => "video/x-ms-asf",
".wmv" => "video/x-ms-wmv",
".bz2" => "application/x-bzip",
".tbz" => "application/x-bzip-compressed-tar",
".tar.bz2" => "application/x-bzip-compressed-tar"
)
server.tag = ""
evasive.max-conns-per-ip = 50
evasive.http-status = 304
evasive.retry-after = 10
static-file.exclude-extensions = (".php", ".pl", ".fcgi")
url.rewrite-if-not-file = (
"^/(.*)/$" => "/error404.php",
)
$HTTP["url"] =~ "\.pdf$" {
# block 1
server.range-requests = "disable"
} # end of $HTTP["url"] =~ "\.pdf$"
$SERVER["socket"] == "0.0.0.0:80" {
# block 2
} # end of $SERVER["socket"] == "0.0.0.0:80"
$SERVER["socket"] == "[::]:80" {
# block 3
server.use-ipv6 = "enable"
} # end of $SERVER["socket"] == "[::]:80"
$SERVER["socket"] =~ ":(80)" {
# block 4
fastcgi.debug = 1
server.stream-response-body = 2
fastcgi.server = (
".php" => (
"localhost" => (
"socket" => "/var/tmp/php-fastcgi.socket",
"bin-path" => "/usr/bin/php-cgi",
"max-procs" => 10,
"bin-environment" => (
"PHP_FCGI_CHILDREN" => "5",
"PHP_FCGI_MAX_REQUESTS" => "500",
),
"allow-x-send-file" => "enable",
),
),
)
} # end of $SERVER["socket"] =~ ":(80)"
}
Hope this info helps to dig inot the issue.
RE: Build issue - Added by gstrauss about 3 years ago
Hope this info helps to dig inot the issue.
The config may be useful as prerequisite information for context, but is lacking any further description of the problem.
Request and response headers? A description of what index.php does?
Configuration: Debug Variables
FYI: #pragma GCC diagnostic push/pop was added in GCC 4.6, I believe.
RE: Build issue - Added by gstrauss about 3 years ago
Connecting to localhost (localhost)|127.0.0.1|:3128... connected. Proxy request sent, awaiting response... 200 OK *Length: unspecified [text/html]*
Looks like your client is connecting through a proxy. Given the age of your compiler, I wouldn't be surprised to find out that the intermediate proxy is also ancient and buggy. Since length wasn't provided in the response, is wget sending an HTTP/1.1 request and the proxy downgrading the request to HTTP/1.0? Does the intermediate proxy not support Transfer-Encoding: chunked. Have you attempted the download with a modern version of curl? Modern curl also supports HTTP/2.
Does the problem occur with server.stream-response-body = 0? (It should not, since lighttpd will capture the complete response and then will send Content-Length with response)
RE: Build issue - Added by Agossi about 3 years ago
Hi,
I have done some debug setup an the issue still occurs.
I have created a simple PHP File (index.php) which just shall print out 10.000 lines (attached).
Using "server.stream-response-body = 0" did not have any effect on the issue that the response is not completly returned.
I also disabled the proxy and now directly connect to the system, but still same results.
Using CURL:
Lighttpd 1.4.64:
curl "http://192.168.1.12/index.php" -o test_debugout_1.4.64.html -v
* Trying 192.168.1.12...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.1.12 (192.168.1.12) port 80 (#0)
> GET /index.php HTTP/1.1
> Host: 192.168.1.12
> User-Agent: curl/7.55.1
> Accept: */*
>
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0< HTTP/1.1 200 OK
< Set-Cookie: PHPSESSID=6a6b163i9vp6ovgn7edar0gpph2m8qgkq22rlfejmpl0ljet; path=/; HttpOnly; SameSite=Strict
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-type: text/html; charset=UTF-8
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15768000; includeSubdomains
< Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data:; object-src 'none'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: same-origin
< Transfer-Encoding: chunked
< Date: Mon, 14 Mar 2022 05:26:56 GMT
<
{ [10416 bytes data]
* Malformed encoding found in chunked-encoding
100 62490 0 62490 0 0 62490 0 --:--:-- --:--:-- --:--:-- 105k
* Closing connection 0
curl: (56) Malformed encoding found in chunked-encoding
Lighttpd: 1.4.60:
curl "http://192.168.1.12/index.php" -o test_debugout_1.4.60.html -v
* Trying 192.168.1.12...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.1.12 (192.168.1.12) port 80 (#0)
> GET /index.php HTTP/1.1
> Host: 192.168.1.12
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Set-Cookie: PHPSESSID=3ts9va1pn6m2kcl6kv6dn97aftv63kbrjat09m24b2vaivl4; path=/; HttpOnly; SameSite=Strict
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-type: text/html; charset=UTF-8
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15768000; includeSubdomains
< Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data:; object-src 'none'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: same-origin
< Transfer-Encoding: chunked
< Date: Mon, 14 Mar 2022 05:24:34 GMT
<
{ [10416 bytes data]
100 301k 0 301k 0 0 301k 0 --:--:-- --:--:-- --:--:-- 321k
* Connection #0 to host 192.168.1.12 left intact
So if I use the old lighttpd 1.4.60 everthing works fine with the same config.
After switching to lighttpd 1.4.61,62,63 or 64 I get the "Malformed encoding found in chunked-encoding" errors.
Also the responses in lighttpd 1.4.64 using curl stop at iteration
Zeile 9160 von 10000 Zeile 9161
with 1.4.60 no issue:
Zeile 9998 von 10000 Zeile 9999 von 10000 Zeile 10000 von 10000
Any hint is welcome or anything I have to look like in linux which lighttpd now depends on since lighttpd 1.4.61 which may not be supported by older distros?
| lighttpd_chunked_encoding_issue.zip (23.2 KB) lighttpd_chunked_encoding_issue.zip | lighttpd chunked encoding issue embedded old linux |
RE: Build issue - Added by gstrauss about 3 years ago
Thank you for trying to put together a test case.
I have tried to reproduce this using your test case, and accessing the server across a local network, but have not been able to trigger a failure. I tried different values of curl --limit-rate 100K and 10K to try to trigger the issue, too.
To try to narrow down if the issue is somewhere closer to lighttpd and the client, or closer to lighttpd and the backend, you might try running index.php using CGI instead of using mod_fastcgi, and see if that makes a difference.
Separately, since you noted that the issue does not occur with lighttpd 1.4.60, but does occur with lighttpd 1.4.61, if you have a chance, would you please try building and testing some commits between lighttpd 1.4.60 and lighttpd 1.4.61?
Does the problem occur with server.stream-response-body = 0? (It should not, since lighttpd will capture the complete response and then will send Content-Length with response)
Using "server.stream-response-body = 0" did not have any effect on the issue that the response is not completly returned.
So it still failed (sometimes) with server.stream-response-body = 0? Can you confirm those responses contained Content-Length response header, and did not contain Transfer-Encoding: chunked response header?
RE: Build issue - Added by Agossi about 3 years ago
Hi there,
To try to narrow down if the issue is somewhere closer to lighttpd and the client, or closer to lighttpd and the backend, you might try running index.php using CGI instead of using mod_fastcgi, and see if that makes a difference.
=> without fastcgi it works. using the normal mod_cgi
So it still failed (sometimes) with server.stream-response-body = 0? Can you confirm those responses contained Content-Length response header, and did not contain Transfer-Encoding: chunked response header?
=> setting it to 0 does not seem to change anything in Transfer Encoding
mod_fastcgi and server.stream-response-body = 0:
curl -v "http://192.168.1.12/index-test.php" -o file.ttt
* Trying 192.168.1.12...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.1.12 (192.168.1.12) port 80 (#0)
> GET /index-test.php HTTP/1.1
> Host: 192.168.1.12
> User-Agent: curl/7.55.1
> Accept: */*
>
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0< HTTP/1.1 200 OK
< Set-Cookie: PHPSESSID=SESSIONID; path=/; HttpOnly; SameSite=Strict
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-type: text/html; charset=UTF-8
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15768000; includeSubdomains
< Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data:; object-src 'none'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: same-origin
< Transfer-Encoding: chunked
< Date: Wed, 16 Mar 2022 04:11:04 GMT
<
{ [10416 bytes data]
* *Malformed encoding found in chunked-encoding
100 62490 0 62490 0 0 62490 0 --:--:-- --:--:-- --:--:-- 126k
* Closing connection 0
curl: (56) Malformed encoding found in chunked-encoding*
mod_cgi and server.stream-response-body = 0:
curl -v "http://192.168.1.12/index-test.php" -o file.ttt
* Trying 192.168.1.12...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.1.12 (192.168.1.12) port 80 (#0)
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0> GET /index-test.php HTTP/1.1
> Host: 192.168.1.12
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Set-Cookie: PHPSESSID=SESSIONID; path=/; HttpOnly; SameSite=Strict
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-type: text/html; charset=UTF-8
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15768000; includeSubdomains
< Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data:; object-src 'none'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: same-origin
< Transfer-Encoding: chunked
< Date: Wed, 16 Mar 2022 04:13:48 GMT
<
{ [2826 bytes data]
100 301k 0 301k 0 0 301k 0 --:--:-- 0:00:01 --:--:-- 264k
* Connection #0 to host 192.168.1.12 left intact
Separately, since you noted that the issue does not occur with lighttpd 1.4.60, but does occur with lighttpd 1.4.61, if you have a chance, would you please try building and testing some commits between lighttpd 1.4.60 and lighttpd 1.4.61?
=> I will try if I have some spare time, espacially in the mod_fastcgi section.
RE: Build issue - Added by Agossi about 3 years ago
Update:
I have replaced the functions "void chunkqueue_mark_written(chunkqueue *cq, off_t len)" and "chunkqueue_steal_with_tempfiles" in chunk.c from the 1.4.60 into 1.4.64 src
and now I do not get any chunked encoding issue.
curl -v "http://192.168.1.12/index-test.php" -o file.ttt
* Trying 192.168.1.12...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.1.12 (192.168.1.12) port 80 (#0)
> GET /index-test.php HTTP/1.1
> Host: 192.168.1.12
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Set-Cookie: PHPSESSID=pkso8m5e4oaohro29i06pbra1edajpipbsmiq7nrh7olses8; path=/; HttpOnly; SameSite=Strict
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-type: text/html; charset=UTF-8
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15768000; includeSubdomains
< Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data:; object-src 'none'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: same-origin
< Transfer-Encoding: chunked
< Date: Wed, 16 Mar 2022 05:25:39 GMT
<
{ [10416 bytes data]
100 301k 0 301k 0 0 301k 0 --:--:-- --:--:-- --:--:-- 351k
* Connection #0 to host 192.168.1.12 left intact
RE: Build issue - Added by Agossi about 3 years ago
Next Update:
If I manually modify the config.h file after running the configure command from:
/* Define to 1 if you have the `pwritev' function. */
#define HAVE_PWRITEV 1
to
/* Define to 1 if you have the `pwritev' function. */
#undef HAVE_PWRITEV
it also seem to work, but config.log shows that it should be supported by my system:
configure:16730: checking for pwritev configure:16730: arm-linux-gnueabi-gcc -o conftest -std=c99 -I/sysroot-target/usr/include -I/usr/include -D_REENTRANT -L/sysroot-target/usr/lib -L/opt/arm/arm-linux-gnueabi/usr/lib conftest.c -lrt >&5 configure:16730: $? = 0 configure:16730: result: yes
RE: Build issue - Added by gstrauss about 3 years ago
Thanks! I'm reviewing all the commits to chunk.c between lighttpd 1.4.60 and lighttpd 1.4.64. I am up to commit 3964a6ac, which uses pwritev and which is what it seems you have zeroed in on above.
RE: Build issue - Added by Agossi about 3 years ago
Thanks! I'm reviewing all the commits to chunk.c between lighttpd 1.4.60 and lighttpd 1.4.64. I am up to commit 3964a6ac, which uses pwritev and which is what it seems you have zeroed in on above.
=> Yes, that is correct by removing the support define (#undef) it does not show the issue.
RE: Build issue - Added by gstrauss about 3 years ago
This is not a fix, but please try lighttpd 1.4.64 with this patch (and without #undef HAVE_PWRITEV). The patch disables an optimization which tries to write existing data in the destination chunkqueue into temporary files along with data from the source chunkqueue, all in a single pwritev(). If this patch works for you, then it will help me to reproduce the bug. It's late and I'll have to pick this up again tomorrow.
--- a/src/chunk.c
+++ b/src/chunk.c
@@ -947,6 +947,9 @@ static ssize_t chunkqueue_append_cqmem_to_tempfile(chunkqueue * const restrict d
off_t dlen = 0;
chunk *c;
+ #if 1
+ c = dest->first;
+ #else
for (c = dest->first; c && c->type == MEM_CHUNK; c = c->next) {
const off_t clen = chunk_remaining_length(c);
iov[iovcnt].iov_base = c->mem->ptr + c->offset;
@@ -956,6 +959,7 @@ static ssize_t chunkqueue_append_cqmem_to_tempfile(chunkqueue * const restrict d
if (__builtin_expect( (iovcnt == sizeof(iov)/sizeof(*iov)), 0))
break; /*(not expecting large number of MEM_CHUNK)*/
}
+ #endif
if (__builtin_expect( (c != NULL), 0) && dest->first->type == MEM_CHUNK) {
/*(expecting only MEM_CHUNK if dest cq starts w/ MEM_CHUNK)*/
/*(use less efficient fallback if that assumption does not hold true)*/
RE: Build issue - Added by gstrauss about 3 years ago
Separately, you noted that the response still contained Transfer-Encoding: chunked, even with server.stream-response-body = 0. I am curious: Is your FastCGI producing a response containing Transfer-Encoding: chunked? That's not necessary with FastCGI, and also not recommended with FastCGI, as lighttpd has to decode the chunked encoding when framing for HTTP/2 (and for HTTP/1.0).
RE: Build issue - Added by Agossi about 3 years ago
Hi,
I have tried the patch without the #undef but the result is still the "curl: (56) Malformed encoding found in chunked-encoding".
Separately, you noted that the response still contained Transfer-Encoding: chunked, even with server.stream-response-body = 0. I am curious: Is your FastCGI producing a response containing Transfer-Encoding: chunked? That's not necessary with FastCGI, and also not recommended with FastCGI, as lighttpd has to decode the chunked encoding when framing for HTTP/2 (and for HTTP/1.0).
=> good question I am just using php7-cgi
php-cgi -v PHP 7.3.26 (cgi-fcgi) (built: Jan 19 2021 07:52:30) Copyright (c) 1997-2018 The PHP Group Zend Engine v3.3.26, Copyright (c) 1998-2018 Zend Technologies
with the following lighttpd config file:
fastcgi.server = ( ".php" =>
( "localhost" =>
(
"socket" => "/tmp/php-fastcgi.socket",
"bin-path" => "/usr/bin/php-cgi",
"max-procs" => 10,
"bin-environment" =>
(
"PHP_FCGI_CHILDREN" => "5",
"PHP_FCGI_MAX_REQUESTS" => "500"
),
"allow-x-send-file" => "enable"
)
)
.....
I have again checked my configs and found a later occurence of the server.stream-response-body set to 2 which has overwritten the set of 0.
Sorry I missed the end of the config files, my fault.
I also checked the history for this "server.stream-response-body = 2 " change and this was made because of limited space in the embedded system
I retried curl with the patched chunk.c and reponse-body now really at 0:
curl -v "http://192.168.1.12/index-test.php" -o file.ttt
* Trying 192.168.1.12...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.1.12 (192.168.1.12) port 80 (#0)
> GET /index-test.php HTTP/1.1
> Host: 192.168.1.12
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Set-Cookie: PHPSESSID=1hnvmn3etckmh2ibt692nt36gtj0l7tuvgoec8c0prv0890a; path=/; HttpOnly; SameSite=Strict
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-type: text/html; charset=UTF-8
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15768000; includeSubdomains
< Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data:; object-src 'none'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: same-origin
< Accept-Ranges: bytes
< Content-Length: 309202
< Date: Wed, 16 Mar 2022 09:09:06 GMT
<
{ [2808 bytes data]
* transfer closed with 243674 bytes remaining to read
21 301k 21 65528 0 0 65528 0 0:00:04 --:--:-- 0:00:04 120k
* Closing connection 0
curl: (18) transfer closed with 243674 bytes remaining to read
and again without any patch but set to 0:
curl -v "http://192.168.1.12/index-test.php" -o file.ttt
* Trying 192.168.1.12...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.1.12 (192.168.1.12) port 80 (#0)
> GET /index-test.php HTTP/1.1
> Host: 192.168.1.12
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Set-Cookie: PHPSESSID=d10p6hcfso0kn2ob0ohjdtvfp6td6g9rqibf9koq9re9gog0; path=/; HttpOnly; SameSite=Strict
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-type: text/html; charset=UTF-8
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15768000; includeSubdomains
< Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data:; object-src 'none'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: same-origin
< Accept-Ranges: bytes
< Content-Length: 309202
< Date: Wed, 16 Mar 2022 09:19:58 GMT
<
{ [2808 bytes data]
* transfer closed with 178146 bytes remaining to read
42 301k 42 127k 0 0 127k 0 0:00:02 --:--:-- 0:00:02 151k
* Closing connection 0
curl: (18) transfer closed with 178146 bytes remaining to read
RE: Build issue - Added by gstrauss about 3 years ago
I also checked the history for this "server.stream-response-body = 2 " change and this was made because of limited space in the embedded system
It is fine to use server.stream-response-body = 2. I was asking to help me to reproduce the issue. I asked you to confirm since the results you initially shared (with Transfer-Encoding: chunked) were not expected with @server.stream-response-body = 0 (which should result in Content-Length)
I have tried the patch without the #undef but the result is still the "curl: (56) Malformed encoding found in chunked-encoding".
I have been staring at the code and do not yet see anything amiss. Would you please double check that you are using lighttpd 1.4.64 with no other changes besides the small patch above in https://redmine.lighttpd.net/boards/2/topics/10325?r=10337#message-10337 ? It's great that you've narrowed down the issue to how lighttpd is trying to optimize chunk.c writes using pwritev() while lighttpd is streaming the response, so we must be getting close.
RE: Build issue - Added by gstrauss about 3 years ago
Can you grab an strace -s 1024 during a failure? I am curious what pwritev() is returning on your system right before a failure.
RE: Build issue - Added by gstrauss about 3 years ago
Any hint is welcome or anything I have to look like in linux which lighttpd now depends on since lighttpd 1.4.61 which may not be supported by older distros?
As a sanity check, please test lighttpd with HAVE_PWRITEV defined but with a substitution to use lseek() and writev() instead of pwritev()
--- a/src/chunk.c
+++ b/src/chunk.c
@@ -988,7 +988,8 @@ static ssize_t chunkqueue_append_cqmem_to_tempfile(chunkqueue * const restrict d
if (c->file.fd < 0) return -1;
#endif
/* coverity[negative_returns : FALSE] */
- ssize_t wr = pwritev(c->file.fd, iov, (int)iovcnt, c->file.length);
+ if (-1 == lseek(c->file.fd, c->file.length, SEEK_SET)) return -1;
+ ssize_t wr = writev(c->file.fd, iov, (int)iovcnt);
/*(memory use in chunkqueues is expected to be limited before spilling
* to tempfiles, so common case will write entire iovec to tempfile,
RE: Build issue - Added by Agossi about 3 years ago
As a sanity check, please test lighttpd with HAVE_PWRITEV defined but with a substitution to use lseek() and writev() instead of pwritev()
=> I kept the patch1 and the lseek , writev patch2 and now the result is:
curl -v "http://192.168.1.12/index-test.php" -o ttt.ttt
* Trying 192.168.1.12...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.1.12 (192.168.1.12) port 80 (#0)
> GET /index-test.php HTTP/1.1
> Host: 192.168.1.12
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Set-Cookie: PHPSESSID=pbj4odtv74jdd1lev0j2p7m7m4fingj9ee9uradqv4c5m5le; path=/; HttpOnly; SameSite=Strict
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-type: text/html; charset=UTF-8
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=15768000; includeSubdomains
< Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data:; object-src 'none'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: same-origin
< Transfer-Encoding: chunked
< Date: Thu, 17 Mar 2022 03:39:01 GMT
<
{ [10416 bytes data]
100 301k 0 301k 0 0 301k 0 --:--:-- --:--:-- --:--:-- 410k
* Connection #0 to host 192.168.1.12 left intact
Can you grab an strace -s 1024 during a failure? I am curious what pwritev() is returning on your system right before a failure.
=> result of strace is attached with no patches added (failing orginal src).
Best regards
| strace_fail_chunked.txt (10.9 KB) strace_fail_chunked.txt | failed (orginal src) |
RE: Build issue - Added by gstrauss about 3 years ago
In the strace, I see the calls to pwritev are writing the number of bytes submitted. That is good. However, the 4th argument to pwritev is the file offset, and that looks obscenely large. Does an strace which succeeds look similar for pwritev?
As far as substituting lseek and writev for pwritev, if that works for you in lighttpd 1.4.64 without any other patches and with HAVE_PWRITEV defined, then that would suggest that pwritev is not operating correctly on your system/libc. What is the kernel? What is the version of libc? Is it glibc? musl? Something else? Maybe lighttpd build scripts are detecting pwritev() support in libc, but the kernel does not support it properly.
Are you building on the same system on which you are running lighttpd? Or are you cross-compiling on a different system?
If substituting lseek and writev for pwritev works (with no other changes), then this is not a bug in lighttpd, though I would appreciate help in modifying the lighttpd build to detect invalid pwritev handling by kernel and/or libc so that HAVE_PWRITEV could be disabled in the build.
Also, if pwritev() is not working properly on your system, then it would also be prudent to test if pwrite is working properly.
RE: Build issue - Added by gstrauss about 3 years ago
Correction: in the strace, I see system calls to pwrite(), not to pwritev().
If you have ltrace on your system, you might check that lighttpd is calling pwritev() with expected arguments.
(BTW, it is possible that the libc implementation might use system call for pwrite() if iovcnt passed to pwritev() is 1)
RE: Build issue - Added by Agossi about 3 years ago
Are you building on the same system on which you are running lighttpd? Or are you cross-compiling on a different system?
=> I am cross compiling the lighttpd in Ubutnu for the arm embedded.
If you have ltrace on your system, you might check that lighttpd is calling pwritev() with expected arguments.
=> Sorry I do not have ltrace on the system.
What is the kernel? What is the version of libc? Is it glibc? musl?
=>
Libc:
/lib/libc.so.6 GNU C Library (EGLIBC) stable release version 2.10.1 Compiled by GNU CC version 4.4.2.
Kernel:
3.8.6 armv6l GNU/Linux
I hope this helps.
RE: Build issue - Added by gstrauss about 3 years ago
I hope this helps.
Well, I hope this helps you. My conclusion is that this is not a bug in lighttpd. When cross-compiling, you probably should disable pwrite and pwritev if those are causing you issues.
According to https://sourceware.org/glibc/wiki/Glibc%20Timeline, glibc 2.10.1 was released 2009.05.18. eglibc was based on the glibc 2.10 branch, so I presume it is about as ancient.
What kind of system that ancient is worth your time to maintain rather than to schedule hardware upgrade? (A ROCKPro64 w/ 4GB RAM for $80 is likely more powerful than your current hardware, and if not, then you should not be running such ancient software.)
Please note well that this has been a huge waste of my time -- for which I am not compensated in any way -- and is lost time that could have been spent on more productive development of lighttpd.
BTW, the man page for pwrite explains what we see in strace
C library/kernel differences
The raw preadv() and pwritev() system calls have call signatures that differ slightly from that of the corresponding GNU C library wrapper functions shown in the SYNOPSIS. The final argument, offset, is unpacked by the
wrapper functions into two arguments in the system calls:
unsigned long pos_l, unsigned long pos
These arguments contain, respectively, the low order and high order 32 bits of offset.
RE: Build issue - Added by Agossi about 3 years ago
Okay, thank you for you help.
It is not that easy to just upgrade to a new system, but you are right, I shall consider an upgrade until HW is EOL.
Sorry that you have spent so much time in that, but I was not aware of the details.
So I will just unset the support for PWRITEV and use lighttpd.
Many thanks again for your time.