Project

General

Profile

Bug #2932

[regression] %0 pattern does not match hostnames without the domain part

Added by moisseev 17 days ago. Updated 5 days ago.

Status:
Fixed
Priority:
Normal
Assignee:
-
Category:
mod_evhost
Target version:
Start date:
2019-02-02
Due date:
% Done:

100%

Estimated time:
Missing in 1.5.x:

Description

%0 pattern in mod_evhost has stopped working if only hostname (without the domain) is used (http://hostname/) since 1.4.53.

output (31.9 KB) output moisseev, 2019-02-08 14:16

Associated revisions

Revision 9459c054 (diff)
Added by gstrauss 5 days ago

[core] fix mixed use of srv->split_vals array (fixes #2932)

regression in mod_evhost in lighttpd 1.4.53
regression in mod_flv_streaming in lighttpd 1.4.51 - lighttpd 1.4.53

(thx moisseev)

x-ref:
"[regression] %0 pattern does not match hostnames without the domain part"
https://redmine.lighttpd.net/issues/2932

History

#1

Updated by gstrauss 17 days ago

More details please. A quick modification of t/test_mod_evhost.c to test this scenario succeeds.

Please provide your lighttpd.conf or at least mod_evhost config and show what you expect and what you are getting. What is lighttpd attempting to access instead of your desired path? Is there any trace from mod_evhost in the lighttpd error log?

You marked this as a regression. Just to confirm: what is the previous version of lighttpd with which you have tested and works as you expect? lighttpd 1.4.52?

#2

Updated by moisseev 16 days ago

gstrauss wrote:

More details please. A quick modification of t/test_mod_evhost.c to test this scenario succeeds.

Please provide your lighttpd.conf or at least mod_evhost config and show what you expect and what you are getting. What is lighttpd attempting to access instead of your desired path? Is there any trace from mod_evhost in the lighttpd error log?

evhost.conf:

server.modules += ( "mod_evhost" )
evhost.path-pattern = "/usr/local/www/%0/" 

When requesting http://host1/, I expect to get /usr/local/www/host1/index.html, but I get /usr/local/www/index.html instead.
It works if I replace %0 with %1 pattern.

There is nothing related in the error.log. Should I enable debugging?

You marked this as a regression. Just to confirm: what is the previous version of lighttpd with which you have tested and works as you expect? lighttpd 1.4.52?

Yes, it works as expected with 1.4.52.

#3

Updated by gstrauss 16 days ago

  • Status changed from New to Need Feedback

There is likely something else going on since mod_evhost works fine for me when using the configuration snippet you have provided.

The factual observation you have made is that something works with lighttpd 1.4.52, but not with lighttpd 1.4.53, but you have not presented sufficient information to identify the actual problem. Your conclusion that there is a problem with mod_evhost is likely wrong, so please provide more information. If you do not wish to share your lighttpd.conf, then please try to reproduce the problem with a simplified lighttpd.conf. Please set debug.log-request-header = "enable" and capture the request headers, too.

#4

Updated by moisseev 15 days ago

For testing I am using the stock configuration installed by FreeBSD port www/lighttpd (lighttpd-1.4.53). The only difference is:

--- modules.conf.sample 2019-01-29 10:05:16.000000000 +0300
+++ modules.conf        2019-02-04 09:39:50.129861000 +0300
@@ -162,7 +162,7 @@
 ##
 ## mod_evhost
 ##
-#include "conf.d/evhost.conf" 
+include "conf.d/evhost.conf" 

 ##
 ## mod_simple_vhost
--- conf.d/evhost.conf.sample   2019-01-29 10:05:16.000000000 +0300
+++ conf.d/evhost.conf  2019-02-04 10:05:47.361329000 +0300
@@ -16,7 +16,7 @@
 ## %3 => subdomain 1 name
 ## %4 => subdomain 2 name
 ##
-evhost.path-pattern        = vhosts_dir + "/%3/htdocs/" 
+evhost.path-pattern        = "/usr/local/www/%0/" 

 ##
 #######################################################################
--- conf.d/debug.conf.sample    2019-01-29 10:05:16.000000000 +0300
+++ conf.d/debug.conf   2019-02-04 09:49:41.928635000 +0300
@@ -19,7 +19,7 @@
 ##
 ## log all request headers.
 ##
-#debug.log-request-header          = "enable" 
+debug.log-request-header          = "enable" 

 ##
 ## similar to log-request-header.

Request headers:

2019-02-04 09:56:25: (request.c.601) fd: 9 request-len: 469 \nGET / HTTP/1.1\r\nHost: host1\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.9) Gecko/20100101 Goanna/4.1 Firefox/60.9 PaleMoon/2
8.3.1\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: ru,en-US;q=0.7,en;q=0.3\r\nAccept-Encoding: gzip, deflate\r\nX-Compress: 1\r\nProxy-Authorization: 659fe3b
4f93eee5f90a698fb46cabc06c7937c0b678d7a9b4ba1002b58d69b3bbd13310d70e42a30\r\nConnection: keep-alive\r\nUpgrade-Insecure-Requests: 1\r\n\r\n
http://host1/

GET / HTTP/1.1
Host: host1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.9) Gecko/20100101 Goanna/4.1 Firefox/60.9 PaleMoon/28.3.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: ru,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
X-Compress: 1
Proxy-Authorization: 659fe3b4f93eee5f90a698fb46cabc06c7937c0b678d7a9b4ba1002b58d69b3bbd13310d70e42a30
Connection: keep-alive
Upgrade-Insecure-Requests: 1

HTTP/1.1 200 OK
Content-Type: text/html
Accept-Ranges: bytes
ETag: "3801695155" 
Last-Modified: Thu, 08 Nov 2018 14:09:07 GMT
Content-Length: 6
Date: Mon, 04 Feb 2019 06:56:25 GMT
Server: lighttpd/1.4.53
----------------------------------------------------------
#5

Updated by gstrauss 15 days ago

For testing I am using the stock configuration installed by FreeBSD port www/lighttpd (lighttpd-1.4.53).

That may be pertinent. I don't have a FreeBSD test system set up at the moment. You might want to experiment with building lighttpd from source to see if the issue persists.

#6

Updated by moisseev 13 days ago

I don't see how it helps. Building from port is just an automated way of building from source. When building from source we basically should do the same things but by hand. It is complicated and time-consuming. If you have any idea what can cause the issue, I would experiment with port modifications (change build options, startup script etc.).

#7

Updated by gstrauss 12 days ago

Try building from upstream source, as a non-privileged user, and skip the install step. Create a local config file separate from all the existing configs provided by the FreeBSD package and run that test config from the build location. Then see if you can reproduce the problem.

I could not reproduce the problem on a Linux system and I even added tests to the test drivers to validate that things were working as I expected.
https://git.lighttpd.net/lighttpd/lighttpd1.4.git/commit/?h=personal/gstrauss/master&id=142e54b2a877397e64bc6bb658f71f1c3c1923d8

#8

Updated by moisseev 12 days ago

I have no idea how to run lighttpd from the build location.

% ~/lighttpd-1.4.53/src/lighttpd -f ~/config/lighttpd.conf
2019-02-07 20:21:50: (plugin.c.230) dlopen() failed for: /usr/local/lib/mod_indexfile.so Cannot open "/usr/local/lib/mod_indexfile.so" 
2019-02-07 20:21:50: (server.c.1177) loading plugins finally failed

Should I build it statically or what?

#9

Updated by gstrauss 11 days ago

cd ~/lighttpd-1.4.53/src/
./lighttpd -D -f ~/config/lighttpd.conf -m $PWD/.libs

(Ctrl-C to quit)
#10

Updated by moisseev 11 days ago

Thank you for the explanation. I've reproduced the problem. The behavior is exactly the same as I described for lighttpd that was built from port.

Modifications I've made in the configuration:

# diff -ruN /home/moiseev/lighttpd-1.4.53/doc/config/ /home/moiseev/config/
diff -ruN /home/moiseev/lighttpd-1.4.53/doc/config/conf.d/evhost.conf /home/moiseev/config/conf.d/evhost.conf
--- /home/moiseev/lighttpd-1.4.53/doc/config/conf.d/evhost.conf 2019-01-27 12:22:20.000000000 +0300
+++ /home/moiseev/config/conf.d/evhost.conf     2019-02-08 09:55:31.982059000 +0300
@@ -16,7 +16,7 @@
 ## %3 => subdomain 1 name
 ## %4 => subdomain 2 name
 ##
-evhost.path-pattern        = vhosts_dir + "/%3/htdocs/" 
+evhost.path-pattern        = "/tmp/www/%0/" 

 ##
 #######################################################################
diff -ruN /home/moiseev/lighttpd-1.4.53/doc/config/lighttpd.conf /home/moiseev/config/lighttpd.conf
--- /home/moiseev/lighttpd-1.4.53/doc/config/lighttpd.conf      2019-01-27 12:22:20.000000000 +0300
+++ /home/moiseev/config/lighttpd.conf  2019-02-08 09:56:12.796802000 +0300
@@ -101,8 +101,8 @@
 ## Run as a different username/groupname.
 ## This requires root permissions during startup.
 ##
-server.username  = "lighttpd" 
-server.groupname = "lighttpd" 
+server.username  = "www" 
+server.groupname = "www" 

 ##
 ## enable core files.
@@ -178,7 +178,7 @@
 ##
 ## linux-sysepoll is recommended on kernel 2.6.
 ##
-server.event-handler = "linux-sysepoll" 
+server.event-handler = "freebsd-kqueue" 

 ##
 ## The basic network interface for all platforms at the syscalls read()
@@ -495,3 +495,4 @@
 #include "/etc/lighttpd/vhosts.d/*.conf" 
 ##
 #######################################################################
+$SERVER["socket"] == "0.0.0.0:80" { }
diff -ruN /home/moiseev/lighttpd-1.4.53/doc/config/modules.conf /home/moiseev/config/modules.conf
--- /home/moiseev/lighttpd-1.4.53/doc/config/modules.conf       2019-01-27 12:22:20.000000000 +0300
+++ /home/moiseev/config/modules.conf   2019-02-08 08:58:28.888771000 +0300
@@ -162,7 +162,7 @@
 ##
 ## mod_evhost
 ##
-#include "conf.d/evhost.conf" 
+include "conf.d/evhost.conf" 

 ##
 ## mod_simple_vhost

# tree /tmp/www
/tmp/www
├── host1
│   └── index.html
├── host2
│   └── index.html
└── index.html

2 directories, 3 files
#11

Updated by gstrauss 11 days ago

Please attach the output of

./lighttpd -f ~/config/lighttpd.conf -m $PWD/.libs -p

#12

Updated by moisseev 11 days ago

#13

Updated by gstrauss 8 days ago

I downloaded a FreeBSD 12.0 image, cloned lighttpd git DevelGit, and built the personal/gstrauss/master branch. The tests all pass, including the commit at the tip of personal/gstrauss/master which expands the tests for mod_evhost and explicitly exercises %0.

#14

Updated by gstrauss 8 days ago

  • Status changed from Need Feedback to Missing Feedback

I have confirmed that this works on FreeBSD 12.0 with a basic configuration.

Have you tested with no other modules loaded to ensure that mod_evhost is even running?
server.modules := ("mod_evhost")

#15

Updated by moisseev 8 days ago

I disabled "mod_access" and "mod_accesslog". Nothing has changed.

# ./lighttpd -D -f /home/moiseev/config/lighttpd.conf -m $PWD/.libs -p | grep server.modules
    server.modules                 = ("mod_evhost")

I think mod_evhost is running since it works when I change %0 pattern to %1.

I built the personal/gstrauss/master branch and tried to run my test config. The result is the same. Can you please give me the instruction on how to run the functional tests?

#16

Updated by gstrauss 8 days ago

$ make check

#17

Updated by moisseev 8 days ago

All 10 tests passed

#18

Updated by gstrauss 8 days ago

That the tests pass suggest that there is something specific to your system, and perhaps something else going on which manifests itself when mod_evhost tries to use con->uri.authority.

Try this:
debug.log-request-handling = "enable"
and looking in the error log to see what is reported for "URI-authority: ...", which is logged after the request is parsed.

Are you able to reproduce this on a different machine? I ask since I was not able to reproduce this.

#19

Updated by moisseev 8 days ago

2019-02-11 13:06:22: (response.c.429) -- splitting Request-URI
2019-02-11 13:06:22: (response.c.430) Request-URI     :  /
2019-02-11 13:06:22: (response.c.431) URI-scheme      :  http
2019-02-11 13:06:22: (response.c.432) URI-authority   :  host1
2019-02-11 13:06:22: (response.c.433) URI-path (raw)  :  /
2019-02-11 13:06:22: (response.c.434) URI-path (clean):  /
2019-02-11 13:06:22: (response.c.435) URI-query       :
2019-02-11 13:06:22: (response.c.580) -- before doc_root
2019-02-11 13:06:22: (response.c.581) Doc-Root     : /srv/www/htdocs
2019-02-11 13:06:22: (response.c.582) Rel-Path     : /
2019-02-11 13:06:22: (response.c.583) Path         :
2019-02-11 13:06:22: (response.c.625) -- after doc_root
2019-02-11 13:06:22: (response.c.626) Doc-Root     : /tmp/www//
2019-02-11 13:06:22: (response.c.627) Rel-Path     : /
2019-02-11 13:06:22: (response.c.628) Path         : /tmp/www//
2019-02-11 13:06:22: (response.c.652) -- logical -> physical
2019-02-11 13:06:22: (response.c.653) Doc-Root     : /tmp/www//
2019-02-11 13:06:22: (response.c.654) Basedir      : /tmp/www//
2019-02-11 13:06:22: (response.c.655) Rel-Path     : /
2019-02-11 13:06:22: (response.c.656) Path         : /tmp/www//
2019-02-11 13:06:22: (response.c.668) -- handling physical path
2019-02-11 13:06:22: (response.c.669) Path         : /tmp/www//
2019-02-11 13:06:22: (response.c.676) -- handling subrequest
2019-02-11 13:06:22: (response.c.677) Path         : /tmp/www//
2019-02-11 13:06:22: (response.c.678) URI          : /
2019-02-11 13:06:22: (response.c.679) Pathinfo     :
2019-02-11 13:06:22: (mod_indexfile.c.159) -- handling the request as Indexfile
2019-02-11 13:06:22: (mod_indexfile.c.160) URI          : /
2019-02-11 13:06:22: (mod_staticfile.c.197) -- handling file as static file

I will do clean install of FreeBSD and try to reproduce the issue.
#20

Updated by moisseev 7 days ago

I reproduced the issue on VMware ESXi 6.5 using clean image FreeBSD-12.0-RELEASE-amd64.vmdk .
I installed just lighttpd build dependencies and git with its dependencies.

# pkg info -q
autoconf-2.69_2
autoconf-wrapper-20131203
automake-1.16.1_1
ca_root_nss-3.42.1
curl-7.64.0
cvsps-2.1_2
expat-2.2.6_1
gettext-runtime-0.19.8.1_2
git-2.20.1
indexinfo-0.3.1
libffi-3.2.1_3
libnghttp2-1.35.1
libtool-2.4.6_1
m4-1.4.18_1,1
p5-Authen-SASL-2.16_1
p5-CGI-4.40
p5-Digest-HMAC-1.03_1
p5-Error-0.17027
p5-GSSAPI-0.28_1
p5-HTML-Parser-3.72
p5-HTML-Tagset-3.20_1
p5-IO-Socket-INET6-2.72_1
p5-IO-Socket-SSL-2.060
p5-Mozilla-CA-20180117
p5-Net-SSLeay-1.85
p5-Socket6-0.29
pcre-8.42_1
perl5-5.28.1
pkg-1.10.5_5
pkgconf-1.5.4,1
python27-2.7.15
readline-7.0.5

#21

Updated by gstrauss 7 days ago

Thank you for trying a different system.

Please duplicate with this patch, to get more information. If you get this trace, then mod_evhost is called. From your posts above, it looks like it is. The trace will validate what has been parsed in the request and may help point to a problem elsewhere.

--- a/src/mod_evhost.c
+++ b/src/mod_evhost.c
@@ -347,6 +347,14 @@ static handler_t mod_evhost_uri_handler(server *srv, connection *con, void *p_d)
                return HANDLER_GO_ON;
        }

+       {
+               data_string *ds;
+               array_reset_data_strings(srv->split_vals);
+               mod_evhost_parse_host(p->tmp_buf, srv->split_vals, con->uri.authority);
+               ds = (data_string *)array_get_element_klen(srv->split_vals, CONST_STR_LEN("%0"));
+               log_error_write(srv, __FILE__, __LINE__, "sbssd", "mod_evhost: authority:", con->uri.authority, "%0", ds && ds->value ? ds->value->ptr : "NULL", p->conf.len);
+       }
+
        mod_evhost_build_doc_root_path(p->tmp_buf, srv->split_vals, con->uri.authority, p->conf.path_pieces, p->conf.len);

        if (HANDLER_ERROR == stat_cache_get_entry(srv, con, p->tmp_buf, &sce)) {

#22

Updated by moisseev 7 days ago

2019-02-12 16:25:05: (mod_evhost.c.355) mod_evhost: authority: host1 %0 NULL 3 
#23

Updated by gstrauss 7 days ago

I am unable to reproduce this using a fresh FreeBSD-12.0-RELEASE-amd64.raw with only packages added so that I can build and run lighttpd.
I imagine that you're just as frustrated as I am. What compiler are you using? What does the compilation line for mod_evhost.c look like? You have mentioned that %1 works for you, and it is not immediately apparent to me why that would work but %0 would not, as %0 is created in the array unconditionally, but for you it is not found.

#24

Updated by gstrauss 6 days ago

On the off-chance that you have modified the output displayed to hide more sensitive data: are you using literal http://host1/ exactly like that?

#25

Updated by moisseev 6 days ago

# cc -v
FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)
Target: x86_64-unknown-freebsd12.0
Thread model: posix
InstalledDir: /usr/bin
# pkg install autoconf automake libtool m4 pkgconf
# cd ~/lighttpd1.4
# ./autogen.sh
# ./configure
# make
libtool: compile:  cc -DHAVE_CONFIG_H -DHAVE_VERSIONSTAMP_H -DLIBRARY_DIR=\"/usr/local/lib\" -DSBIN_DIR=\"/usr/local/sbin\" -I. -I.. -D_REENTRANT -I/usr/local/include -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGE_FILES -g -O2 -Wall -W -Wshadow -pedantic -MT mod_evhost.lo -MD -MP -MF .deps/mod_evhost.Tpo -c mod_evhost.c  -fPIC -DPIC -o .libs/mod_evhost.o
/bin/sh ../libtool  --tag=CC    --mode=link cc   -g -O2 -Wall -W -Wshadow -pedantic  -module -export-dynamic -avoid-version  -o mod_evhost.la -rpath /usr/local/lib mod_evhost.lo

It is a test setup, so I am not hiding anything. I am using http://host1/ exactly. I added host1 to the hosts file on the machine from where I am accessing the URI.

#26

Updated by gstrauss 6 days ago

I had been compiling with gcc. I compiled clean and then with clang, but still could not reproduce the issue. (clang 6.0.1 on FreeBSD 12.0)
I'll try rebuilding a clean image later this evening.

#27

Updated by gstrauss 6 days ago

I am unable to reproduce your issue with clean image. Would you be more precise in how you reproduce this?

# pkg install git autoconf automake libtool m4 pkgconf
# git clone https://github.com/lighttpd/lighttpd1.4
# cd lighttpd1.4
# git checkout lighttpd-1.4.53
# ./autogen.sh
# ./configure
# make
# cd src
# vi t.conf
  server.document-root = "/tmp" 
  server.modules = ("mod_evhost")
  evhost.path-pattern = "/tmp/www/%0/" 
# echo default > /tmp/index.txt
# mkdir -p /tmp/www/host1
# echo host1 > /tmp/www/host1/index.txt
# ./lighttpd -D -f t.conf -m $PWD/.libs &
# telnet localhost 80
  GET /index.txt HTTP/1.0
  Host: host1

#### result 200 OK with response body: "host1", *not* "default" 
# fg
# Ctrl-C

#28

Updated by moisseev 6 days ago

I have done exactly what you did (except I have applied your patch for tracing) and I get exactly the same result.
Previously I used a web browser for testing. I think Connection: keep-alive header is the culprit.
I extended your test as following:

# echo www > /tmp/www/index.txt
# ./lighttpd -D -f t.conf -m $PWD/.libs &

# telnet localhost 80

GET /index.txt HTTP/1.0
Host: host1

2019-02-13 17:40:07: (mod_evhost.c.355) mod_evhost: authority: host1 %0 host1 3
HTTP/1.0 200 OK

host1
Connection closed by foreign host.
# telnet localhost 80

GET /index.txt HTTP/1.0
Host: host1
Connection: keep-alive

2019-02-13 17:40:25: (mod_evhost.c.355) mod_evhost: authority: host1 %0 NULL 3
HTTP/1.0 200 OK

www
Connection closed by foreign host.
# telnet localhost 80

GET /index.txt HTTP/1.0
Host: host1

2019-02-13 17:40:50: (mod_evhost.c.355) mod_evhost: authority: host1 %0 NULL 3
HTTP/1.0 200 OK

www
Connection closed by foreign host.

#29

Updated by gstrauss 5 days ago

  • Status changed from Missing Feedback to Patch Pending
  • Target version changed from 1.4.x to 1.4.54

Aha! Thank you for your persistence. The allocated array was reused, and unfortunately there are two flavors of array: key/value and value-only (with generated key). The Connection header used it one way, and mod_evhost started using it the other way in lighttpd 1.4.53, and the reused data element messed up the first element ("%0") inserted into the array in mod_evhost. The only other module which reuses this array is mod_flv_streaming.c, and has likely suffered a similar problem since lighttpd 1.4.51, released 14 Oct 2018.

--- a/src/array.c
+++ b/src/array.c
@@ -64,6 +64,7 @@ void array_reset(array *a) {

        for (i = 0; i < a->used; i++) {
                a->data[i]->fn->reset(a->data[i]);
+               a->data[i]->is_index_key = 0;
        }

        a->used = 0;
@@ -76,6 +77,7 @@ void array_reset_data_strings(array *a) {
        for (size_t i = 0; i < a->used; ++i) {
                data_string * const ds = (data_string *)a->data[i];
                /*force_assert(ds->type == TYPE_STRING);*/
+               ds->is_index_key = 0;
                buffer_reset(ds->key);
                buffer_reset(ds->value);
        }
#30

Updated by gstrauss 5 days ago

  • Status changed from Patch Pending to Fixed
  • % Done changed from 0 to 100

Also available in: Atom