Project

General

Profile

Actions
Copy link

Bug #3046

closed

Failure on second request in http proxy backend

Added by flynn almost 4 years ago. Updated almost 4 years ago.

Status:
Fixed
Priority:
Normal
Category:
mod_proxy
Target version:
1.4.57
ASK QUESTIONS IN Forums:
No

Description

This new issue was found while testing/debugging #3044:
  • mode proxy in HTTP/1.1 mode server.feature-flags += ("proxy.force-http10" => "disable")
  • multiple requests in one TCP session, e.g. wget with multiple URLs in one cmdline
  • first request is responded fine
  • second request hangs after sending
I used tcpdump to debug the HTTP-headers and found:
  • wget is sending Connection: Keep-Alive to lighttpd
  • lighttpd is sending Connection: close to the backend, regardless the setting of proxy.force-http10
  • backend does not send any Connection header
  • after an ACK package on the first response no furhter TCP packet is send to the backend
I made also the following tests:
  • wget request with multiple urls on the same cmdline direct to the backend without lighttpd as proxy.
    The client (wget) sends Connection: Keep-Alive and the backend responds with Connection: Keep-Alive Header.
    This indicates to me, that the backend can handle HTTP/1.1 keep-alive correctly.
  • every thing works fine with lighttpd in HTTP/1.0 proxy mode with server.feature-flags += ("proxy.force-http10" => "enable")

Either lighttpd should forward Keep-alive header to the backend or close the connection after the first request, even if no Connection:close is in the response.

Actions
Copy link
 #1

Updated by gstrauss almost 4 years ago

  • Category changed from mod_proxy_backend_http to mod_proxy

mod_proxy always sends Connection: close to the backend, even with Connection: close, upgrade

I'll try to reproduce the behavior you are reporting to see when the connection to the backend is closed.

Actions
Copy link
 #2

Updated by gstrauss almost 4 years ago

RFC 7230 Section 6.1. Connection

   The "close" connection option is defined for a sender to signal that
   this connection will be closed after completion of the response.  For
   example,

     Connection: close

   in either the request or the response header fields indicates that
   the sender is going to close the connection after the current
   request/response is complete

lighttpd mod_proxy always sends Connection: close to a backend, whether HTTP/1.0 or HTTP/1.1.

When another lighttpd instance is the backend, that lighttpd backend responds with Connection: close and closes the connection, as expected from a polite backend.

lighttpd mod_proxy (incorrectly) expects such behavior, and does not look at the response, instead simply transferring data until the connection to the backend is closed. Such behavior happened to work when making HTTP/1.0 requests to backends, where Connection: close is the default, as is the behavior of any CGI gateway backend whose response does not provide Content-Length. However, with HTTP/1.1 requests, lighttpd should be more careful and should handle a) no Content-Length and no Transfer-Encoding: chunked, b) Content-Length c) Transfer-Encoding: chunked. In b) and c), lighttpd should -- but currently does not -- close the connection to the backend after the exact Content-Length specified has been received or after the Transfer-Encoding: chunked response-body-ending 0\r\n chunk (and optional trailers) have been received.

  • lighttpd currently does not track the Content-Length received from a backend.
  • lighttpd currently does not peek into Transfer-Encoding: chunked if the client is HTTP/1.1. chunked body from the backend is transferred to client as-is.
  • In the case of HTTP/2 client, lighttpd decodes HTTP/1.1 Transfer-Encoding: chunked from the backend, but mod_proxy does not check when the final chunk (and optional trailers) have been received.

I'll work on adding the necessary accounting for mod_proxy to track the state of the response data sent from the backend.

Actions
Copy link
 #3

Updated by gstrauss almost 4 years ago

I have patches for parsing chunked encoding from the backend, including the optimization to avoid copying data when we are (re-)sending chunked encoding to client.
lighttpd git on branch personal/gstrauss/master

Remaining work to do for this issue is to handle Content-Length, if provided, and to continue to handle case where Content-Length (and Transfer-Encoding: chunked) are not provided by the backend.

Actions
Copy link
 #4

Updated by gstrauss almost 4 years ago

I pushed a sketch of patches (untested) to my dev branch personal/gstrauss/master. When I am better rested tomorrow, I need to run through a battery of scenarios to check that it is working as desired and does not break existing usage. At the moment, I do not recommend these new patches on anything other than a test server.

Actions
Copy link
 #5

Updated by flynn almost 4 years ago

I stay with server.feature-flags += ("proxy.force-http10" => "enable") and wait until you have finished your patches.

Actions
Copy link
 #6

Updated by gstrauss almost 4 years ago

  • Status changed from New to Patch Pending

Patches are ready and tested. I will push them to the master branch shortly and this issue will be automatically closed with the patches. If the patches do not resolve this issue, please reopen this issue.

lighttpd now detects when the backend has completed sending the response (if Content-Length or Transfer-Encoding: chunked), and will close the connection to the backend.

lighttpd is now a bit stricter with responses from backends, and will (silently) truncate responses sent to the client if Content-Length is sent from the backend and then the backend sends more data than specified in Content-Length.

I have tested with server.stream-response-body = 0 and server.stream-response-body = 1 to lighttpd server running CGI, and to lighttpd server running mod_proxy back to a lighttpd server running CGI; with clients running HTTP/1.0, HTTP/1.1, and HTTP/2. The CGI script responds without Content-Length and without Transfer-Encoding: chunked; with Content-Length; or with Transfer-Encoding: chunked

#!/bin/sh
set -x
curl --http1.0 "http://127.0.0.1:8080/cgi.pl" 
curl --http1.0 "http://127.0.0.1:8080/cgi.pl?c" 
curl --http1.0 "http://127.0.0.1:8080/cgi.pl?t" 
curl --http1.1 "http://127.0.0.1:8080/cgi.pl" 
curl --http1.1 "http://127.0.0.1:8080/cgi.pl?c" 
curl --http1.1 "http://127.0.0.1:8080/cgi.pl?t" 
curl --http2-prior-knowledge "http://127.0.0.1:8080/cgi.pl" 
curl --http2-prior-knowledge "http://127.0.0.1:8080/cgi.pl?c" 
curl --http2-prior-knowledge "http://127.0.0.1:8080/cgi.pl?t"

cgi.pl

#!/bin/sh

echo 1>&2 "query-string: $QUERY_STRING" 

if [[ "$QUERY_STRING" = "c" ]]; then
  echo "Status: 200" 
  echo "Content-Length: 6" 
  echo
  sleep 1
  echo a
  sleep 1
  echo b
  sleep 1
  echo c
  sleep 1
  echo excess
elif [[ "$QUERY_STRING" = "t" ]]; then
  echo "Status: 200" 
  echo "Transfer-Encoding: chunked" 
  echo
  sleep 1
  printf "2\r\na\n\r\n" 
  sleep 1
  printf "2\r\nb\n\r\n" 
  sleep 1
  printf "2\r\nc\n\r\n0\r\n\r\n" 
  sleep 1
  echo excess
else
  echo "Status: 200" 
  echo
  sleep 1
  echo a
  sleep 1
  echo b
  sleep 1
  echo c
fi

Actions
Copy link
 #7

Updated by gstrauss almost 4 years ago

  • Status changed from Patch Pending to Fixed
Actions
Copy link
 #8

Updated by flynn almost 4 years ago

I tested the patches and can confirm, that it works now with standard settings (server.feature-flags += ("proxy.force-http10" => "disable")).

Actions
Copy link

Also available in: Atom