Project

General

Profile

Actions

Bug #2299

closed

Fragmentation is broken when using SSL

Added by js about 13 years ago. Updated almost 13 years ago.

Status:
Invalid
Priority:
Normal
Category:
core
Target version:
-
ASK QUESTIONS IN Forums:

Description

Hello,

when trying to use SSL and sending one line per packet, lighttpd does not send a reply.

Sending
"GET / HTTP/1.0\r\n"
"Host: foo.org\r\n"
"User-Agent: foo\r\n"
"\r\n"
as 4 packets, there is no answer from lighttpd.

However, doing this works:
"GET / HTTP/1.0\r\n"
"Host: foo.org\r\n"
"User-Agent: foo\r\n\r\n"
(3 packets).

Actions #1

Updated by js about 13 years ago

Oh, forgot to mention: When using HTTP, it works. The problem only happens with HTTPS. I send exactly the same data using HTTP and HTTPS, but only HTTPS does not work.

Actions #2

Updated by darix about 13 years ago

sounds similar to #2105

Actions #3

Updated by js about 13 years ago

Not sure how similar it actually is. As said, it works when using HTTP, the problem only exists with HTTPS.

Actions #4

Updated by spaam about 13 years ago

what version is this?

Actions #5

Updated by js about 13 years ago

Oh, sorry: 1.4.26 from pkgsrc.

Actions #6

Updated by darix about 13 years ago

  • Status changed from New to Need Feedback

can you try with 1.4.28?

Actions #7

Updated by js about 13 years ago

Unfortunately not as .26 is the newest version in pkgsrc.

Actions #8

Updated by js about 13 years ago

I updated lighttpd in pkgsrc to .28 now and sent in a patch. Still the same.

Actions #9

Updated by Olaf-van-der-Spek about 13 years ago

  • Target version set to 1.4.29

What tool do you use to send the request?

Actions #10

Updated by js about 13 years ago

I wrote something myself (which sent each parameter in a separate packet, this is how I noticed it), but you can easily reproduce it this way:
Use openssl s_client -connect webkeks.org:443
Wait until it connected, type:
GET / HTTP/1.1
Host: webkeks.org
User-Agent: Foo
Now press enter twice. Nothing will happen.

Actions #11

Updated by Olaf-van-der-Spek about 13 years ago

You're right, I can confirm it on 1.4.28.

Actions #12

Updated by andreas_stoe about 13 years ago

As far as I see it, this is not really a bug. The HTTP 1.1 protocol requires the "connection" field to be set in every message:

HTTP/1.1 applications that do not support persistent connections MUST include the "close" connection option in every message.

If you add this to your request it will work in 1.4.28.

However, this bug existed in 1.4.26, but has been fixed. See http://redmine.lighttpd.net/issues/2197

Actions #13

Updated by stbuehler about 13 years ago

Testing with openssl s_client is dangerous, as your terminal probably doesn't send \r\n newlines. try it like this:

(printf 'GET / HTTP/1.1\r\nHost: webkeks.org\r\n\r\n'; cat) | openssl s_client -connect webkeks.org:44
Actions #14

Updated by js about 13 years ago

Sorry, but you both are wrong:

@andreas_stoebe: As you can see above, the request is HTTP/1.0, so this is not the issue. And even then, the problem is fragmentation and not that.

@stbuehler: This is exactly what you DON'T want to demonstrate the bug, as this does not do fragmentation. Actually typing it creates fragmentation because telnet sends as soon as it can. CRLF can be enabled in telnet, read the manpage. It's the default in mine.

Actions #15

Updated by stbuehler almost 13 years ago

js wrote:

Sorry, but you both are wrong:
[...]
@stbuehler: This is exactly what you DON'T want to demonstrate the bug, as this does not do fragmentation. Actually typing it creates fragmentation because telnet sends as soon as it can. CRLF can be enabled in telnet, read the manpage. It's the default in mine.

  1. That wasn't my point, my point was how to create valid requests
  2. I am pretty sure your telnet (and openssl s_client) is line buffered (and does NOT send characters as soon as it can)
  3. I doubt your telnet CRLF setting has any impact on openssl s_client

So here the complete command line to test how a server responds to line-splitted requests:

(printf 'GET / HTTP/1.1\r\n'; sleep 1; printf 'Host: webkeks.org\r\n'; sleep 1; printf '\r\n'; cat) | openssl s_client -connect webkeks.org:443

(this one works fine btw, so i don't see any bug here)
And before anyone asks what the cat is for: without the cat, the stream gets closed, which means "cancel request" to a web server (if the web server happens to check for it).

Or pipe the request data through a shell script like this; choose a high enough sleep value so that the ACK for one character comes back before you send the next one:

IFS="" 
while read -N 1 c; do
        printf "%s" "$c" 
        sleep 0.1
done

Actions #16

Updated by js almost 13 years ago

This will still send one packet as pipes are buffered.

As I already said above, I did not trigger the problem with s_client, but also with ObjC code I wrote. The problem is fragmentation, try it yourself using libssl. Do multiple SSL_send calls and you'll see what I mean.

Actions #17

Updated by js almost 13 years ago

Oh, and your example has another bug: As I said before, if only specifying Host:, it works. The problem happens as soon as you add one more field.

Guess next time I will just attach a .c file that generates the packets that cause the bug…

Actions #18

Updated by stbuehler almost 13 years ago

  • Status changed from Need Feedback to Invalid
  • Target version deleted (1.4.29)
# (printf 'GET / HTTP/1.1\r\n'; sleep 1; printf 'Host: webkeks.org\r\n'; sleep 1; printf 'User-Agent: foo\r\n'; sleep 1; printf '\r\n'; sleep 5;) | ltrace -tt -s 256 -e read,write,SSL_read,SSL_write openssl s_client -connect webkeks.org:443 > /dev/null
15:17:43.308231 SSL_write(0xc1e0a0, 0xc13d60, 0, 8, 0depth=2 /C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Certification Authority
verify error:num=19:self signed certificate in certificate chain
verify return:0
)                                                                                                                 = 0
15:17:43.588224 read(0, "GET / HTTP/1.1\r\n", 8192)                                                                                                                    = 16
15:17:43.588590 SSL_write(0xc1e0a0, 0xc13d60, 16, 8, 0)                                                                                                                = 16
15:17:44.229070 read(0, "Host: webkeks.org\r\n", 8192)                                                                                                                 = 19
15:17:44.229324 SSL_write(0xc1e0a0, 0xc13d60, 19, 8, 0)                                                                                                                = 19
15:17:45.231389 read(0, "User-Agent: foo\r\n", 8192)                                                                                                                   = 17
15:17:45.231686 SSL_write(0xc1e0a0, 0xc13d60, 17, 8, 0)                                                                                                                = 17
15:17:46.233613 read(0, "\r\n", 8192)                                                                                                                                  = 2
15:17:46.233834 SSL_write(0xc1e0a0, 0xc13d60, 2, 8, 0)                                                                                                                 = 2
15:17:46.296820 SSL_read(0xc1e0a0, 0xc15d70, 1024, 8, 0)                                                                                                               = 209
15:17:46.297504 write(1, "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nAccept-Ranges: bytes\r\nETag: "3100938754"\r\nLast-Modified: Sun, 15 Mar 2009 18:50:51 GMT\r\nContent-Length: 1757\r\nDate: Fri, 01 Apr 2011 13:17:44 GMT\r\nServer: lighttpd\r\n\r\n", 209) = 209
15:17:46.308163 SSL_read(0xc1e0a0, 0xc15d70, 1024, 8, 0)                                                                                                               = 1024
15:17:46.367021 write(1, "<?xml version="1.0" encoding="UTF-8"?>\n<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"\n "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">\n<html xmlns="http://www.w3.org/1999/xhtml">\n<head>\n<title>webkeks.org</title>\n<style type="text/css">\n<!--\nbody {\n\tbac"..., 1024) = 1024
15:17:46.367514 SSL_read(0xc1e0a0, 0xc15d70, 1024, 0, 0)                                                                                                               = 733
15:17:46.367648 write(1, "-size: 48px;\n\tfont-weight: bold;\n}\n//-->\n</style>\n</head>\n<body>\n<!--\n<div id='menubar'>\n<span class='menu'><a href='/blog'>Blog</a>\n</span><span class='menu'><a href='/jabber'>Jabber</a>\n</span><span class='menu'><a href='/jwchat'>JWChat</a>\n</span><span "..., 733) = 733
15:17:51.236407 read(0, "", 8192)                                                                                                                                      = 0
DONE
15:17:51.238166 +++ exited (status 0) +++

"pipes are buffered" - doesn't sound like you have any idea what this is about.

Cannot reproduce, closed.

Actions #19

Updated by js almost 13 years ago

  • Target version set to 1.4.29

Ok, indeed, the fragmentation bug seems to be fixed. When I upgraded, I just tried with OpenSSL, which seems to actually only send \n, unlike telnet. So, the problem was that I did not correctly verify that the bug from .26 still exists. I just wrote a C program which tests it and indeed it works since .29.

About the pipes are buffered stuff: Use for example find | grep and you will see that not each time find outputs a line that would match it is displayed, this is due to pipe buffering. It seems that in this case it was not buffered, though.

Actions #20

Updated by icy almost 13 years ago

Pipes might get buffered but they don't wait 1 second :)

Actions #21

Updated by stbuehler almost 13 years ago

  • Target version deleted (1.4.29)

.29 is not released. perhaps your bug was related to #2197. target version without a commit that fixed it doesn't make sense.

and that is why i think you have no idea what "pipes are buffered" means. It means that a sender can finish write to it without a client reading from it at the same time (unless the buffer is full). but if there is data in the pipe and a client wants to read it will always get the data; it just has to wait to get some time on the cpu of course, which is why you sometimes will get more than one write() block in a read() when the cpu was too busy.

Actions #22

Updated by js almost 13 years ago

Sorry, I meant .28 of course.

Actions

Also available in: Atom