Lighttpd

Could you provide your config file? Which plugins you used?

modules.conf is missing. I try to reproduce your problem with only fcgi php. I use the urls you provided, with http_load run nearly 3 hours, 100 reqs/sec, everything workss fine.

Reproduce is a hard work. But I believe if I reproduce it once, I can reproduce it again and again, and finally solve the problem.

In fact I'm a poor student, the author of http://gitorious.org/currf2. I have bet my future on that project. I have to use it to solve a real bug to show its power to my professor. This is why I interest in your problem. If you use x86 + Linux, you are welcome to try it in your environment.

I noticed the same messages in lighttpd's log. We handle about 12 millions requests per day. As for us this message appears slightly more often. At least one time per day. But I think this is not lighttpd bug but client side bug (browser, proxy or whatever else) or network bug (TCP error protection isn't the best it the world). Really, why not? Client's browser can have a bug and in very rare situations duplicate request headers. pi3orama, you could never reproduce this message just because you dont use this browser :) Think we can summarize User-Agent headers from this messages and see how many browsers (and browser versions) sends us this garbled headers.

For my side I use this command. Not so accurate because it assumes that User-Agent header will appear in first 6 lines of the request and that it won't appear multiple times in this lines.

grep -h -A 5 'invalid character in key' /var/log/lighttpd/error.log* | grep 'User-Agent'

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.1.15) Gecko/20101026 MRA 5.7 (build 03773) Firefox/3.5.15 sputnik 2.3.0.102
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.1.15) Gecko/20101026 MRA 5.7 (build 03686) Firefox/3.5.15 sputnik 2.3.0.86
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:2.0b8pre) Gecko/20101130 Firefox/4.0b8pre
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:2.0b8pre) Gecko/20101130 Firefox/4.0b8pre
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; ru) Opera 10.63
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; ru) Opera 10.63
User-Agent: Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.6.30 Version/10.63

All 4 INDEPENDENT browser vendors have managed to trigger this symptom in lighttpd and you're saying that it's the browser's fault? Sorry, I don't buy it.

However, if it occurs daily, then isolating the client is only a matter of capturing it on a packet dump. If confirmed, post it and we can close this bug. If not, it could shed some light on what triggers this bug.

Using 1.4.28 we're also seeing this occasionally. Random characters in the headers will be replaced with something else. For instance:

Cache-Contr/l: no-cache

It seems it's usually '/', '(' or ')', but we may be getting a lot more corruption that either doesn't happen in the header, or the corrupted character is no longer an invalid character so it just becomes an unknown header. It doesn't happen often, we handle about 15 million requests per day and I see only about 8 of the errors per day. But I'm worried that these are only the ones that are logged (random character is both in header and invalid) and that we're getting more frequent corruption elsewhere.

It's very possible it's a frequent network issue but that I'm only seeing those that have multiple errors and cause a TCP checksum collision, but it's not browser related as I'm seeing it from a variety of browsers. I'm not seeing any TCP errors on the interface, so it's not between me and the upstream router.