Lighttpd

According to the RFC, headers can (and should) use list syntax and be combined into a single line.
See https://tools.ietf.org/html/rfc7230#section-3.2 Header Fields
The only exception noted by the RFC is the historical (late 1990s) Set-Cookie header.

A quick look at https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag suggests that X-Robots-Tag is also non-compliant and subject to corruption by proxies which, following RFC recommendations, might combine them into a single line.

<sigh> RFCs exist for many reasons, including the attempt to define conformance and avoid messes like this.

gstrauss wrote:

A quick look […] suggests that X-Robots-Tag is also non-compliant

Since the RFC states that "A sender MUST NOT generate multiple header fields" and "A recipient MAY combine multiple header fields", it seems that the lighttpd implementation is indeed RFC-compliant. I don't know how X-Robots-Tag will behave when concatinated, but I understand that it would be incorrect to ask lighttpd to change its implementation.

Thank you for your explaination, gstrauss!

I don't seem to be able to close this ticket myself, so if someone else could be so kind?

A partial workaround is to set the header based on the User-Agent, adjusting the user-agent string as appropriate since I made up the ones below without verifying them.

$HTTP["user-agent"] =~ "googlebot" {
    setenv.add-response-header += (
            "X-Robots-Tag" => "googlebot: noindex, nofollow",
    )
else $HTTP["user-agent"] =~ "mediapartners-google" {
    setenv.add-response-header += (
            "X-Robots-Tag" => "mediapartners-google: noindex, nofollow",
    )
}
else $HTTP["user-agent"] =~ "facebook" {
    setenv.add-response-header += (
            "X-Robots-Tag" => "facebookexternalhit: noindex, nofollow",
    )
}

That won't work for all cases since 'unavailable_after: ...' looks like it is expected to be in its own header. Then again, implementation details will change depending on browser and crawler.

Beware of the (very unfortunate) limitations in lighttpd 1.4.x config parsing if you use the workaround above and have multiple setenv.add-response-header directives.

See https://redmine.lighttpd.net/boards/2/topics/6541 for further explanation.