Lighttpd

Are you really sure that you are using the same config? maybe you forgot to remove POST from "^(PUT|HEAD|PATCH|DELETE)$" the other server?

you got some duplicates of some config values. for example server.tag.

Think I simply copy pasted into the gist twice. I've updated the gist to reflect the one I've just uploaded to both servers to make sure.

I've double checked, and here's the permissions and cat output from the broken server on the configs if it helps: https://gist.github.com/4275115

Definitely removed POST from that check, even removing that whole bit doesn't help.

Checked the enabled modules?

Run /usr/share/lighttpd/include-conf-enabled.pl (and paste the output).

/usr/share/lighttpd/create-mime.assign.pl should only set mimetypes, which should be unrelated.

Any proxy/load-balancer sitting before lighttpd?

Results from running that script:

[/etc/lighttpd]
[13:43:56] root $ /usr/share/lighttpd/include-conf-enabled.pl
include "conf-enabled/10-fastcgi.conf" 
include "conf-enabled/10-isitup.conf" 
include "conf-enabled/12-cloudflare.conf" 
include "conf-enabled/15-fastcgi-php.conf" 
[/etc/lighttpd]
[13:43:59] root $

I am running behind CloudFlare, however the sub-domain's should be bypassing any of their proxy entirely.

The 12-cloudflare.conf is as follows:

[/etc/lighttpd]
[13:44:03] root $ cat conf-available/12-cloudflare.conf
server.modules += ( "mod_extforward" )

extforward.forwarder = (
        "204.93.240.0/24" => "trust",
        "204.93.177.0/24" => "trust",
        "199.27.128.0/21" => "trust",
        "173.245.48.0/20" => "trust",
        "103.22.200.0/22" => "trust",
        "141.101.64.0/18" => "trust",
        "108.162.192.0/18" => "trust",
        "190.93.240.0/20" => "trust",
        "188.114.96.0/20" => "trust" 
)

extforward.headers = ("CF-Connecting-IP")

accesslog.format = "%t %h %{CF-Connecting-IP}i %>s \"%r\" \"%{Referer}i\" \"%{User-Agent}i\"" 
[/etc/lighttpd]
[13:47:39] root $

Well. Lets go the standard way:

Set debug.log-request-handling = "enable" and check error.log,
also see http://redmine.lighttpd.net/projects/lighttpd/wiki/DebugVariables

Looks promising:

https://gist.github.com/4275499

Not sure if those (PUT|POST|HEAD|PATCH|DELETE) are normal or cached from when I did have POST in that check... Is force-reload not actually reloading the configs in conf-available?

There seems to be something around line 200 kinda out of the blue also.

Edit: Output so it's clear I'm not being dumb.

[/etc/lighttpd]
[14:24:21] root $ cat conf-available/10-isitup.conf
$HTTP["request-method"] =~ "^(PUT|HEAD|PATCH|DELETE)$" {
        url.access-deny = ( "" )
}

$HTTP["host"] =~ "^www\.(.*)" {
        url.redirect = (
                "^/(.*)" => "http://%1/$1" 
        )
}

url.rewrite-once = (
"^/$"             => "/index.php",
"^/robots\.txt$"  => "/robots.txt",
"^/static/(.*)$"  => "/static/$1",
"^/offline$"      => "/offline.php",
"^/error$"        => "/error.php",
"^/save/(.*)$"    => "/index.php?d=$1&save",
"^/d/(.*)$"       => "/index.php?d=$1",
"^/clear$"        => "/index.php?clear",
"^/(.*)\.json$"   => "/api.php?domain=$1&output=json",
"^/(.*)\.json\?(.*)$" => "/api.php?domain=$1&output=json&$2",
"^/(.*)\.txt$"    => "/api.php?domain=$1&output=txt",
"^/(.*)\.(php|html|css|js|xml|png|jpg|gif|ico)(\?(.*))?$" => "$0",
"^/(.*)$"         => "/check.php?domain=$1" )
[/etc/lighttpd]
[14:24:28] root $ /etc/init.d/lighttpd
Usage: /etc/init.d/lighttpd {start|stop|restart|reload|force-reload|status}
[/etc/lighttpd]
[14:25:04] root $ /etc/init.d/lighttpd stop && /etc/init.d/lighttpd start
Stopping web server: lighttpd.
Starting web server: lighttpd.
[/etc/lighttpd]
[14:25:17] root $

Wild guess: conf-enabled/10-isitup.conf isn't a symlink to conf-available/10-isitup.conf anymore.

lighttpd -p -f /etc/lighttpd/lighttpd.conf

shows the complete config how lighty sees it.

Ha!

That is really odd as I'm fairly sure I used lighttpd-enable mod for them.

Indeed it seemed I had full files in conf-enabled rather than syslinks.

Thank you very much for you help!