Lighttpd

I redacted some info because I do not want to talk about the internal purpose this is for.

var.log_root    = "/ldisk/thehost/lighttpd-log" 
var.server_root = "/ldisk/thehost/thename" 
var.state_dir   = "/var/run" 
var.home_dir    = "/var/spool/lighttpd" 
server.port = 59
server.use-ipv6 = "disable" 
server.username  = "www" 
server.groupname = "www" 
server.core-files = "disable" 
server.document-root = "/ldisk/thehost/thename" 
server.tag = "lighttpd-nber-thehost" 
server.pid-file = state_dir + "/lighttpd.pid" 
server.errorlog             = log_root + "/error.log" 
include "/usr/local/etc/lighttpd/conf.d/access_log.conf" 
debug.log-request-handling        = "disable" 
debug.log-request-header-on-error = "enable" 
debug.log-timeouts                 = "enable" 
server.event-handler = "poll" 
server.network-backend = "sendfile" 
server.listen-backlog = 2048
server.max-fds = 2048
server.stat-cache-engine = "disable" 
server.max-connections = 1024
server.max-keep-alive-idle = 2
server.max-keep-alive-requests = 2
server.max-request-size = 64
server.max-read-idle = 20
server.max-write-idle = 60
index-file.names += (
  "index.xhtml", "index.html", "index.htm", "default.htm", "index.php" 
)
  server.range-requests = "disable" 
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi", ".scgi" )
include "/usr/local/etc/lighttpd/conf.d/mime.conf" 
server.modules += ( "mod_access", "mod_auth", "mod_status" )
include "/ldisk/thehost/etc/access.conf" 
status.status-url          = "/server-status" 
status.config-url          = "/server-config" 
status.statistics-url      = "/server-statistics" 
status.enable-sort         = "enable" 

auth.backend               = "htpasswd" 
auth.backend.htpasswd.userfile = "/ldisk/thehost/etc/htpasswd" 
auth.require = ( "" =>
                   (
                     "method"  => "basic",
                     "realm"   => "redacted",
                     "require" => "user=nber" 
                   )
                 )
server.follow-symlink = "disable" 
ssl.engine = "enable" 
ssl.pemfile = "/thepath/cert/nber-server-2017.pem" 
ssl.use-sslv2 = "disable" 
ssl.use-sslv3 = "disable" 
ssl.cipher-list = "EECDH+AESGCM:EDH+AESGCM:AES128+EECDH:AES128+EDH" 
ssl.honor-cipher-order = "enable" 

When the issue occurs, have you queried /server-status? Or do you query and save /server-status at any interval to check that lighttpd is processing requests and closing connections?

As you said, this may not be lighttpd, but here are some settings to reduce lighttpd resource usage that you may want to try (after you look in the documentation to see what each one does):

server.bsd-accept-filter = "disable" 
server.network-backend = "writev" 
ssl.read-ahead = "disable" 
server.max-keep-alive-requests = 0

In versions before lighttpd 1.4.46, the httpready filter was enabled by default in FreeBSD. In lighttpd 1.4.46, it will be disabled by default. You should first try server.bsd-accept-filter = "disable" to see if that fixes your issue. See also https://github.com/lighttpd/lighttpd1.4/pull/65

For the log, you said you already tried server.event-handler = "poll". For more details, see 12440e89 as this will be fixed in lighttpd 1.4.46

Thanks for those helpful suggestions. I have made the config changes you recommend and we will see what happens.

After making those recommended config changes we have not seen the problem for 4 days (vs several times per day before). So I am pretty confident this is solved. Thank you for your help.

Thanks for the update. Do you know which (one or more) of the following made the difference?

server.event-handler = "poll" 
server.bsd-accept-filter = "disable" 
server.network-backend = "writev" 
ssl.read-ahead = "disable" 
server.max-keep-alive-requests = 0

Until lighttpd 1.4.46, you should use server.event-handler = "poll" instead of "kqueue", and you probably want server.bsd-accept-filter = "disable" unless you know what the enabled behavior is and really want it. Setting server.network-backend = "writev" is fine, but "freebsd-sendfile" is preferred on FreeBSD, but only if it works for you. ssl.read-ahead = "disable" is intended for low-memory systems, and you should prefer to use keepalive rather than disabling keepalive, so avoid server.max-keep-alive-requests = 0 unless that is what you really intend.