Project

General

Profile

Accumulation of "Recv-Q" when I check with netstat

Added by gotham about 3 years ago

I'm running lighttpd on embedded Linux devices (some are 32-Bit ARM, and some are 64-Bit ARM). I'm trying to configure lighttpd so that it only allows one TCP connection at a time. If there is an attempt to make a second connection before the first socket has been closed, I want the second attempt to simply get "TCP RST".

I've tried putting the following in my config file:

server.max-connections         = 1
server.listen-backlog = 0
server.max-keep-alive-requests = 0
server.max-read-idle = 3
server.max-write-idle = 3

however I'm not getting the intended behaviour. The second concurrent TCP connection succeeds in connecting, however I can see in the log:

sockets disabled, connection limit reached

And then if I run "netstat -ap", I can see that the figure under "tcp Recv-Q" is increasing by 1 every time I try to make another connection. I think this means that there's a backlog of bytes which haven't been processed yet by lighttpd.

Can someone tell me how to get lighttpd to refuse the second and third TCP connection with "TCP RST" and to not accumulate bytes in "tcp Recv-Q"?


Replies (5)

RE: Accumulation of "Recv-Q" when I check with netstat - Added by gstrauss about 3 years ago

man 2 listen

The behavior of the backlog argument on TCP sockets changed with Linux 2.2. Now it specifies the queue length for completely established sockets waiting to be accepted, instead of the number of incomplete connection requests. The maximum length of the queue for incomplete sockets can be set using /proc/sys/net/ipv4/tcp_max_syn_backlog. When syncookies are enabled there is no logical maximum length and this setting is ignored. See tcp(7) for more information.

If the backlog argument is greater than the value in /proc/sys/net/core/somaxconn, then it is silently truncated to that value. Since Linux 5.4, the default in this file is 4096; in earlier kernels, the default value is 128. In kernels before 2.4.25, this limit was a hard coded value, SOMAXCONN, with the value 128.

.

How/Why have you come to the conclusion that what you have described is the solution to your problem? What is the problem you are trying to solve?

RE: Accumulation of "Recv-Q" when I check with netstat - Added by gotham about 3 years ago

Lighttpd is running on my embedded device, and it is taking data input from the network and writing it to a file. A separate process is observing that file and reacting to its contents.

There should only ever be one node on the network accessing my embedded device at one time. The first node that attempts to connect to my embedded device should be able to complete a TCP handshake. When the 2nd node tries to connect, it should get "TCP RST". Every subsequent node that tries to connect should receive "TCP RST" until the first node's socket has been closed, at which point a new node can connect.

I thought the way to go about this was simply to set "server.max-connections" to 1, however the problem I'm seeing is that 2nd node isn't getting "TCP RST", the 2nd node is actually able to do a TCP handshake, and then on my embedded device I'm seeing an accumulation of "tcp Recv-Q".

RE: Accumulation of "Recv-Q" when I check with netstat - Added by gstrauss about 3 years ago

You did not carefully read what I wrote, pointing you to the man page with kernel implications.

You have repeated your first post instead of trying to answer my question. Do you know the difference between "how", "why", and "what"? You repeated "what" you are trying to implement, not even "what" problem you are trying to solve.

Why is server.listen-backlog = 1 not an option?

If the lighttpd server is running a script, that script can take a lock (e.g. flock --nb) to ensure that only one script is processing data at a time. Every other connection that lighttpd accepts can run the script, which can fail on trylock, and return 503 Service Unavailable. This can be done with a shell script and does not require tuning TCP settings and kernel parameters.

RE: Accumulation of "Recv-Q" when I check with netstat - Added by gotham about 3 years ago

gstrauss wrote:

You did not carefully read what I wrote, pointing you to the man page with kernel implications.

I read your excerpts from the man pages quite closely and I contemplated whether those operating-system-wide settings could be applied to a single process. Applying them to the entire operating system isn't an option for me.

Do you know the difference between "how", "why", and "what"? You repeated "what" you are trying to implement,
not even "what" problem you are trying to solve.

My mother spoke English to me as a baby, I think I was 3 when I got the hang of question words. Just as an aside, a friend of mine has a 3yr old daughter and he has her riding a bike without stabilisers already -- now that's impressive.

Why is server.listen-backlog = 1 not an option?

I've tried setting this to 1, and also 0, but I'm still not getting the behaviour I want. I want the second TCP connection attempt to get TCP RST.

If the lighttpd server is running a script, that script can take a lock (e.g. flock or fcntl F_LOCK) to ensure that only one script
is processing data at a time. Every other connection that lighttpd accepts can run the script, which can fail on trylock, and
return 503 Service Unavailable. This can be done with a shell script and does not require tuning TCP settings and kernel parameters.

It's not as simple as I initially described. I'm dealing with a lighttpd codebase that has been patched to take data directly from a TCP socket and write it into a FIFO pipe in the Linux file system.

I prefer to change a setting than to write a script. I thought that setting "server.max-connection = 1" would mean that the 2nd and 3rd connections would get TCP RST, but instead they're completing a TCP handshake.

Do I have to write a patch for lighttp that does something like the following?

static std::mutex g_mtx_single_tcp_connection;

void Func_That_Does_TCP_Handhakes(void)
{
    if ( false == g_mtx_single_connection.trylock() )
    {
        /* Put code in here to send TCP RST */
        return;
    }

    /* The original code for this function goes here */
}

void Func_That_Closes_TCP_Socket(void)
{
    /* The original code for this function goes here */

    g_mtx_single_connection.unlock();
}

RE: Accumulation of "Recv-Q" when I check with netstat - Added by gstrauss about 3 years ago

server.max-connections = 1 limits lighttpd to accepting only one connection at a time.

It's not as simple as I initially described. I'm dealing with a lighttpd codebase that has been patched to take data directly from a TCP socket and write it into a FIFO pipe in the Linux file system.

That sounds like something mod_sockproxy can do into a socket.

Why use lighttpd at all? Look at netcat (or nc), or at xinetd

Also, if you have patched lighttpd as you have said, then I suggest re-reading my quote from man 2 listen above. You seem to misunderstand listen backlog and how it is handled in the kernel.

If you want to use raw sockets and to implement userspace TCP, then yes, you'll have to patch lighttpd yourself. I still wonder why lighttpd is being used for your special purpose. Or why you are using TCP. Or why additional connections "must" immediately receive TCP RST and you do not know how to do this after TCP connection accept(). Hint man 7 socket and SO_LINGER.

    (1-5/5)