netty connection reset by peer errors in logs

[Gil Ganz <gi...@gmail.com>]

Hey
We have an issue in few of our 4.0.4 clusters, these are on-prem, multiple
datacenters around the world clusters, and our logs have many errors like
this :


ERROR [Messaging-EventLoop-3-26] 2022-09-01 05:57:28,142
InboundMessageHandler.java:300 -
/10.11.0.1:7000->/10.106.0.2:7000-URGENT_MESSAGES-96981741
unexpected exception caught while processing inbound messages; terminating
connection
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed:
Connection reset by peer

There are no other errors accompanying this, not on this server or the
second one mentioned in the error, and since servers are not that loaded or
have significant gc pauses, it seems like this is due to network hiccups
between datacenters.
Is there a way to suppress these messages?

Gil

Re: netty connection reset by peer errors in logs

[Gil Ganz <gi...@gmail.com>]

Reason I would like to suppress it is I think this is due to network
disconnects we know are happening, and looks like it's not going to change.
Since it doesn't happen that often, and not causing a real issue , I would
like to have cleaner logs if possible.

On Thu, Sep 1, 2022 at 9:55 AM Erick Ramirez <er...@apache.org>
wrote:

> That error message indicates that 2 nodes are unable to communicate with
> each other over the internode (gossip) port. It makes no sense to suppress
> it since it's an indication that there's a problem that you need to
> address. Cheers!
>

Re: netty connection reset by peer errors in logs

[Erick Ramirez <er...@apache.org>]

That error message indicates that 2 nodes are unable to communicate with
each other over the internode (gossip) port. It makes no sense to suppress
it since it's an indication that there's a problem that you need to
address. Cheers!