io.netty.channel.unix.Errors$NativeIoException: Connection reset by peer

[velocix cephusers <ve...@gmail.com>]

Hi,

We are having a 5 node Cassandra cluster running in version 3.0.13.
Recently we upgrade the Cassandra cpp driver on the application side
from cassandra-cpp-driver-2.6.0-1.el7.centos.x86_64
to cassandra-cpp-driver-2.15.3-1.el7.x86_64. After the upgrade, Cassandra
system.log is continuously filled with the below message.

INFO  [SharedPool-Worker-11] 2021-04-26 07:11:18,445 Message.java:615 -
Unexpected exception during request; channel = [id: 0x08a9bc0f, L:/
10.50.11.123:9042 ! R:/10.50.11.182:44734]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed:
Connection reset by peer
        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown
Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]

Even though these INFO level log prints are causing system.log to be
rotated 4 or 5 times a day, there is no functional impact seen.

What could be the problem here? Let me know if more details are needed.

Regards,
Renoy Paulose

Re: io.netty.channel.unix.Errors$NativeIoException: Connection reset by peer

[Erick Ramirez <er...@datastax.com>]

That message gets logged when the node tries to respond back to the client
but the driver has already given up waiting for the cluster to respond so
the connection is no longer active.

It typically happens when running an expensive query and the coordinator is
still waiting for the replicas to respond but the driver already reached
the client-side timeout. It can also happen when the driver has been
configured with a very low timeout value so the coordinator never gets a
chance to respond back.

Check for the timeouts configured on the driver. I'd also recommend
reviewing the app queries for clues. Cheers!

>

Re: io.netty.channel.unix.Errors$NativeIoException: Connection reset by peer

[sunil pawar <su...@gmail.com>]

Hi Renoy,

The message below shows there are connection hick-ups between Remote and
Local machines.
   Unexpected exception during request; channel = [id: 0x08a9bc0f, L:/
10.50.11.123:9042 ! R:/10.50.11.182:44734]

L stands for local machine address and R stands for Remote machine address.
Please check if there are any connection issues or if any firewall
restrictions between two machines.
As you have upgraded the driver recently, I will suggest checking if any
configuration is missing.

Thanks,
Sunil Pawar


On Mon, Apr 26, 2021 at 12:50 PM velocix cephusers <
velocixcephusers@gmail.com> wrote:

> Hi,
>
> We are having a 5 node Cassandra cluster running in version 3.0.13.
> Recently we upgrade the Cassandra cpp driver on the application side
> from cassandra-cpp-driver-2.6.0-1.el7.centos.x86_64
> to cassandra-cpp-driver-2.15.3-1.el7.x86_64. After the upgrade, Cassandra
> system.log is continuously filled with the below message.
>
> INFO  [SharedPool-Worker-11] 2021-04-26 07:11:18,445 Message.java:615 -
> Unexpected exception during request; channel = [id: 0x08a9bc0f, L:/
> 10.50.11.123:9042 ! R:/10.50.11.182:44734]
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
> failed: Connection reset by peer
>         at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown
> Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
>
> Even though these INFO level log prints are causing system.log to be
> rotated 4 or 5 times a day, there is no functional impact seen.
>
> What could be the problem here? Let me know if more details are needed.
>
> Regards,
> Renoy Paulose
>