Host not available

[Soheil Pourbafrani <so...@gmail.com>]

I have 3 node Cassandra 3.11 cluster, one node 4GB memory, and others 2GB.
All of them have 4 CPU core. I want to insert data into the table and read
it to visualize at the san\me time. When I insert data using Flink
Cassandra Connector, in rate 200 inserts per sec, the reader application
can't connect to the hosts, but when the insertion process is not running,
the reader application can connect and fetch data.

All Cassandra,yaml properties are defaults.
Two nodes are seed nodes and keyspace replication factor is 2.

How can I optimize the cluster and solve the problem?

In addition, I have the following log:

 INFO  [epollEventLoopGroup-2-3] 2018-04-30 12:05:11,225 Message.java:623 -
Unexpected exception during request; channel = [id: 0xb84fd5cc, L:/
192.168.1.218:9042 - R:/1$
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed:
Connection reset by peer

Re: Host not available

[Kyrylo Lebediev <Ky...@epam.com>]

First of all you need to identify what's the bottleneck in your case.

First things to check:

1) jvm - heap is too small for such workloads

Enable GC logging in /etc/cassandra/cassandra-env.sh, then analyze its output during workload. In case you observe messages about long pauses  or Full GC's, most probably it's the root-cause.

2) hardware - monitor cpu, disks when writes are active.


there is a really good command 'nodetool tpstats' which helps identify bottleneck / understand what's going on.


Two articles that may help (credits to Chris and Jon):

https://blog.pythian.com/guide-to-cassandra-thread-pools/

http://thelastpickle.com/blog/2018/04/11/gc-tuning.html


Regards,

Kyrill

________________________________
From: Soheil Pourbafrani <so...@gmail.com>
Sent: Monday, April 30, 2018 10:40:10 AM
To: user@cassandra.apache.org
Subject: Host not available

I have 3 node Cassandra 3.11 cluster, one node 4GB memory, and others 2GB. All of them have 4 CPU core. I want to insert data into the table and read it to visualize at the san\me time. When I insert data using Flink Cassandra Connector, in rate 200 inserts per sec, the reader application can't connect to the hosts, but when the insertion process is not running, the reader application can connect and fetch data.

All Cassandra,yaml properties are defaults.
Two nodes are seed nodes and keyspace replication factor is 2.

How can I optimize the cluster and solve the problem?

In addition, I have the following log:

 INFO  [epollEventLoopGroup-2-3] 2018-04-30 12:05:11,225 Message.java:623 - Unexpected exception during request; channel = [id: 0xb84fd5cc, L:/192.168.1.218:9042<http://192.168.1.218:9042> - R:/1$
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer