RE: Cassandra scale-out with no traffic on newly joined nodes

["ZAIDI, ASAD" <az...@att.com>]

Can you share please what replica placement strategy ( in keyspace definition) and   partitioner is used across nodes ?
~Asad



From: Sandeep Nethi <ne...@gmail.com>
Sent: Tuesday, September 8, 2020 1:12 AM
To: user@cassandra.apache.org
Subject: Re: Cassandra scale-out with no traffic on newly joined nodes

That would be my last option to add a new host as contant point but as per my understanding cassandra should auto discover newly joined nodes and server or load balance connections automatically but it's not happening. So, i'm trying to understand what could be the root cause here.

Another problem that I have noticed is, the number of native client connections on newly joined nodes vs old nodes is 1:7 and client connections are not balanced across nodes (old nodes are overloaded compared to newly joined nodes). Since i have a 6 node cluster with 3 racks, is it mandatory to have rack awareness in driver load balancer?

On Tue, Sep 8, 2020 at 6:02 PM manish khandelwal <ma...@gmail.com>> wrote:
Can you add new host as contact points and see if traffic lands on them or not?
Also you can verify new nodes are added in system.peers of host name which you are giving as contact points

On Tue, Sep 8, 2020 at 11:27 AM Sandeep Nethi <ne...@gmail.com>> wrote:
Yes, all nodes are UN and no issues identified. Infact i could see some client connections on new nodes with telnet but not seeing any traffic.

Cassandra version: 3.11.6
Load Balancing policy used is default with no custom policies.

Thanks,

On Tue, Sep 8, 2020 at 5:52 PM Erick Ramirez <er...@datastax.com>> wrote:
That shouldn't be a problem for the control connection.

I would suggest looking into the load-balancing policy configured on the driver. Also, are all the new nodes fully up and fully joined the cluster?

Re: Cassandra scale-out with no traffic on newly joined nodes

[Sandeep Nethi <ne...@gmail.com>]

NetworkTopology strategy and murmer3 partitioner is being used.

Just an update, nodes were able to receive queries from coordinator nodes
but only old nodes are acting as coordinators.

As I explained before number of native connections on new nodes is very
less compared to old nodes.

How can i balance these connections without restarting client application,
any idea?


Thanks,
Sandeep

On Sat, 12 Sep 2020 at 5:02 AM, ZAIDI, ASAD <az...@att.com> wrote:

>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Can you share please what replica placement strategy ( in keyspace
> definition) and   partitioner is used across nodes ?
>
>
> ~Asad
>
>
>
>
>
>
>
>
>
>
>
> *From:* Sandeep Nethi <ne...@gmail.com>
>
>
> *Sent:* Tuesday, September 8, 2020 1:12 AM
>
>
> *To:* user@cassandra.apache.org
>
>
> *Subject:* Re: Cassandra scale-out with no traffic on newly joined nodes
>
>
>
>
>
>
>
>
>
> That would be my last option to add a new host as contant point but as per
> my understanding cassandra should auto discover newly joined nodes and
> server or
>
> load balance connections automatically but it's not happening. So, i'm
> trying to understand what could be the root cause here.
>
>
>
>
>
>
>
>
>
>
>
>
>
> Another problem that I have noticed is, the number of native client
> connections on newly joined nodes vs old nodes is 1:7 and client
> connections are not balanced
>
> across nodes (old nodes are overloaded compared to newly joined nodes).
> Since i have a 6 node cluster with 3 racks, is it mandatory to have rack
> awareness in driver load balancer?
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Tue, Sep 8, 2020 at 6:02 PM manish khandelwal <
> manishkhandelwal03@gmail.com> wrote:
>
>
>
>
>
>
>
>
> Can you add new host as contact points and see if traffic lands on them or
> not?
>
>
>
>
> Also you can verify new nodes are added in system.peers of host name which
> you are giving as contact points
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Tue, Sep 8, 2020 at 11:27 AM Sandeep Nethi <ne...@gmail.com>
> wrote:
>
>
>
>
>
>
>
>
> Yes, all nodes are UN and no issues identified. Infact i could see some
> client connections on new nodes with telnet but not seeing any traffic.
>
>
>
>
>
>
>
>
>
>
>
> Cassandra version: 3.11.6
>
>
>
>
>
>
> Load Balancing policy used is default with no custom policies.
>
>
>
>
>
>
>
>
>
>
>
>
>
> Thanks,
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Tue, Sep 8, 2020 at 5:52 PM Erick Ramirez <er...@datastax.com>
> wrote:
>
>
>
>
>
>
>
>
>
>
>
>
> That shouldn't be a problem for the control connection.
>
>
>
>
>
>
>
>
>
>
>
>
>
> I would suggest looking into the load-balancing policy configured on the
> driver. Also, are all the new nodes fully up and fully joined the cluster?
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>