You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Pranay akula <pr...@gmail.com> on 2016/09/19 14:53:04 UTC
High load on few nodes in a DC.
when our cluster was under load i am seeing 1 or 2 nodes are on more load
consistently when compared to others in dc i am not seeing any GC pauses or
wide partitions is this can be those nodes are continuously serving as
coordinators ?? how can i find what is the reason for high load on those
two nodes ?? We are using Vnode.
Thanks
Pranay.
Re: High load on few nodes in a DC.
Posted by Pranay akula <pr...@gmail.com>.
Yes we are using token aware but not shuffling replicas.
On Wed, Sep 21, 2016 at 10:04 AM, Romain Hardouin <ro...@yahoo.fr>
wrote:
> Hi,
>
> Do you shuffle the replicas with TokenAwarePolicy?
> TokenAwarePolicy(LoadBalancingPolicy childPolicy, boolean
> shuffleReplicas)
>
> Best,
>
> Romain
> Le Mardi 20 septembre 2016 15h47, Pranay akula <pr...@gmail.com>
> a écrit :
>
>
> I was a able to find the hotspots causing the load,but the size of these
> partitions are in KB and no tombstones and no.of sstables is only 2 what
> else i need to debug to find the reason for high load for some nodes.
> we are also using unlogged batches is that can be the reason ?? how to
> find which node is serving as a coordinator for un logged batches?? we are
> using token-aware policy.
>
> thanks
>
>
>
> On Mon, Sep 19, 2016 at 12:29 PM, Pranay akula <pranay.akula2323@gmail.com
> > wrote:
>
> I was able to see most used partitions but the nodes with less load are
> serving more read and write requests for that particular partitions when
> compared to nodes with high load, how can i find if these nodes are serving
> as co-coordinators for those read and write requests ?? how can i find the
> token range for these particular partitions and which node is the primary
> for these partition ??
>
>
> Thanks
>
> On Mon, Sep 19, 2016 at 11:04 AM, Pranay akula <pranay.akula2323@gmail.com
> > wrote:
>
> Hai Jeff,
>
> Thank, we are using RF 3 and cassandra version 2.1.8.
>
> Thanks
> Pranay.
>
> On Mon, Sep 19, 2016 at 10:55 AM, Jeff Jirsa <je...@crowdstrike.com>
> wrote:
>
> Is your replication_factor 2? Or is it 3? What version are you using?
>
> The most likely answer is some individual partition that’s either being
> written/read more than others, or is somehow impacting the cluster (wide
> rows are a natural candidate).
>
> You don’t mention your version, but most modern versions of Cassandra ship
> with ‘nodetool toppartitions’, which will help you identify frequently
> written/read partitions – perhaps you can use that to identify a hotspot
> due to some external behavior (some partition being read thousands of
> times, over and over could certainly drive up load).
>
> - Jeff
>
> *From: *Pranay akula <pr...@gmail.com>
> *Reply-To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
> *Date: *Monday, September 19, 2016 at 7:53 AM
> *To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
> *Subject: *High load on few nodes in a DC.
>
> when our cluster was under load i am seeing 1 or 2 nodes are on more
> load consistently when compared to others in dc i am not seeing any GC
> pauses or wide partitions is this can be those nodes are continuously
> serving as coordinators ?? how can i find what is the reason for high load
> on those two nodes ?? We are using Vnode.
>
>
> Thanks
> Pranay.
>
>
>
>
>
>
>
Re: High load on few nodes in a DC.
Posted by Romain Hardouin <ro...@yahoo.fr>.
Hi,
Do you shuffle the replicas with TokenAwarePolicy?TokenAwarePolicy(LoadBalancingPolicy childPolicy, boolean shuffleReplicas)
Best,
Romain Le Mardi 20 septembre 2016 15h47, Pranay akula <pr...@gmail.com> a écrit :
I was a able to find the hotspots causing the load,but the size of these partitions are in KB and no tombstones and no.of sstables is only 2 what else i need to debug to find the reason for high load for some nodes. we are also using unlogged batches is that can be the reason ?? how to find which node is serving as a coordinator for un logged batches?? we are using token-aware policy.
thanks
On Mon, Sep 19, 2016 at 12:29 PM, Pranay akula <pr...@gmail.com> wrote:
I was able to see most used partitions but the nodes with less load are serving more read and write requests for that particular partitions when compared to nodes with high load, how can i find if these nodes are serving as co-coordinators for those read and write requests ?? how can i find the token range for these particular partitions and which node is the primary for these partition ??
Thanks
On Mon, Sep 19, 2016 at 11:04 AM, Pranay akula <pr...@gmail.com> wrote:
Hai Jeff,
Thank, we are using RF 3 and cassandra version 2.1.8.
ThanksPranay.
On Mon, Sep 19, 2016 at 10:55 AM, Jeff Jirsa <je...@crowdstrike.com> wrote:
Is your replication_factor 2? Or is it 3? What version are you using? The most likely answer is some individual partition that’s either being written/read more than others, or is somehow impacting the cluster (wide rows are a natural candidate). You don’t mention your version, but most modern versions of Cassandra ship with ‘nodetool toppartitions’, which will help you identify frequently written/read partitions – perhaps you can use that to identify a hotspot due to some external behavior (some partition being read thousands of times, over and over could certainly drive up load). - Jeff From: Pranay akula <pr...@gmail.com>
Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Date: Monday, September 19, 2016 at 7:53 AM
To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Subject: High load on few nodes in a DC. when our cluster was under load i am seeing 1 or 2 nodes are on more load consistently when compared to others in dc i am not seeing any GC pauses or wide partitions is this can be those nodes are continuously serving as coordinators ?? how can i find what is the reason for high load on those two nodes ?? We are using Vnode. ThanksPranay.
Re: High load on few nodes in a DC.
Posted by Pranay akula <pr...@gmail.com>.
I was a able to find the hotspots causing the load,but the size of these
partitions are in KB and no tombstones and no.of sstables is only 2 what
else i need to debug to find the reason for high load for some nodes.
we are also using unlogged batches is that can be the reason ?? how to
find which node is serving as a coordinator for un logged batches?? we are
using token-aware policy.
thanks
On Mon, Sep 19, 2016 at 12:29 PM, Pranay akula <pr...@gmail.com>
wrote:
> I was able to see most used partitions but the nodes with less load are
> serving more read and write requests for that particular partitions when
> compared to nodes with high load, how can i find if these nodes are serving
> as co-coordinators for those read and write requests ?? how can i find the
> token range for these particular partitions and which node is the primary
> for these partition ??
>
>
> Thanks
>
> On Mon, Sep 19, 2016 at 11:04 AM, Pranay akula <pranay.akula2323@gmail.com
> > wrote:
>
>> Hai Jeff,
>>
>> Thank, we are using RF 3 and cassandra version 2.1.8.
>>
>> Thanks
>> Pranay.
>>
>> On Mon, Sep 19, 2016 at 10:55 AM, Jeff Jirsa <je...@crowdstrike.com>
>> wrote:
>>
>>> Is your replication_factor 2? Or is it 3? What version are you using?
>>>
>>>
>>>
>>> The most likely answer is some individual partition that’s either being
>>> written/read more than others, or is somehow impacting the cluster (wide
>>> rows are a natural candidate).
>>>
>>>
>>>
>>> You don’t mention your version, but most modern versions of Cassandra
>>> ship with ‘nodetool toppartitions’, which will help you identify frequently
>>> written/read partitions – perhaps you can use that to identify a hotspot
>>> due to some external behavior (some partition being read thousands of
>>> times, over and over could certainly drive up load).
>>>
>>>
>>>
>>> - Jeff
>>>
>>>
>>>
>>> *From: *Pranay akula <pr...@gmail.com>
>>> *Reply-To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
>>> *Date: *Monday, September 19, 2016 at 7:53 AM
>>> *To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
>>> *Subject: *High load on few nodes in a DC.
>>>
>>>
>>>
>>> when our cluster was under load i am seeing 1 or 2 nodes are on more
>>> load consistently when compared to others in dc i am not seeing any GC
>>> pauses or wide partitions is this can be those nodes are continuously
>>> serving as coordinators ?? how can i find what is the reason for high load
>>> on those two nodes ?? We are using Vnode.
>>>
>>>
>>>
>>>
>>>
>>> Thanks
>>>
>>> Pranay.
>>>
>>
>>
>
Re: High load on few nodes in a DC.
Posted by Pranay akula <pr...@gmail.com>.
I was able to see most used partitions but the nodes with less load are
serving more read and write requests for that particular partitions when
compared to nodes with high load, how can i find if these nodes are serving
as co-coordinators for those read and write requests ?? how can i find the
token range for these particular partitions and which node is the primary
for these partition ??
Thanks
On Mon, Sep 19, 2016 at 11:04 AM, Pranay akula <pr...@gmail.com>
wrote:
> Hai Jeff,
>
> Thank, we are using RF 3 and cassandra version 2.1.8.
>
> Thanks
> Pranay.
>
> On Mon, Sep 19, 2016 at 10:55 AM, Jeff Jirsa <je...@crowdstrike.com>
> wrote:
>
>> Is your replication_factor 2? Or is it 3? What version are you using?
>>
>>
>>
>> The most likely answer is some individual partition that’s either being
>> written/read more than others, or is somehow impacting the cluster (wide
>> rows are a natural candidate).
>>
>>
>>
>> You don’t mention your version, but most modern versions of Cassandra
>> ship with ‘nodetool toppartitions’, which will help you identify frequently
>> written/read partitions – perhaps you can use that to identify a hotspot
>> due to some external behavior (some partition being read thousands of
>> times, over and over could certainly drive up load).
>>
>>
>>
>> - Jeff
>>
>>
>>
>> *From: *Pranay akula <pr...@gmail.com>
>> *Reply-To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
>> *Date: *Monday, September 19, 2016 at 7:53 AM
>> *To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
>> *Subject: *High load on few nodes in a DC.
>>
>>
>>
>> when our cluster was under load i am seeing 1 or 2 nodes are on more
>> load consistently when compared to others in dc i am not seeing any GC
>> pauses or wide partitions is this can be those nodes are continuously
>> serving as coordinators ?? how can i find what is the reason for high load
>> on those two nodes ?? We are using Vnode.
>>
>>
>>
>>
>>
>> Thanks
>>
>> Pranay.
>>
>
>
Re: High load on few nodes in a DC.
Posted by Pranay akula <pr...@gmail.com>.
Hai Jeff,
Thank, we are using RF 3 and cassandra version 2.1.8.
Thanks
Pranay.
On Mon, Sep 19, 2016 at 10:55 AM, Jeff Jirsa <je...@crowdstrike.com>
wrote:
> Is your replication_factor 2? Or is it 3? What version are you using?
>
>
>
> The most likely answer is some individual partition that’s either being
> written/read more than others, or is somehow impacting the cluster (wide
> rows are a natural candidate).
>
>
>
> You don’t mention your version, but most modern versions of Cassandra ship
> with ‘nodetool toppartitions’, which will help you identify frequently
> written/read partitions – perhaps you can use that to identify a hotspot
> due to some external behavior (some partition being read thousands of
> times, over and over could certainly drive up load).
>
>
>
> - Jeff
>
>
>
> *From: *Pranay akula <pr...@gmail.com>
> *Reply-To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
> *Date: *Monday, September 19, 2016 at 7:53 AM
> *To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
> *Subject: *High load on few nodes in a DC.
>
>
>
> when our cluster was under load i am seeing 1 or 2 nodes are on more
> load consistently when compared to others in dc i am not seeing any GC
> pauses or wide partitions is this can be those nodes are continuously
> serving as coordinators ?? how can i find what is the reason for high load
> on those two nodes ?? We are using Vnode.
>
>
>
>
>
> Thanks
>
> Pranay.
>
Re: High load on few nodes in a DC.
Posted by Jeff Jirsa <je...@crowdstrike.com>.
Is your replication_factor 2? Or is it 3? What version are you using?
The most likely answer is some individual partition that’s either being written/read more than others, or is somehow impacting the cluster (wide rows are a natural candidate).
You don’t mention your version, but most modern versions of Cassandra ship with ‘nodetool toppartitions’, which will help you identify frequently written/read partitions – perhaps you can use that to identify a hotspot due to some external behavior (some partition being read thousands of times, over and over could certainly drive up load).
- Jeff
From: Pranay akula <pr...@gmail.com>
Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Date: Monday, September 19, 2016 at 7:53 AM
To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Subject: High load on few nodes in a DC.
when our cluster was under load i am seeing 1 or 2 nodes are on more load consistently when compared to others in dc i am not seeing any GC pauses or wide partitions is this can be those nodes are continuously serving as coordinators ?? how can i find what is the reason for high load on those two nodes ?? We are using Vnode.
Thanks
Pranay.