You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Andras Szerdahelyi <an...@ignitionone.com> on 2013/03/14 14:25:51 UTC
33million hinted handoffs from nowhere
Hi list,
I am experiencing seemingly uncontrollable and unexplained growth of my HintedHandoff CF on a single node. Unexplained because there are no hinted handoffs being logged on the node, uncontrollable because I see 33 million inserts in cfstats and the size of the stables is over 10 gigs all in an hour of uptime.
I have done the following to try and reproduce this:
- shut down my cluster
- on all nodes: remove sstables from the HintsColumnFamily data dir
- on all nodes: remove commit logs
- start all nodes but the one that’s showing this problem
- nothing is writing to any of the nodes. There are no hinted handoff going on anywhere
- bring back the node in question last
- few seconds after boot:
Column Family: HintsColumnFamily
SSTable count: 1
Space used (live): 44946532
Space used (total): 44946532
Number of Keys (estimate): 256
Memtable Columns Count: 17840
Memtable Data Size: 17569909
Memtable Switch Count: 2
Read Count: 0
Read Latency: NaN ms.
Write Count: 184836
Write Latency: 0.668 ms.
Pending Tasks: 0
Bloom Filter False Postives: 0
Bloom Filter False Ratio: 0.00000
Bloom Filter Space Used: 16
Compacted row minimum size: 20924301
Compacted row maximum size: 25109160
Compacted row mean size: 25109160
Re: 33million hinted handoffs from nowhere
Posted by aaron morton <aa...@thelastpickle.com>.
Just noticed this on your email
> ps: all nodes are cassandra-1.1.6-dse-p1
If you are on DSE I'd recommend hitting their forums or the support channels.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 26/03/2013, at 1:41 AM, Andras Szerdahelyi <an...@ignitionone.com> wrote:
> Thanks again!
>
> Nodetool gossipinfo correctly lists the existing nodes only
>
> - last change to ring topology was months ago
> - I started the problem node with -Dcassanda.load_ring_state=false and
> observed no unusual behaviour ( this is with hinted handoff OFF. With
> hinted handoff ON I see the same behaviour as before )
> - tried to assassinate these endpoints but got UnknownHostExceptions for
> all 3
> - tried remove token but got java.lang.UnsupportedOperationException:
> Token not found. For all of these
>
> I have an update however. I changed which node I use to coordinate
> mutations and its happening elsewhere too, same tokens.
> I'm clueless as to what could have caused my ring to end up in such an
> inconsistent state.. How can there be pending hints for endpoints
> gossipinfo does not know about?
>
> Regards,
> Andras
>
>
> On 21/03/13 17:56, "aaron morton" <aa...@thelastpickle.com> wrote:
>
>> Take a look a nodetool gossipinfo it will tell you what nodes the node
>> thinks are around.
>>
>> If you can see something in gossip that should not be there you have a
>> few choices.
>>
>> * if it's less than 3 days since a change to ring topology wait and see
>> if C* sorts it out.
>> * try restarting nodes with -Dcassanda.load_ring_state=false as a JVM opt
>> in cassandra-env.sh. This may not work because when the node restarts
>> others will tell it the bad info
>> * try the unsafeAssassinateEndpoint() call on the Gossiper MBean via JMX
>> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassand
>> ra/gms/GossiperMBean.java#L28
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Consultant
>> New Zealand
>>
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 20/03/2013, at 11:10 PM, Andras Szerdahelyi
>> <an...@ignitionone.com> wrote:
>>
>>> Thanks, Aaron.
>>>
>>> I re-enabled hinted handoff and noted the following
>>> € no host is marked down in nodetool ring
>>> € No host is logged as down or dead in logs
>>> € No "started hinted handoff for.." is logged
>>> € The hinted handoff manager Mbean lists pending hints to ..
>>> (drumroll) 3 non-existent nodes?
>>>
>>> Here's my ring
>>>
>>> Note: Ownership information does not include topology, please specify a
>>> keyspace.
>>> Address DC Rack Status State Load
>>> Owns Token
>>>
>>> 113427455640312821154458202477256070785
>>> XX.XX.1.113 ione-us-atl rack1 Up Normal 382.08 GB
>>> 33.33% 0
>>> XX.XX.31.10 ione-us-lvg rack1 Up Normal 266.04 GB
>>> 0.00% 100
>>> XX.XX.0.71 ione-be-bru rack1 Up Normal 85.86 GB
>>> 0.00% 200
>>> XX.XX.2.86 ione-analytics-us-atlrack1 Up Normal 153.6 GB
>>> 0.00% 300
>>> XX.XX.1.45 ione-us-atl-ssdrack1 Up Normal 296.72 GB
>>> 0.00% 400
>>> XX.XX.2.85 ione-analytics-us-atlrack1 Up Normal 100.3 GB
>>> 33.33% 56713727820156410577229101238628035542
>>> XX.XX.1.204 ione-us-atl rack1 Up Normal 341.55 GB
>>> 16.67% 85070591730234615865843651857942052864
>>> XX.XX.11 ione-us-lvg rack1 Up Normal 320.22 GB
>>> 0.00% 85070591730234615865843651857942052964
>>> XX.XX.2.87 ione-analytics-us-atlrack1 Up Normal 166.48
>>> GB 16.67% 113427455640312821154458202477256070785
>>>
>>> And these are nodes pending hints according to the Mbean
>>>
>>> 166860289390734216023086131251507064403
>>> 143927757573010354572009627285182898319
>>> 24295500190543334543807902779534181373
>>>
>>> Err.. Unbalanced ring ? Opscenter says otherwise ( "OpsCenter has
>>> detected that the token ranges are evenly distributed across the nodes
>>> in each data center. Load rebalancing is not necessary at this time." )
>>>
>>> I appreciate your help so far! In the mean time hintedhandOFF because
>>> my mutation TP can't keep up with this traffic, not to mention
>>> compaction..
>>>
>>> Thanks,
>>> Andras
>>>
>>> ps: all nodes are cassandra-1.1.6-dse-p1
>>>
>>> From: aaron morton <aa...@thelastpickle.com>
>>> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>>> Date: Monday 18 March 2013 17:51
>>> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>>> Subject: Re: 33million hinted handoffs from nowhere
>>>
>>> You can check which nodes hints are being held for using the JMX api.
>>> Look for the org.apache.cassandra.db:type=HintedHandoffManager MBean and
>>> call the listEndpointsPendingHints() function.
>>>
>>> There are two points where hints may be stored, if the node is down
>>> when the request started or if the node timed out and did not return
>>> before rpc_timeout. To check for the first, look for log lines about a
>>> node being "dead" on the coordinator. To check for the second look for
>>> dropped messages on the other nodes. This will be logged, or you can use
>>> nodetool tpstats to look for them.
>>>
>>> Cheers
>>>
>>> -----------------
>>> Aaron Morton
>>> Freelance Cassandra Consultant
>>> New Zealand
>>>
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 15/03/2013, at 2:30 AM, Andras Szerdahelyi
>>> <an...@ignitionone.com> wrote:
>>>
>>>> ( The previous letter was sent prematurely, sorry. )
>>>>
>>>> This node is the only node being written to, but the Cfs being written
>>>> replicate to almost all of the other nodes
>>>> My understanding is that hinted handoff is mutations kept around on
>>>> the coordinator node, to be replayed when the target node re-appears on
>>>> the ring. All my nodes are up and again, no hinted handoff is logged on
>>>> the node itself
>>>>
>>>> Thanks!
>>>> Andras
>>>>
>>>> From: Andras Szerdahelyi <an...@ignitionone.com>
>>>> Date: Thursday 14 March 2013 14:25
>>>> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>>>> Subject: 33million hinted handoffs from nowhere
>>>>
>>>> Hi list,
>>>>
>>>> I am experiencing seemingly uncontrollable and unexplained growth of
>>>> my HintedHandoff CF on a single node. Unexplained because there are no
>>>> hinted handoffs being logged on the node, uncontrollable because I see
>>>> 33 million inserts in cfstats and the size of the stables is over 10
>>>> gigs all in an hour of uptime.
>>>>
>>>>
>>>> I have done the following to try and reproduce this:
>>>>
>>>> - shut down my cluster
>>>> - on all nodes: remove sstables from the HintsColumnFamily data dir
>>>> - on all nodes: remove commit logs
>>>> - start all nodes but the one that¹s showing this problem
>>>> - nothing is writing to any of the nodes. There are no hinted handoff
>>>> going on anywhere
>>>> - bring back the node in question last
>>>> - few seconds after boot:
>>>>
>>>> Column Family: HintsColumnFamily
>>>> SSTable count: 1
>>>> Space used (live): 44946532
>>>> Space used (total): 44946532
>>>> Number of Keys (estimate): 256
>>>> Memtable Columns Count: 17840
>>>> Memtable Data Size: 17569909
>>>> Memtable Switch Count: 2
>>>> Read Count: 0
>>>> Read Latency: NaN ms.
>>>> Write Count: 184836
>>>> Write Latency: 0.668 ms.
>>>> Pending Tasks: 0
>>>> Bloom Filter False Postives: 0
>>>> Bloom Filter False Ratio: 0.00000
>>>> Bloom Filter Space Used: 16
>>>> Compacted row minimum size: 20924301
>>>> Compacted row maximum size: 25109160
>>>> Compacted row mean size: 25109160
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: 33million hinted handoffs from nowhere
Posted by Andras Szerdahelyi <an...@ignitionone.com>.
Thanks again!
Nodetool gossipinfo correctly lists the existing nodes only
- last change to ring topology was months ago
- I started the problem node with -Dcassanda.load_ring_state=false and
observed no unusual behaviour ( this is with hinted handoff OFF. With
hinted handoff ON I see the same behaviour as before )
- tried to assassinate these endpoints but got UnknownHostExceptions for
all 3
- tried remove token but got java.lang.UnsupportedOperationException:
Token not found. For all of these
I have an update however. I changed which node I use to coordinate
mutations and its happening elsewhere too, same tokens.
I'm clueless as to what could have caused my ring to end up in such an
inconsistent state.. How can there be pending hints for endpoints
gossipinfo does not know about?
Regards,
Andras
On 21/03/13 17:56, "aaron morton" <aa...@thelastpickle.com> wrote:
>Take a look a nodetool gossipinfo it will tell you what nodes the node
>thinks are around.
>
>If you can see something in gossip that should not be there you have a
>few choices.
>
>* if it's less than 3 days since a change to ring topology wait and see
>if C* sorts it out.
>* try restarting nodes with -Dcassanda.load_ring_state=false as a JVM opt
>in cassandra-env.sh. This may not work because when the node restarts
>others will tell it the bad info
>* try the unsafeAssassinateEndpoint() call on the Gossiper MBean via JMX
>https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassand
>ra/gms/GossiperMBean.java#L28
>
>Cheers
>
>-----------------
>Aaron Morton
>Freelance Cassandra Consultant
>New Zealand
>
>@aaronmorton
>http://www.thelastpickle.com
>
>On 20/03/2013, at 11:10 PM, Andras Szerdahelyi
><an...@ignitionone.com> wrote:
>
>> Thanks, Aaron.
>>
>> I re-enabled hinted handoff and noted the following
>> € no host is marked down in nodetool ring
>> € No host is logged as down or dead in logs
>> € No "started hinted handoff for.." is logged
>> € The hinted handoff manager Mbean lists pending hints to ..
>>(drumroll) 3 non-existent nodes?
>>
>> Here's my ring
>>
>> Note: Ownership information does not include topology, please specify a
>>keyspace.
>> Address DC Rack Status State Load
>>Owns Token
>>
>> 113427455640312821154458202477256070785
>> XX.XX.1.113 ione-us-atl rack1 Up Normal 382.08 GB
>>33.33% 0
>> XX.XX.31.10 ione-us-lvg rack1 Up Normal 266.04 GB
>>0.00% 100
>> XX.XX.0.71 ione-be-bru rack1 Up Normal 85.86 GB
>>0.00% 200
>> XX.XX.2.86 ione-analytics-us-atlrack1 Up Normal 153.6 GB
>> 0.00% 300
>> XX.XX.1.45 ione-us-atl-ssdrack1 Up Normal 296.72 GB
>> 0.00% 400
>> XX.XX.2.85 ione-analytics-us-atlrack1 Up Normal 100.3 GB
>> 33.33% 56713727820156410577229101238628035542
>> XX.XX.1.204 ione-us-atl rack1 Up Normal 341.55 GB
>>16.67% 85070591730234615865843651857942052864
>> XX.XX.11 ione-us-lvg rack1 Up Normal 320.22 GB
>>0.00% 85070591730234615865843651857942052964
>> XX.XX.2.87 ione-analytics-us-atlrack1 Up Normal 166.48
>>GB 16.67% 113427455640312821154458202477256070785
>>
>> And these are nodes pending hints according to the Mbean
>>
>> 166860289390734216023086131251507064403
>> 143927757573010354572009627285182898319
>> 24295500190543334543807902779534181373
>>
>> Err.. Unbalanced ring ? Opscenter says otherwise ( "OpsCenter has
>>detected that the token ranges are evenly distributed across the nodes
>>in each data center. Load rebalancing is not necessary at this time." )
>>
>> I appreciate your help so far! In the mean time hintedhandOFF because
>>my mutation TP can't keep up with this traffic, not to mention
>>compaction..
>>
>> Thanks,
>> Andras
>>
>> ps: all nodes are cassandra-1.1.6-dse-p1
>>
>> From: aaron morton <aa...@thelastpickle.com>
>> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>> Date: Monday 18 March 2013 17:51
>> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>> Subject: Re: 33million hinted handoffs from nowhere
>>
>> You can check which nodes hints are being held for using the JMX api.
>>Look for the org.apache.cassandra.db:type=HintedHandoffManager MBean and
>>call the listEndpointsPendingHints() function.
>>
>> There are two points where hints may be stored, if the node is down
>>when the request started or if the node timed out and did not return
>>before rpc_timeout. To check for the first, look for log lines about a
>>node being "dead" on the coordinator. To check for the second look for
>>dropped messages on the other nodes. This will be logged, or you can use
>>nodetool tpstats to look for them.
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Consultant
>> New Zealand
>>
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 15/03/2013, at 2:30 AM, Andras Szerdahelyi
>><an...@ignitionone.com> wrote:
>>
>>> ( The previous letter was sent prematurely, sorry. )
>>>
>>> This node is the only node being written to, but the Cfs being written
>>>replicate to almost all of the other nodes
>>> My understanding is that hinted handoff is mutations kept around on
>>>the coordinator node, to be replayed when the target node re-appears on
>>>the ring. All my nodes are up and again, no hinted handoff is logged on
>>>the node itself
>>>
>>> Thanks!
>>> Andras
>>>
>>> From: Andras Szerdahelyi <an...@ignitionone.com>
>>> Date: Thursday 14 March 2013 14:25
>>> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>>> Subject: 33million hinted handoffs from nowhere
>>>
>>> Hi list,
>>>
>>> I am experiencing seemingly uncontrollable and unexplained growth of
>>>my HintedHandoff CF on a single node. Unexplained because there are no
>>>hinted handoffs being logged on the node, uncontrollable because I see
>>>33 million inserts in cfstats and the size of the stables is over 10
>>>gigs all in an hour of uptime.
>>>
>>>
>>> I have done the following to try and reproduce this:
>>>
>>> - shut down my cluster
>>> - on all nodes: remove sstables from the HintsColumnFamily data dir
>>> - on all nodes: remove commit logs
>>> - start all nodes but the one that¹s showing this problem
>>> - nothing is writing to any of the nodes. There are no hinted handoff
>>>going on anywhere
>>> - bring back the node in question last
>>> - few seconds after boot:
>>>
>>> Column Family: HintsColumnFamily
>>> SSTable count: 1
>>> Space used (live): 44946532
>>> Space used (total): 44946532
>>> Number of Keys (estimate): 256
>>> Memtable Columns Count: 17840
>>> Memtable Data Size: 17569909
>>> Memtable Switch Count: 2
>>> Read Count: 0
>>> Read Latency: NaN ms.
>>> Write Count: 184836
>>> Write Latency: 0.668 ms.
>>> Pending Tasks: 0
>>> Bloom Filter False Postives: 0
>>> Bloom Filter False Ratio: 0.00000
>>> Bloom Filter Space Used: 16
>>> Compacted row minimum size: 20924301
>>> Compacted row maximum size: 25109160
>>> Compacted row mean size: 25109160
>>>
>>>
>>>
>>>
>>
>
Re: 33million hinted handoffs from nowhere
Posted by aaron morton <aa...@thelastpickle.com>.
Take a look a nodetool gossipinfo it will tell you what nodes the node thinks are around.
If you can see something in gossip that should not be there you have a few choices.
* if it's less than 3 days since a change to ring topology wait and see if C* sorts it out.
* try restarting nodes with -Dcassanda.load_ring_state=false as a JVM opt in cassandra-env.sh. This may not work because when the node restarts others will tell it the bad info
* try the unsafeAssassinateEndpoint() call on the Gossiper MBean via JMX https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/gms/GossiperMBean.java#L28
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 20/03/2013, at 11:10 PM, Andras Szerdahelyi <an...@ignitionone.com> wrote:
> Thanks, Aaron.
>
> I re-enabled hinted handoff and noted the following
> • no host is marked down in nodetool ring
> • No host is logged as down or dead in logs
> • No "started hinted handoff for.." is logged
> • The hinted handoff manager Mbean lists pending hints to .. (drumroll) 3 non-existent nodes?
>
> Here's my ring
>
> Note: Ownership information does not include topology, please specify a keyspace.
> Address DC Rack Status State Load Owns Token
> 113427455640312821154458202477256070785
> XX.XX.1.113 ione-us-atl rack1 Up Normal 382.08 GB 33.33% 0
> XX.XX.31.10 ione-us-lvg rack1 Up Normal 266.04 GB 0.00% 100
> XX.XX.0.71 ione-be-bru rack1 Up Normal 85.86 GB 0.00% 200
> XX.XX.2.86 ione-analytics-us-atlrack1 Up Normal 153.6 GB 0.00% 300
> XX.XX.1.45 ione-us-atl-ssdrack1 Up Normal 296.72 GB 0.00% 400
> XX.XX.2.85 ione-analytics-us-atlrack1 Up Normal 100.3 GB 33.33% 56713727820156410577229101238628035542
> XX.XX.1.204 ione-us-atl rack1 Up Normal 341.55 GB 16.67% 85070591730234615865843651857942052864
> XX.XX.11 ione-us-lvg rack1 Up Normal 320.22 GB 0.00% 85070591730234615865843651857942052964
> XX.XX.2.87 ione-analytics-us-atlrack1 Up Normal 166.48 GB 16.67% 113427455640312821154458202477256070785
>
> And these are nodes pending hints according to the Mbean
>
> 166860289390734216023086131251507064403
> 143927757573010354572009627285182898319
> 24295500190543334543807902779534181373
>
> Err.. Unbalanced ring ? Opscenter says otherwise ( "OpsCenter has detected that the token ranges are evenly distributed across the nodes in each data center. Load rebalancing is not necessary at this time." )
>
> I appreciate your help so far! In the mean time hintedhandOFF because my mutation TP can't keep up with this traffic, not to mention compaction..
>
> Thanks,
> Andras
>
> ps: all nodes are cassandra-1.1.6-dse-p1
>
> From: aaron morton <aa...@thelastpickle.com>
> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Date: Monday 18 March 2013 17:51
> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Subject: Re: 33million hinted handoffs from nowhere
>
> You can check which nodes hints are being held for using the JMX api. Look for the org.apache.cassandra.db:type=HintedHandoffManager MBean and call the listEndpointsPendingHints() function.
>
> There are two points where hints may be stored, if the node is down when the request started or if the node timed out and did not return before rpc_timeout. To check for the first, look for log lines about a node being "dead" on the coordinator. To check for the second look for dropped messages on the other nodes. This will be logged, or you can use nodetool tpstats to look for them.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 15/03/2013, at 2:30 AM, Andras Szerdahelyi <an...@ignitionone.com> wrote:
>
>> ( The previous letter was sent prematurely, sorry. )
>>
>> This node is the only node being written to, but the Cfs being written replicate to almost all of the other nodes
>> My understanding is that hinted handoff is mutations kept around on the coordinator node, to be replayed when the target node re-appears on the ring. All my nodes are up and again, no hinted handoff is logged on the node itself
>>
>> Thanks!
>> Andras
>>
>> From: Andras Szerdahelyi <an...@ignitionone.com>
>> Date: Thursday 14 March 2013 14:25
>> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>> Subject: 33million hinted handoffs from nowhere
>>
>> Hi list,
>>
>> I am experiencing seemingly uncontrollable and unexplained growth of my HintedHandoff CF on a single node. Unexplained because there are no hinted handoffs being logged on the node, uncontrollable because I see 33 million inserts in cfstats and the size of the stables is over 10 gigs all in an hour of uptime.
>>
>>
>> I have done the following to try and reproduce this:
>>
>> - shut down my cluster
>> - on all nodes: remove sstables from the HintsColumnFamily data dir
>> - on all nodes: remove commit logs
>> - start all nodes but the one that’s showing this problem
>> - nothing is writing to any of the nodes. There are no hinted handoff going on anywhere
>> - bring back the node in question last
>> - few seconds after boot:
>>
>> Column Family: HintsColumnFamily
>> SSTable count: 1
>> Space used (live): 44946532
>> Space used (total): 44946532
>> Number of Keys (estimate): 256
>> Memtable Columns Count: 17840
>> Memtable Data Size: 17569909
>> Memtable Switch Count: 2
>> Read Count: 0
>> Read Latency: NaN ms.
>> Write Count: 184836
>> Write Latency: 0.668 ms.
>> Pending Tasks: 0
>> Bloom Filter False Postives: 0
>> Bloom Filter False Ratio: 0.00000
>> Bloom Filter Space Used: 16
>> Compacted row minimum size: 20924301
>> Compacted row maximum size: 25109160
>> Compacted row mean size: 25109160
>>
>>
>>
>>
>
Re: 33million hinted handoffs from nowhere
Posted by Andras Szerdahelyi <an...@ignitionone.com>.
Thanks, Aaron.
I re-enabled hinted handoff and noted the following
* no host is marked down in nodetool ring
* No host is logged as down or dead in logs
* No "started hinted handoff for.." is logged
* The hinted handoff manager Mbean lists pending hints to .. (drumroll) 3 non-existent nodes?
Here's my ring
Note: Ownership information does not include topology, please specify a keyspace.
Address DC Rack Status State Load Owns Token
113427455640312821154458202477256070785
XX.XX.1.113 ione-us-atl rack1 Up Normal 382.08 GB 33.33% 0
XX.XX.31.10 ione-us-lvg rack1 Up Normal 266.04 GB 0.00% 100
XX.XX.0.71 ione-be-bru rack1 Up Normal 85.86 GB 0.00% 200
XX.XX.2.86 ione-analytics-us-atlrack1 Up Normal 153.6 GB 0.00% 300
XX.XX.1.45 ione-us-atl-ssdrack1 Up Normal 296.72 GB 0.00% 400
XX.XX.2.85 ione-analytics-us-atlrack1 Up Normal 100.3 GB 33.33% 56713727820156410577229101238628035542
XX.XX.1.204 ione-us-atl rack1 Up Normal 341.55 GB 16.67% 85070591730234615865843651857942052864
XX.XX.11 ione-us-lvg rack1 Up Normal 320.22 GB 0.00% 85070591730234615865843651857942052964
XX.XX.2.87 ione-analytics-us-atlrack1 Up Normal 166.48 GB 16.67% 113427455640312821154458202477256070785
And these are nodes pending hints according to the Mbean
166860289390734216023086131251507064403
143927757573010354572009627285182898319
24295500190543334543807902779534181373
Err.. Unbalanced ring ? Opscenter says otherwise ( "OpsCenter has detected that the token ranges are evenly distributed across the nodes in each data center. Load rebalancing is not necessary at this time." )
I appreciate your help so far! In the mean time hintedhandOFF because my mutation TP can't keep up with this traffic, not to mention compaction..
Thanks,
Andras
ps: all nodes are cassandra-1.1.6-dse-p1
From: aaron morton <aa...@thelastpickle.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Monday 18 March 2013 17:51
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: 33million hinted handoffs from nowhere
You can check which nodes hints are being held for using the JMX api. Look for the org.apache.cassandra.db:type=HintedHandoffManager MBean and call the listEndpointsPendingHints() function.
There are two points where hints may be stored, if the node is down when the request started or if the node timed out and did not return before rpc_timeout. To check for the first, look for log lines about a node being "dead" on the coordinator. To check for the second look for dropped messages on the other nodes. This will be logged, or you can use nodetool tpstats to look for them.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 15/03/2013, at 2:30 AM, Andras Szerdahelyi <an...@ignitionone.com>> wrote:
( The previous letter was sent prematurely, sorry. )
This node is the only node being written to, but the Cfs being written replicate to almost all of the other nodes
My understanding is that hinted handoff is mutations kept around on the coordinator node, to be replayed when the target node re-appears on the ring. All my nodes are up and again, no hinted handoff is logged on the node itself
Thanks!
Andras
From: Andras Szerdahelyi <an...@ignitionone.com>>
Date: Thursday 14 March 2013 14:25
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: 33million hinted handoffs from nowhere
Hi list,
I am experiencing seemingly uncontrollable and unexplained growth of my HintedHandoff CF on a single node. Unexplained because there are no hinted handoffs being logged on the node, uncontrollable because I see 33 million inserts in cfstats and the size of the stables is over 10 gigs all in an hour of uptime.
I have done the following to try and reproduce this:
- shut down my cluster
- on all nodes: remove sstables from the HintsColumnFamily data dir
- on all nodes: remove commit logs
- start all nodes but the one that’s showing this problem
- nothing is writing to any of the nodes. There are no hinted handoff going on anywhere
- bring back the node in question last
- few seconds after boot:
Column Family: HintsColumnFamily
SSTable count: 1
Space used (live): 44946532
Space used (total): 44946532
Number of Keys (estimate): 256
Memtable Columns Count: 17840
Memtable Data Size: 17569909
Memtable Switch Count: 2
Read Count: 0
Read Latency: NaN ms.
Write Count: 184836
Write Latency: 0.668 ms.
Pending Tasks: 0
Bloom Filter False Postives: 0
Bloom Filter False Ratio: 0.00000
Bloom Filter Space Used: 16
Compacted row minimum size: 20924301
Compacted row maximum size: 25109160
Compacted row mean size: 25109160
Re: 33million hinted handoffs from nowhere
Posted by aaron morton <aa...@thelastpickle.com>.
You can check which nodes hints are being held for using the JMX api. Look for the org.apache.cassandra.db:type=HintedHandoffManager MBean and call the listEndpointsPendingHints() function.
There are two points where hints may be stored, if the node is down when the request started or if the node timed out and did not return before rpc_timeout. To check for the first, look for log lines about a node being "dead" on the coordinator. To check for the second look for dropped messages on the other nodes. This will be logged, or you can use nodetool tpstats to look for them.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 15/03/2013, at 2:30 AM, Andras Szerdahelyi <an...@ignitionone.com> wrote:
> ( The previous letter was sent prematurely, sorry. )
>
> This node is the only node being written to, but the Cfs being written replicate to almost all of the other nodes
> My understanding is that hinted handoff is mutations kept around on the coordinator node, to be replayed when the target node re-appears on the ring. All my nodes are up and again, no hinted handoff is logged on the node itself
>
> Thanks!
> Andras
>
> From: Andras Szerdahelyi <an...@ignitionone.com>
> Date: Thursday 14 March 2013 14:25
> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Subject: 33million hinted handoffs from nowhere
>
> Hi list,
>
> I am experiencing seemingly uncontrollable and unexplained growth of my HintedHandoff CF on a single node. Unexplained because there are no hinted handoffs being logged on the node, uncontrollable because I see 33 million inserts in cfstats and the size of the stables is over 10 gigs all in an hour of uptime.
>
>
> I have done the following to try and reproduce this:
>
> - shut down my cluster
> - on all nodes: remove sstables from the HintsColumnFamily data dir
> - on all nodes: remove commit logs
> - start all nodes but the one that’s showing this problem
> - nothing is writing to any of the nodes. There are no hinted handoff going on anywhere
> - bring back the node in question last
> - few seconds after boot:
>
> Column Family: HintsColumnFamily
> SSTable count: 1
> Space used (live): 44946532
> Space used (total): 44946532
> Number of Keys (estimate): 256
> Memtable Columns Count: 17840
> Memtable Data Size: 17569909
> Memtable Switch Count: 2
> Read Count: 0
> Read Latency: NaN ms.
> Write Count: 184836
> Write Latency: 0.668 ms.
> Pending Tasks: 0
> Bloom Filter False Postives: 0
> Bloom Filter False Ratio: 0.00000
> Bloom Filter Space Used: 16
> Compacted row minimum size: 20924301
> Compacted row maximum size: 25109160
> Compacted row mean size: 25109160
>
>
>
>
Re: 33million hinted handoffs from nowhere
Posted by Andras Szerdahelyi <an...@ignitionone.com>.
( The previous letter was sent prematurely, sorry. )
This node is the only node being written to, but the Cfs being written replicate to almost all of the other nodes
My understanding is that hinted handoff is mutations kept around on the coordinator node, to be replayed when the target node re-appears on the ring. All my nodes are up and again, no hinted handoff is logged on the node itself
Thanks!
Andras
From: Andras Szerdahelyi <an...@ignitionone.com>>
Date: Thursday 14 March 2013 14:25
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: 33million hinted handoffs from nowhere
Hi list,
I am experiencing seemingly uncontrollable and unexplained growth of my HintedHandoff CF on a single node. Unexplained because there are no hinted handoffs being logged on the node, uncontrollable because I see 33 million inserts in cfstats and the size of the stables is over 10 gigs all in an hour of uptime.
I have done the following to try and reproduce this:
- shut down my cluster
- on all nodes: remove sstables from the HintsColumnFamily data dir
- on all nodes: remove commit logs
- start all nodes but the one that’s showing this problem
- nothing is writing to any of the nodes. There are no hinted handoff going on anywhere
- bring back the node in question last
- few seconds after boot:
Column Family: HintsColumnFamily
SSTable count: 1
Space used (live): 44946532
Space used (total): 44946532
Number of Keys (estimate): 256
Memtable Columns Count: 17840
Memtable Data Size: 17569909
Memtable Switch Count: 2
Read Count: 0
Read Latency: NaN ms.
Write Count: 184836
Write Latency: 0.668 ms.
Pending Tasks: 0
Bloom Filter False Postives: 0
Bloom Filter False Ratio: 0.00000
Bloom Filter Space Used: 16
Compacted row minimum size: 20924301
Compacted row maximum size: 25109160
Compacted row mean size: 25109160