You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Rahul Neelakantan <ra...@rahul.be> on 2014/09/05 15:12:47 UTC
Questions about cleaning up/purging Hinted Handoffs
The questions are for Cassandra 1.0.8 but may apply to later versions too.
1) When the coordinator notices that a node it is collecting hints for is
down for more than the max hint window, it stops collecting hints for that
node
- Does it automatically purge the hints it collected so far, or does
removing them require manual intervention?
- What is the recommended way to remove hints if they are not
automatically removed? Stop node and delete hints CF db files? or use
truncate CF?
2) When the coordinator replays hints to a node that has come back within
the max hint window, does it purge those hints once they are replayed or
does cleanup of those hints require manual intervention?
3) If a node is flapping because of network issues and gossip keeps
thinking the node is down, causing the coordinator to reset the max hint
window for that node and start collecting hints again for the node
- Does it automatically purge the hints collected in the previous windows?
- Or Does it replay all hints collected in all windows that the node was
down?
- Rahul
Re: Questions about cleaning up/purging Hinted Handoffs
Posted by Rahul Neelakantan <ra...@rahul.be>.
Will try... Thank you
Rahul Neelakantan
> On Sep 10, 2014, at 12:01 AM, Rahul Menon <ra...@apigee.com> wrote:
>
> I use jmxterm. http://wiki.cyclopsgroup.org/jmxterm/ attach it to your c* process and then use the org.apache.cassandra.db:HintedHandoffManager bean and run deleteHintsforEndpoint <ip> to drop hints for each ip.
>
>
>
>
>> On Wed, Sep 10, 2014 at 3:37 AM, Rahul Neelakantan <ra...@rahul.be> wrote:
>> RF=3, two DCs. (Going to 1.2.x in a few weeks)
>>
>> What's the procedure to drop via JMX?
>>
>> - Rahul
>> 1-678-451-4545 (US)
>> +91 99018-06625 (India)
>>
>>
>>> On Sep 9, 2014, at 9:23 AM, Rahul Menon <ra...@apigee.com> wrote:
>>>
>>> Yep, the hinted handoff in 1.0.8 is abysmal at best. What is your replication facter, i have had huge hints pile up, where i had to drop the entire coloumn family and then run a repair. Either that or you can use the JMX HintedHandoffManager and delete hints per endpoint. Also it maybe worthwhile to investigate why you have hints pile up
>>>
>>> Rahul
>>>
>>>> On Mon, Sep 8, 2014 at 11:09 PM, Robert Coli <rc...@eventbrite.com> wrote:
>>>>> On Fri, Sep 5, 2014 at 3:20 PM, Rahul Neelakantan <ra...@rahul.be> wrote:
>>>>> The reason I asked about he hints is because I see hints being replayed but the large compacted hints stable still sticks around, perhaps it is a bug with that version .
>>>>
>>>> I've seen this behavior with HH in older versions, so probably.
>>>>
>>>> =Rob
>
Re: Questions about cleaning up/purging Hinted Handoffs
Posted by Rahul Menon <ra...@apigee.com>.
I use jmxterm. http://wiki.cyclopsgroup.org/jmxterm/ attach it to your c*
process and then use the org.apache.cassandra.db:HintedHandoffManager bean
and run deleteHintsforEndpoint <ip> to drop hints for each ip.
On Wed, Sep 10, 2014 at 3:37 AM, Rahul Neelakantan <ra...@rahul.be> wrote:
> RF=3, two DCs. (Going to 1.2.x in a few weeks)
>
> What's the procedure to drop via JMX?
>
> - Rahul
> 1-678-451-4545 (US)
> +91 99018-06625 (India)
>
>
> On Sep 9, 2014, at 9:23 AM, Rahul Menon <ra...@apigee.com> wrote:
>
> Yep, the hinted handoff in 1.0.8 is abysmal at best. What is your
> replication facter, i have had huge hints pile up, where i had to drop the
> entire coloumn family and then run a repair. Either that or you can use the
> JMX HintedHandoffManager and delete hints per endpoint. Also it maybe
> worthwhile to investigate why you have hints pile up
>
> Rahul
>
> On Mon, Sep 8, 2014 at 11:09 PM, Robert Coli <rc...@eventbrite.com> wrote:
>
>> On Fri, Sep 5, 2014 at 3:20 PM, Rahul Neelakantan <ra...@rahul.be> wrote:
>>
>>> The reason I asked about he hints is because I see hints being replayed
>>> but the large compacted hints stable still sticks around, perhaps it is a
>>> bug with that version .
>>>
>>
>> I've seen this behavior with HH in older versions, so probably.
>>
>> =Rob
>>
>>
>
>
Re: Questions about cleaning up/purging Hinted Handoffs
Posted by Rahul Neelakantan <ra...@rahul.be>.
RF=3, two DCs. (Going to 1.2.x in a few weeks)
What's the procedure to drop via JMX?
- Rahul
1-678-451-4545 (US)
+91 99018-06625 (India)
On Sep 9, 2014, at 9:23 AM, Rahul Menon <ra...@apigee.com> wrote:
> Yep, the hinted handoff in 1.0.8 is abysmal at best. What is your replication facter, i have had huge hints pile up, where i had to drop the entire coloumn family and then run a repair. Either that or you can use the JMX HintedHandoffManager and delete hints per endpoint. Also it maybe worthwhile to investigate why you have hints pile up
>
> Rahul
>
> On Mon, Sep 8, 2014 at 11:09 PM, Robert Coli <rc...@eventbrite.com> wrote:
>> On Fri, Sep 5, 2014 at 3:20 PM, Rahul Neelakantan <ra...@rahul.be> wrote:
>>> The reason I asked about he hints is because I see hints being replayed but the large compacted hints stable still sticks around, perhaps it is a bug with that version .
>>
>> I've seen this behavior with HH in older versions, so probably.
>>
>> =Rob
>
Re: Questions about cleaning up/purging Hinted Handoffs
Posted by Rahul Menon <ra...@apigee.com>.
Yep, the hinted handoff in 1.0.8 is abysmal at best. What is your
replication facter, i have had huge hints pile up, where i had to drop the
entire coloumn family and then run a repair. Either that or you can use the
JMX HintedHandoffManager and delete hints per endpoint. Also it maybe
worthwhile to investigate why you have hints pile up
Rahul
On Mon, Sep 8, 2014 at 11:09 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Fri, Sep 5, 2014 at 3:20 PM, Rahul Neelakantan <ra...@rahul.be> wrote:
>
>> The reason I asked about he hints is because I see hints being replayed
>> but the large compacted hints stable still sticks around, perhaps it is a
>> bug with that version .
>>
>
> I've seen this behavior with HH in older versions, so probably.
>
> =Rob
>
>
Re: Questions about cleaning up/purging Hinted Handoffs
Posted by Robert Coli <rc...@eventbrite.com>.
On Fri, Sep 5, 2014 at 3:20 PM, Rahul Neelakantan <ra...@rahul.be> wrote:
> The reason I asked about he hints is because I see hints being replayed
> but the large compacted hints stable still sticks around, perhaps it is a
> bug with that version .
>
I've seen this behavior with HH in older versions, so probably.
=Rob
Re: Questions about cleaning up/purging Hinted Handoffs
Posted by Rahul Neelakantan <ra...@rahul.be>.
Yup... Upgrades are in the works.. Thanks for the response.
The reason I asked about he hints is because I see hints being replayed but the large compacted hints stable still sticks around, perhaps it is a bug with that version .
Rahul Neelakantan
> On Sep 5, 2014, at 1:03 PM, Robert Coli <rc...@eventbrite.com> wrote:
>
>> On Fri, Sep 5, 2014 at 6:12 AM, Rahul Neelakantan <ra...@rahul.be> wrote:
>> The questions are for Cassandra 1.0.8 but may apply to later versions too.
>
> Upgrade your very old version of Cassandra ASAP. I barely remember the implementation of HH in your old version.
>
>> 1) When the coordinator notices that a node it is collecting hints for is down for more than the max hint window, it stops collecting hints for that node
>> - Does it automatically purge the hints it collected so far, or does removing them require manual intervention?
>
> It tries to deliver the hints it has stored so far, when it can. When it delivers them, it deletes them.
>
>> - What is the recommended way to remove hints if they are not automatically removed? Stop node and delete hints CF db files? or use truncate CF?
>
> Either way is fine. There's also a JMX method (maybe not in your old version) to purge hints.
>
>> 2) When the coordinator replays hints to a node that has come back within the max hint window, does it purge those hints once they are replayed or does cleanup of those hints require manual intervention?
>
> The former.
>
>> 3) If a node is flapping because of network issues and gossip keeps thinking the node is down, causing the coordinator to reset the max hint window for that node and start collecting hints again for the node
>> - Does it automatically purge the hints collected in the previous windows?
>
> No.
>
>> - Or Does it replay all hints collected in all windows that the node was down?
>
> Yes.
>
> =Rob
>
Re: Questions about cleaning up/purging Hinted Handoffs
Posted by Robert Coli <rc...@eventbrite.com>.
On Fri, Sep 5, 2014 at 6:12 AM, Rahul Neelakantan <ra...@rahul.be> wrote:
> The questions are for Cassandra 1.0.8 but may apply to later versions too.
>
Upgrade your very old version of Cassandra ASAP. I barely remember the
implementation of HH in your old version.
> 1) When the coordinator notices that a node it is collecting hints for is
> down for more than the max hint window, it stops collecting hints for that
> node
> - Does it automatically purge the hints it collected so far, or does
> removing them require manual intervention?
>
It tries to deliver the hints it has stored so far, when it can. When it
delivers them, it deletes them.
> - What is the recommended way to remove hints if they are not
> automatically removed? Stop node and delete hints CF db files? or use
> truncate CF?
>
Either way is fine. There's also a JMX method (maybe not in your old
version) to purge hints.
> 2) When the coordinator replays hints to a node that has come back within
> the max hint window, does it purge those hints once they are replayed or
> does cleanup of those hints require manual intervention?
>
The former.
> 3) If a node is flapping because of network issues and gossip keeps
> thinking the node is down, causing the coordinator to reset the max hint
> window for that node and start collecting hints again for the node
> - Does it automatically purge the hints collected in the previous windows?
>
No.
> - Or Does it replay all hints collected in all windows that the node was
> down?
>
Yes.
=Rob