You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by jerome <je...@hotmail.com> on 2016/09/02 20:53:07 UTC

Is it possible to replay hints after running nodetool drain?

Hello,


As part of routine maintenance for our cluster, my colleagues and I will run a nodetool drain before stopping a Cassandra node, performing maintenance, and bringing it back up. We run maintenance as a cron-job with a lock stored in a different cluster to ensure only node is ever down at a time. We would like to make sure the node has replayed all its hints before bringing it down to minimize the potential window in which users might read out-of-date data (we read at a consistency level of ONE). Is it possible to replay hints after performing a nodetool drain? The documentation leads me to believe its not since Cassandra will stop listening for connections from other nodes, but I was unable to find anything definitive either way. If a node won't replay hints after a nodetool drain, is there perhaps another way to tell Cassandra to stop listening for client connections but continue to replay hints to other nodes.


Thanks,

Jerome

Re: Is it possible to replay hints after running nodetool drain?

Posted by jerome <je...@hotmail.com>.

Hi Romain,


I see. Thanks for the feedback, it's much appreciated. Since the only way to force it is through JMX I think we'll continue to use our current method.


Best,

Jerome

________________________________
From: Romain Hardouin <ro...@yahoo.fr>
Sent: Monday, September 5, 2016 1:30:24 PM
To: user@cassandra.apache.org
Subject: Re: Is it possible to replay hints after running nodetool drain?

Hi,

You don't have to worry about that unless you write with CL = ANY.
The sole method to force hints that I know is to invoke scheduleHintDelivery on "org.apache.cassandra.db:type=HintedHandoffManager" via JMX but it takes an endpoint as argument.
If you have lots of nodes and several DCs, make sure to properly set hinted_handoff_throttle_in_kb and max_hints_delivery_threads.

Best,

Romain



Le Samedi 3 septembre 2016 2h59, jerome <je...@hotmail.com> a ?crit :


Hi Matija,

Thanks for your help! The downtime is minimal, usually less than five minutes. Since it is so short we're not so concerned about the node that's down missing data, we just want to make sure that before it goes down it replays all the hints it has so that there won't be any gaps in data on any other nodes for the hints it has while it's down.

Thanks,
Jerome
________________________________
From: Matija Gobec <ma...@gmail.com>
Sent: Friday, September 2, 2016 6:05:01 PM
To: user@cassandra.apache.org
Subject: Re: Is it possible to replay hints after running nodetool drain?

Hi Jerome,

The node being drained stops listening to requests but the other nodes being coordinators for given requests will store hints for that downed node for a configured period of time (max_hint_window_in_ms is 3 hours by default). If the downed node is back online in this time window it will receive hints from other nodes in the cluster and eventually catch up.
What is your typical maintenance downtime?

Regards,
Matija

On Fri, Sep 2, 2016 at 10:53 PM, jerome <je...@hotmail.com>> wrote:
Hello,

As part of routine maintenance for our cluster, my colleagues and I will run a nodetool drain before stopping a Cassandra node, performing maintenance, and bringing it back up. We run maintenance as a cron-job with a lock stored in a different cluster to ensure only node is ever down at a time. We would like to make sure the node has replayed all its hints before bringing it down to minimize the potential window in which users might read out-of-date data (we read at a consistency level of ONE). Is it possible to replay hints after performing a nodetool drain? The documentation leads me to believe its not since Cassandra will stop listening for connections from other nodes, but I was unable to find anything definitive either way. If a node won't replay hints after a nodetool drain, is there perhaps another way to tell Cassandra to stop listening for client connections but continue to replay hints to other nodes.

Thanks,
Jerome

Re: Is it possible to replay hints after running nodetool drain?

Posted by Romain Hardouin <ro...@yahoo.fr>.

Hi,
You don't have to worry about that unless you write with CL = ANY. The sole method to force hints that I know is to invoke scheduleHintDelivery on "org.apache.cassandra.db:type=HintedHandoffManager" via JMX but it takes an endpoint as argument. If you have lots of nodes and several DCs, make sure to properly set hinted_handoff_throttle_in_kb and max_hints_delivery_threads.
Best,
Romain
 

    Le Samedi 3 septembre 2016 2h59, jerome <je...@hotmail.com> a écrit :
 

 #yiv4261622774 #yiv4261622774 -- P {margin-top:0;margin-bottom:0;}#yiv4261622774 Hi Matija,
Thanks for your help! The downtime is minimal, usually less than five minutes. Since it is so short we're not so concerned about the node that's down missing data, we just want to make sure that before it goes down it replays all the hints it has so that there won't be any gaps in data on any other nodes for the hints it has while it's down.
Thanks,Jerome From: Matija Gobec <ma...@gmail.com>
Sent: Friday, September 2, 2016 6:05:01 PM
To: user@cassandra.apache.org
Subject: Re: Is it possible to replay hints after running nodetool drain? Hi Jerome,
The node being drained stops listening to requests but the other nodes being coordinators for given requests will store hints for that downed node for a configured period of time (max_hint_window_in_ms is 3 hours by default). If the downed node is back online in this time window it will receive hints from other nodes in the cluster and eventually catch up.What is your typical maintenance downtime?
Regards,Matija
On Fri, Sep 2, 2016 at 10:53 PM, jerome <je...@hotmail.com> wrote:

Hello,
As part of routine maintenance for our cluster, my colleagues and I will run a nodetool drain before stopping a Cassandra node, performing maintenance, and bringing it back up. We run maintenance as a cron-job with a lock stored in a different cluster to ensure only node is ever down at a time. We would like to make sure the node has replayed all its hints before bringing it down to minimize the potential window in which users might read out-of-date data (we read at a consistency level of ONE). Is it possible to replay hints after performing a nodetool drain? The documentation leads me to believe its not since Cassandra will stop listening for connections from other nodes, but I was unable to find anything definitive either way. If a node won't replay hints after a nodetool drain, is there perhaps another way to tell Cassandra to stop listening for client connections but continue to replay hints to other nodes.
Thanks,Jerome

Re: Is it possible to replay hints after running nodetool drain?

Posted by jerome <je...@hotmail.com>.

Hi Matija,


Thanks for your help! The downtime is minimal, usually less than five minutes. Since it is so short we're not so concerned about the node that's down missing data, we just want to make sure that before it goes down it replays all the hints it has so that there won't be any gaps in data on any other nodes for the hints it has while it's down.


Thanks,

Jerome

________________________________
From: Matija Gobec <ma...@gmail.com>
Sent: Friday, September 2, 2016 6:05:01 PM
To: user@cassandra.apache.org
Subject: Re: Is it possible to replay hints after running nodetool drain?

Hi Jerome,

The node being drained stops listening to requests but the other nodes being coordinators for given requests will store hints for that downed node for a configured period of time (max_hint_window_in_ms is 3 hours by default). If the downed node is back online in this time window it will receive hints from other nodes in the cluster and eventually catch up.
What is your typical maintenance downtime?

Regards,
Matija

On Fri, Sep 2, 2016 at 10:53 PM, jerome <je...@hotmail.com>> wrote:

Hello,


As part of routine maintenance for our cluster, my colleagues and I will run a nodetool drain before stopping a Cassandra node, performing maintenance, and bringing it back up. We run maintenance as a cron-job with a lock stored in a different cluster to ensure only node is ever down at a time. We would like to make sure the node has replayed all its hints before bringing it down to minimize the potential window in which users might read out-of-date data (we read at a consistency level of ONE). Is it possible to replay hints after performing a nodetool drain? The documentation leads me to believe its not since Cassandra will stop listening for connections from other nodes, but I was unable to find anything definitive either way. If a node won't replay hints after a nodetool drain, is there perhaps another way to tell Cassandra to stop listening for client connections but continue to replay hints to other nodes.


Thanks,

Jerome

Re: Is it possible to replay hints after running nodetool drain?

Posted by Matija Gobec <ma...@gmail.com>.

Hi Jerome,

The node being drained stops listening to requests but the other nodes
being coordinators for given requests will store hints for that downed node
for a configured period of time (max_hint_window_in_ms is 3 hours by
default). If the downed node is back online in this time window it will
receive hints from other nodes in the cluster and eventually catch up.
What is your typical maintenance downtime?

Regards,
Matija

On Fri, Sep 2, 2016 at 10:53 PM, jerome <je...@hotmail.com> wrote:

> Hello,
>
>
> As part of routine maintenance for our cluster, my colleagues and I
> will run a nodetool drain before stopping a Cassandra node, performing
> maintenance, and bringing it back up. We run maintenance as a cron-job with
> a lock stored in a different cluster to ensure only node is ever down at a
> time. We would like to make sure the node has replayed all its hints before
> bringing it down to minimize the potential window in which users might read
> out-of-date data (we read at a consistency level of ONE). Is it possible to
> replay hints after performing a nodetool drain? The documentation leads me
> to believe its not since Cassandra will stop listening for connections from
> other nodes, but I was unable to find anything definitive either way. If a
> node won't replay hints after a nodetool drain, is there perhaps another
> way to tell Cassandra to stop listening for client connections but continue
> to replay hints to other nodes.
>
>
> Thanks,
>
> Jerome
>