You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Paulo Ricardo Motta Gomes <pa...@chaordicsystems.com> on 2014/05/12 19:18:35 UTC
Disable reads during node rebuild
Hello,
I'm not able to replace a dead node using the ordinary procedure
(boostrap+join), and would like to rebuild the replacement node from
another DC. The problem is that if I start a node with auto_bootstrap=false
to perform the rebuild, it automatically starts serving empty reads
(CL=LOCAL_ONE).
Is there a way to disable reads from a node while performing rebuild from
another datacenter? I tried starting the node in write survery mode, but
the nodetool rebuild command does not work in this mode.
Thanks,
--
*Paulo Motta*
Chaordic | *Platform*
*www.chaordic.com.br <http://www.chaordic.com.br/>*
+55 48 3232.3200
Re: Disable reads during node rebuild
Posted by sankalp kohli <ko...@gmail.com>.
This might be useful
Nodetool command to disable
reads<https://issues.apache.org/jira/browse/CASSANDRA-6760>
On Wed, May 14, 2014 at 8:31 AM, Paulo Ricardo Motta Gomes <
paulo.motta@chaordicsystems.com> wrote:
> That's a nice workaround, will be really helpful in emergency situations
> like this.
>
> Thanks,
>
>
> On Mon, May 12, 2014 at 6:58 PM, Aaron Morton <aa...@thelastpickle.com>wrote:
>
>> I'm not able to replace a dead node using the ordinary procedure
>> (boostrap+join), and would like to rebuild the replacement node from
>> another DC.
>>
>> Normally when you want to add a new DC to the cluster the command to use
>> is nodetool rebuild $DC_NAME .(with auto_bootstrap: false) That will get
>> the node to stream data from the $DC_NAME
>>
>> The problem is that if I start a node with auto_bootstrap=false to
>> perform the rebuild, it automatically starts serving empty reads
>> (CL=LOCAL_ONE).
>>
>> When adding a new DC the nodes wont be processing reads, that is not
>> the case for you.
>>
>> You should disable the client API’s to prevent the clients from calling
>> the new nodes, use -Dcassandra.start_rpc=false and
>> -Dcassandra.start_native_transport=false in cassandra-env.sh or appropriate
>> settings in cassandra.yaml
>>
>> Disabling reads from other nodes will be harder. IIRC during bootstrap a
>> different timeout (based on ring_delay) is used to detect if the
>> bootstrapping node is down. However if the node is running and you use
>> nodetool rebuild i’m pretty sure the normal gossip failure detectors will
>> kick in. Which means you cannot disable gossip to prevent reads. Also we
>> would want the node to be up for writes.
>>
>> But what you can do is artificially set the severity of the node high so
>> the dynamic snitch will route around it. See
>> https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/locator/DynamicEndpointSnitchMBean.java#L37
>>
>>
>> * Set the value to something high on the node you will be rebuilding, the
>> number or cores on the system should do. (jmxterm is handy for this
>> http://wiki.cyclopsgroup.org/jmxterm)
>> * Check nodetool gossipinfo on the other nodes to see the SEVERITY app
>> state has propagated.
>> * Watch completed ReadStage tasks on the node you want to rebuild. If you
>> have read repair enabled it will still get some traffic.
>> * Do rebuild
>> * Reset severity to 0
>>
>> Hope that helps.
>> Aaron
>>
>> -----------------
>> Aaron Morton
>> New Zealand
>> @aaronmorton
>>
>> Co-Founder & Principal Consultant
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>> On 13/05/2014, at 5:18 am, Paulo Ricardo Motta Gomes <
>> paulo.motta@chaordicsystems.com> wrote:
>>
>> Hello,
>>
>> I'm not able to replace a dead node using the ordinary procedure
>> (boostrap+join), and would like to rebuild the replacement node from
>> another DC. The problem is that if I start a node with auto_bootstrap=false
>> to perform the rebuild, it automatically starts serving empty reads
>> (CL=LOCAL_ONE).
>>
>> Is there a way to disable reads from a node while performing rebuild from
>> another datacenter? I tried starting the node in write survery mode, but
>> the nodetool rebuild command does not work in this mode.
>>
>> Thanks,
>>
>> --
>> *Paulo Motta*
>>
>> Chaordic | *Platform*
>> *www.chaordic.com.br <http://www.chaordic.com.br/>*
>> +55 48 3232.3200
>>
>>
>>
>
>
> --
> *Paulo Motta*
>
> Chaordic | *Platform*
> *www.chaordic.com.br <http://www.chaordic.com.br/>*
> +55 48 3232.3200
>
Re: Disable reads during node rebuild
Posted by Paulo Ricardo Motta Gomes <pa...@chaordicsystems.com>.
That's a nice workaround, will be really helpful in emergency situations
like this.
Thanks,
On Mon, May 12, 2014 at 6:58 PM, Aaron Morton <aa...@thelastpickle.com>wrote:
> I'm not able to replace a dead node using the ordinary procedure
> (boostrap+join), and would like to rebuild the replacement node from
> another DC.
>
> Normally when you want to add a new DC to the cluster the command to use
> is nodetool rebuild $DC_NAME .(with auto_bootstrap: false) That will get
> the node to stream data from the $DC_NAME
>
> The problem is that if I start a node with auto_bootstrap=false to perform
> the rebuild, it automatically starts serving empty reads (CL=LOCAL_ONE).
>
> When adding a new DC the nodes wont be processing reads, that is not the
> case for you.
>
> You should disable the client API’s to prevent the clients from calling
> the new nodes, use -Dcassandra.start_rpc=false and
> -Dcassandra.start_native_transport=false in cassandra-env.sh or appropriate
> settings in cassandra.yaml
>
> Disabling reads from other nodes will be harder. IIRC during bootstrap a
> different timeout (based on ring_delay) is used to detect if the
> bootstrapping node is down. However if the node is running and you use
> nodetool rebuild i’m pretty sure the normal gossip failure detectors will
> kick in. Which means you cannot disable gossip to prevent reads. Also we
> would want the node to be up for writes.
>
> But what you can do is artificially set the severity of the node high so
> the dynamic snitch will route around it. See
> https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/locator/DynamicEndpointSnitchMBean.java#L37
>
>
> * Set the value to something high on the node you will be rebuilding, the
> number or cores on the system should do. (jmxterm is handy for this
> http://wiki.cyclopsgroup.org/jmxterm)
> * Check nodetool gossipinfo on the other nodes to see the SEVERITY app
> state has propagated.
> * Watch completed ReadStage tasks on the node you want to rebuild. If you
> have read repair enabled it will still get some traffic.
> * Do rebuild
> * Reset severity to 0
>
> Hope that helps.
> Aaron
>
> -----------------
> Aaron Morton
> New Zealand
> @aaronmorton
>
> Co-Founder & Principal Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> On 13/05/2014, at 5:18 am, Paulo Ricardo Motta Gomes <
> paulo.motta@chaordicsystems.com> wrote:
>
> Hello,
>
> I'm not able to replace a dead node using the ordinary procedure
> (boostrap+join), and would like to rebuild the replacement node from
> another DC. The problem is that if I start a node with auto_bootstrap=false
> to perform the rebuild, it automatically starts serving empty reads
> (CL=LOCAL_ONE).
>
> Is there a way to disable reads from a node while performing rebuild from
> another datacenter? I tried starting the node in write survery mode, but
> the nodetool rebuild command does not work in this mode.
>
> Thanks,
>
> --
> *Paulo Motta*
>
> Chaordic | *Platform*
> *www.chaordic.com.br <http://www.chaordic.com.br/>*
> +55 48 3232.3200
>
>
>
--
*Paulo Motta*
Chaordic | *Platform*
*www.chaordic.com.br <http://www.chaordic.com.br/>*
+55 48 3232.3200
Re: Disable reads during node rebuild
Posted by Aaron Morton <aa...@thelastpickle.com>.
> I'm not able to replace a dead node using the ordinary procedure (boostrap+join), and would like to rebuild the replacement node from another DC.
Normally when you want to add a new DC to the cluster the command to use is nodetool rebuild $DC_NAME .(with auto_bootstrap: false) That will get the node to stream data from the $DC_NAME
> The problem is that if I start a node with auto_bootstrap=false to perform the rebuild, it automatically starts serving empty reads (CL=LOCAL_ONE).
When adding a new DC the nodes wont be processing reads, that is not the case for you.
You should disable the client API’s to prevent the clients from calling the new nodes, use -Dcassandra.start_rpc=false and -Dcassandra.start_native_transport=false in cassandra-env.sh or appropriate settings in cassandra.yaml
Disabling reads from other nodes will be harder. IIRC during bootstrap a different timeout (based on ring_delay) is used to detect if the bootstrapping node is down. However if the node is running and you use nodetool rebuild i’m pretty sure the normal gossip failure detectors will kick in. Which means you cannot disable gossip to prevent reads. Also we would want the node to be up for writes.
But what you can do is artificially set the severity of the node high so the dynamic snitch will route around it. See https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/locator/DynamicEndpointSnitchMBean.java#L37
* Set the value to something high on the node you will be rebuilding, the number or cores on the system should do. (jmxterm is handy for this http://wiki.cyclopsgroup.org/jmxterm)
* Check nodetool gossipinfo on the other nodes to see the SEVERITY app state has propagated.
* Watch completed ReadStage tasks on the node you want to rebuild. If you have read repair enabled it will still get some traffic.
* Do rebuild
* Reset severity to 0
Hope that helps.
Aaron
-----------------
Aaron Morton
New Zealand
@aaronmorton
Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 13/05/2014, at 5:18 am, Paulo Ricardo Motta Gomes <pa...@chaordicsystems.com> wrote:
> Hello,
>
> I'm not able to replace a dead node using the ordinary procedure (boostrap+join), and would like to rebuild the replacement node from another DC. The problem is that if I start a node with auto_bootstrap=false to perform the rebuild, it automatically starts serving empty reads (CL=LOCAL_ONE).
>
> Is there a way to disable reads from a node while performing rebuild from another datacenter? I tried starting the node in write survery mode, but the nodetool rebuild command does not work in this mode.
>
> Thanks,
>
> --
> Paulo Motta
>
> Chaordic | Platform
> www.chaordic.com.br
> +55 48 3232.3200
Re: Disable reads during node rebuild
Posted by Paulo Ricardo Motta Gomes <pa...@chaordicsystems.com>.
That'll be really useful, thanks!!
On Wed, May 14, 2014 at 7:47 PM, Aaron Morton <aa...@thelastpickle.com>wrote:
> As of 2.0.7, driftx has added this long-requested feature.
>
> Thanks
>
> A
> -----------------
> Aaron Morton
> New Zealand
> @aaronmorton
>
> Co-Founder & Principal Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> On 13/05/2014, at 9:36 am, Robert Coli <rc...@eventbrite.com> wrote:
>
> On Mon, May 12, 2014 at 10:18 AM, Paulo Ricardo Motta Gomes <
> paulo.motta@chaordicsystems.com> wrote:
>
>> Is there a way to disable reads from a node while performing rebuild from
>> another datacenter? I tried starting the node in write survery mode, but
>> the nodetool rebuild command does not work in this mode.
>>
>
> As of 2.0.7, driftx has added this long-requested feature.
>
> https://issues.apache.org/jira/browse/CASSANDRA-6961
>
> Note that it is impossible to completely close the race window here as
> long as writes are incoming, this functionality just dramatically shortens
> it.
>
> =Rob
>
>
>
>
--
*Paulo Motta*
Chaordic | *Platform*
*www.chaordic.com.br <http://www.chaordic.com.br/>*
+55 48 3232.3200
Re: Disable reads during node rebuild
Posted by Aaron Morton <aa...@thelastpickle.com>.
> As of 2.0.7, driftx has added this long-requested feature.
Thanks
A
-----------------
Aaron Morton
New Zealand
@aaronmorton
Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 13/05/2014, at 9:36 am, Robert Coli <rc...@eventbrite.com> wrote:
> On Mon, May 12, 2014 at 10:18 AM, Paulo Ricardo Motta Gomes <pa...@chaordicsystems.com> wrote:
> Is there a way to disable reads from a node while performing rebuild from another datacenter? I tried starting the node in write survery mode, but the nodetool rebuild command does not work in this mode.
>
> As of 2.0.7, driftx has added this long-requested feature.
>
> https://issues.apache.org/jira/browse/CASSANDRA-6961
>
> Note that it is impossible to completely close the race window here as long as writes are incoming, this functionality just dramatically shortens it.
>
> =Rob
>
Re: Disable reads during node rebuild
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, May 12, 2014 at 10:18 AM, Paulo Ricardo Motta Gomes <
paulo.motta@chaordicsystems.com> wrote:
> Is there a way to disable reads from a node while performing rebuild from
> another datacenter? I tried starting the node in write survery mode, but
> the nodetool rebuild command does not work in this mode.
>
As of 2.0.7, driftx has added this long-requested feature.
https://issues.apache.org/jira/browse/CASSANDRA-6961
Note that it is impossible to completely close the race window here as long
as writes are incoming, this functionality just dramatically shortens it.
=Rob