[Cassandra] Replacing a cassandra node

[Emalayan Vairavanathan <sv...@yahoo.com>]

Hi All,

I have a question.

In the case where replace a cassandra node (call it node A) with 
another one that has the exact same IP (ie. during a node failure), what exactly should we do?  Currently I understand that we should at least 
run "nodetool repair".  I noticed that the cassandra system logs of other nodes detect that node A has its host uuid change 
once you have re-installed it and recreated its data directory. I am not sure however if this includes updating all the hinted handoffs. If it 
did, I would see no need to run "nodetool cleanup" but am uncertain exactly.  Your help is much appreciated.

Thank you
Emalayan

Re: [Cassandra] Replacing a cassandra node

[Eric Stevens <mi...@gmail.com>]

Well I guess I'm glad to know I didn't just miss something obvious.  I
voted for the issue on Jira; I don't know if that's used to prioritize
features or not, but it can't hurt, right? =)


On Wed, Jun 26, 2013 at 3:03 PM, Robert Coli <rc...@eventbrite.com> wrote:

> On Fri, Jun 21, 2013 at 5:25 AM, Eric Stevens <mi...@gmail.com> wrote:
> > Is there a way to replace a failed server using vnodes?  I only had
> occasion
> > to do this once, on a relatively small cluster.  ...
> > Of course that caused a bunch of key reassignments,
> > so I'm sure it would be less work for the cluster if I could bring a new
> > server online with the same vnodes as the failed server.
>
> Currently, you can supply a comma separated list of tokens to
> replace_token.
>
> In the future, this :
>
> https://issues.apache.org/jira/browse/CASSANDRA-5337 : vnode-aware
> replacenode command
>
> It's unclear to me in what way you should express your support for
> this ticket in order for the project to be aware that you, as an
> operator, would like this capability.
>
> =Rob
>

Re: [Cassandra] Replacing a cassandra node

[Robert Coli <rc...@eventbrite.com>]

On Fri, Jun 21, 2013 at 5:25 AM, Eric Stevens <mi...@gmail.com> wrote:
> Is there a way to replace a failed server using vnodes?  I only had occasion
> to do this once, on a relatively small cluster.  ...
> Of course that caused a bunch of key reassignments,
> so I'm sure it would be less work for the cluster if I could bring a new
> server online with the same vnodes as the failed server.

Currently, you can supply a comma separated list of tokens to replace_token.

In the future, this :

https://issues.apache.org/jira/browse/CASSANDRA-5337 : vnode-aware
replacenode command

It's unclear to me in what way you should express your support for
this ticket in order for the project to be aware that you, as an
operator, would like this capability.

=Rob

Re: [Cassandra] Replacing a cassandra node

[Eric Stevens <mi...@gmail.com>]

Is there a way to replace a failed server using vnodes?  I only had
occasion to do this once, on a relatively small cluster.  At the time I
just needed to get the new server online and wasn't concerned about the
performance implications, so I just removed the failed server from the
cluster and bootstrapped a new one.  Of course that caused a bunch of key
reassignments, so I'm sure it would be less work for the cluster if I could
bring a new server online with the same vnodes as the failed server.


On Thu, Jun 20, 2013 at 2:40 PM, Robert Coli <rc...@eventbrite.com> wrote:

> On Thu, Jun 20, 2013 at 10:40 AM, Emalayan Vairavanathan
> <sv...@yahoo.com> wrote:
> > In the case where replace a cassandra node (call it node A) with another
> one
> > that has the exact same IP (ie. during a node failure), what exactly
> should
> > we do?  Currently I understand that we should at least run "nodetool
> > repair".
>
> If you lost the data from the node, then what you want is "replace_token."
>
> If you didn't lose the data from the node (and can tolerate stale
> reads until the repair completes) you want to start the node with
> auto_bootstrap set to false and then repair.
>
> =Rob
>

Re: [Cassandra] Replacing a cassandra node

[Robert Coli <rc...@eventbrite.com>]

On Thu, Jun 20, 2013 at 10:40 AM, Emalayan Vairavanathan
<sv...@yahoo.com> wrote:
> In the case where replace a cassandra node (call it node A) with another one
> that has the exact same IP (ie. during a node failure), what exactly should
> we do?  Currently I understand that we should at least run "nodetool
> repair".

If you lost the data from the node, then what you want is "replace_token."

If you didn't lose the data from the node (and can tolerate stale
reads until the repair completes) you want to start the node with
auto_bootstrap set to false and then repair.

=Rob