[jira] [Comment Edited] (CASSANDRA-8436) Replacing a dead node by deleting it and bootstrapping a new one

["Omri Bahumi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14237663#comment-14237663 ] 

Omri Bahumi edited comment on CASSANDRA-8436 at 12/8/14 9:24 AM:
-----------------------------------------------------------------

[~brandon.williams] I'm afraid you misunderstood the issue.
Although being worded as a question It's actually a feature request/discussion.

We're using AWS EC2 combined with Autoscaling Groups for launching our Cassandra cluster with a predefined configuration.
All our nodes are configured with "auto_bootstrap: true" so we'll be able to resize the Cassandra cluster and new nodes will simply join the ring (as http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html suggests it should be done).

The problem is with replacing a node that died. When a node dies on EC2 (being a part of an Autoscaling group), ASG automatically launches a new one.

I would like to automate the process of dead node replacement as well.

The "-Dcassandra.replace_address=..." method isn't really Ops friendly. (CASSANDRA-7356 improves it somehow, but IMHO the process of replacing a dead node should be automated somehow)

I was thinking of automating the node removal when a node dies.
If a dead node has been deleted and a new one is bootstrapped, will the results be the same as if I was running "-Dcassandra.replace_address=..." on the replacement node? (as http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_node_t.html suggests it should be done)
I guess that it's not the same, so the feature I'm requesting is a way to automate dead node replacement.


was (Author: omribahumi):
[~brandon.williams] I'm afraid you misunderstood the issue.
Although being worded as a question It's actually a feature request/discussion.

We're using AWS EC2 combined with Autoscaling Groups for launching our Cassandra cluster with a predefined configuration.
All our nodes are configured with "auto_bootstrap: true" so we'll be able to resize the Cassandra cluster and new nodes will simply join the ring (as http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html suggests it should be done).

The problem is with replacing a node that died. When a node dies on EC2 (being a part of an Autoscaling group), ASG automatically launches a new one.

I would like to automate the process of dead node replacement as well.

The "-Dcassandra.replace_address=..." method isn't really Ops friendly. (CASSANDRA-7356 improves it somehow, but IMHO the process of extracting the previous node's IP address should be automated somehow)

I was thinking of automating the node removal when a node dies.
If a dead node has been deleted and a new one is bootstrapped, will the results be the same as if I was running "-Dcassandra.replace_address=..." on the replacement node? (as http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_replace_node_t.html suggests it should be done)
I guess that it's not the same, so the feature I'm requesting is a way to automate dead node replacement.

> Replacing a dead node by deleting it and bootstrapping a new one
> ----------------------------------------------------------------
>
>                 Key: CASSANDRA-8436
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8436
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Omri Bahumi
>
> I brought this subject up in the mailing list, now I'm bringing it up in here.
> I'm trying to automate our Cassandra infrastructure. We're using an Autoscaling Group for keeping the Cassandra instances alive.
> After the initial cluster creation, nodes are launched with auto_bootstrap enabled.
> I was thinking to automate the process of node deletion (when a node terminates) and have the new launched node replace it.
> Reading the documentation, replacing a dead node should be done with "-Dcassandra.replace_address=<ip-address>".
> Is deleting the node and bootstrapping a new one a feasible solution?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)