You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by maneela a <ma...@yahoo.com> on 2011/04/27 09:07:05 UTC
Expanding single node to 2 node cluster
Hi,
I had a 2 node cassandra cluster with replication factor 2 and OrderPreservingPartitioner but we did not provide InitialToken in the configuration files. One of the node was affected in the recent AWS EBS outage and had been partitioned from cluster. However, I continued to allowed all write operations to other survived node because I thought AWS could recovered EBS issues with in 24 hours so Survived node might take care of propagating 2nd replica from its hinted column family to the bad node when it recovered from EBS issues. Unfortunately AWS had taken longer than we expected almost 4 days. So instead of recovering 2nd node by playing hinted CF from node1, I did the following sequence of events in order to get 2nd node back to cluster
1) shut down cassandra service on good node2) removed all hinted CF files3) Taken EBS snapshot4) Launched new EBS volumes from above snapshot and mounted them on 2nd node5) Also copied commitlogs from node1 to node2in other words, I cloned node1 and mounted on node2, my assumption is cluster with 2 nodes with replication 2 should likely be mirrored images
6) brought up service on both nodes7) I am not seeing both IP address as part of ring when I ran nodetool command
root@domU-12-31-39-0F-CA-61:/mnt/logs/cassandra# nodetool -h localhost ringAddress Status Load Range Ring10.193.201.139Up 434.77 GB RVtMj8gWiKG0baPy |<--|
root@ip-10-196-107-47:/data/cassandra/data/system# nodetool -h localhost ringAddress Status Load Range Ring10.193.201.139Up 434.77 GB RVtMj8gWiKG0baPy |<--|
I guess this behavior is happening because both nodes are having same data including Locationinfo CF and commit logs as well. Can someone direct me what should be done here to get both IPs as part of ring?
Thanksniru
Re: Expanding single node to 2 node cluster
Posted by Aaron Morton <aa...@thelastpickle.com>.
You could try...
- delete / move the system data directory
- set the initial_token for each node to what they were before
- restart and recreate the schema
- run repair and then clean
It would have been a good idea to drain the nodes, this would checkpoint the logs and clear them.
If you do not know the initial tokens, I would start a new empty node as suggested and do the the same.
Hope that helps.
Aaron
On 27/04/2011, at 7:07 PM, maneela a <ma...@yahoo.com> wrote:
> Hi,
>
> I had a 2 node cassandra cluster with replication factor 2 and OrderPreservingPartitioner but we did not provide InitialToken in the configuration files. One of the node was affected in the recent AWS EBS outage and had been partitioned from cluster. However, I continued to allowed all write operations to other survived node because I thought AWS could recovered EBS issues with in 24 hours so Survived node might take care of propagating 2nd replica from its hinted column family to the bad node when it recovered from EBS issues. Unfortunately AWS had taken longer than we expected almost 4 days. So instead of recovering 2nd node by playing hinted CF from node1, I did the following sequence of events in order to get 2nd node back to cluster
>
> 1) shut down cassandra service on good node
> 2) removed all hinted CF files
> 3) Taken EBS snapshot
> 4) Launched new EBS volumes from above snapshot and mounted them on 2nd node
> 5) Also copied commitlogs from node1 to node2
> in other words, I cloned node1 and mounted on node2, my assumption is cluster with 2 nodes with replication 2 should likely be mirrored images
>
> 6) brought up service on both nodes
> 7) I am not seeing both IP address as part of ring when I ran nodetool command
>
> root@domU-12-31-39-0F-CA-61:/mnt/logs/cassandra# nodetool -h localhost ring
> Address Status Load Range Ring
> 10.193.201.139Up 434.77 GB RVtMj8gWiKG0baPy |<--|
>
> root@ip-10-196-107-47:/data/cassandra/data/system# nodetool -h localhost ring
> Address Status Load Range Ring
> 10.193.201.139Up 434.77 GB RVtMj8gWiKG0baPy |<--|
>
>
> I guess this behavior is happening because both nodes are having same data including Locationinfo CF and commit logs as well. Can someone direct me what should be done here to get both IPs as part of ring?
>
> Thanks
> niru
>
Re: Expanding single node to 2 node cluster
Posted by Maki Watanabe <wa...@gmail.com>.
Why don't you just add new node to ring and removetoken of the bad one?
2011/4/27 maneela a <ma...@yahoo.com>
>
> Hi,
> I had a 2 node cassandra cluster with replication factor 2 and OrderPreservingPartitioner but we did not provide InitialToken in the configuration files. One of the node was affected in the recent AWS EBS outage and had been partitioned from cluster. However, I continued to allowed all write operations to other survived node because I thought AWS could recovered EBS issues with in 24 hours so Survived node might take care of propagating 2nd replica from its hinted column family to the bad node when it recovered from EBS issues. Unfortunately AWS had taken longer than we expected almost 4 days. So instead of recovering 2nd node by playing hinted CF from node1, I did the following sequence of events in order to get 2nd node back to cluster
> 1) shut down cassandra service on good node
> 2) removed all hinted CF files
> 3) Taken EBS snapshot
> 4) Launched new EBS volumes from above snapshot and mounted them on 2nd node
> 5) Also copied commitlogs from node1 to node2
> in other words, I cloned node1 and mounted on node2, my assumption is cluster with 2 nodes with replication 2 should likely be mirrored images
> 6) brought up service on both nodes
> 7) I am not seeing both IP address as part of ring when I ran nodetool command
> root@domU-12-31-39-0F-CA-61:/mnt/logs/cassandra# nodetool -h localhost ring
> Address Status Load Range Ring
> 10.193.201.139Up 434.77 GB RVtMj8gWiKG0baPy |<--|
> root@ip-10-196-107-47:/data/cassandra/data/system# nodetool -h localhost ring
> Address Status Load Range Ring
> 10.193.201.139Up 434.77 GB RVtMj8gWiKG0baPy |<--|
>
> I guess this behavior is happening because both nodes are having same data including Locationinfo CF and commit logs as well. Can someone direct me what should be done here to get both IPs as part of ring?
> Thanks
> niru
>
--
w3m