You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Hiren Shah <hs...@dotomi.com> on 2011/09/21 19:29:08 UTC
Cassandra reconfiguration
Hi,
We have cassandra in production in three data centers with five nodes each. We are getting rid of one of the data centers, so I want to reconfig my cluster to two data centers, seven nodes each. I first need to shut down DC3. I have already shut down traffic to DC3 and am running repairs. My priority #1 is to maintain the stability of overall cluster (the two live data centers). Some data loss is tolerable, but it will be ideal if I can avoid that.
What is the best way to decommission entire data center? Should I issue decommission to the nodes one by one? Will that put additional load on remaining nodes in DC3? Or should I just shut them down and then run removetoken for these nodes? Or is there a better way?
Thanks in advance,
Hiren Shah
Re: Cassandra reconfiguration
Posted by aaron morton <aa...@thelastpickle.com>.
> Our RF is 2
Are you using the SimpleStrategy or the NetworkTopologyStrategy ? I assumed NTS in a multi DC setup.
I think I sent before my brain caught up before.
I was thinking that if you had an NTS setup like [ {DC1 : 3}, {DC2: 3}, {DC3:3}] you would change it to
[ {DC1 :3}, {DC2:3}, {DC3 :0}] and then decommission the nodes in DC3.
I dont want to say to much more without knowing how the data is distributed, the token assignments and the placement strategy.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 23/09/2011, at 5:46 AM, Hiren Shah wrote:
> I just started decommissioning one of the nodes. It is streaming data to other nodes in the data center, though, which seems waste of time because all the nodes will be going away. Wouldn’t it be better to just bring cassandra down on these nodes? And then run removetoken or change the topology and restart nodes on other two datacenters?
>
> Does it make sense to issue decommission to other nodes before one is finished? That should also avoid the unnecessary streaming in theory.
>
> Thanks,
> Hiren
>
> From: Hiren Shah [mailto:hshah@dotomi.com]
> Sent: Thursday, September 22, 2011 8:18 AM
> To: user@cassandra.apache.org
> Subject: RE: Cassandra reconfiguration
>
> Our RF is 2 and we are planning to keep that. Thanks for addressing the decommissioning puzzle. That is where I could not find any doc.
>
> Hiren
>
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: Wednesday, September 21, 2011 6:01 PM
> To: user@cassandra.apache.org
> Subject: Re: Cassandra reconfiguration
>
> What are the token assignments and what is the RF ?
>
> Without knowing the details I would guess…
>
> Make the RF changes you want for DC 1 and 2 and repair.
> decomission the nodes in DC3 one at a time.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 22/09/2011, at 5:29 AM, Hiren Shah wrote:
>
>
> Hi,
>
> We have cassandra in production in three data centers with five nodes each. We are getting rid of one of the data centers, so I want to reconfig my cluster to two data centers, seven nodes each. I first need to shut down DC3. I have already shut down traffic to DC3 and am running repairs. My priority #1 is to maintain the stability of overall cluster (the two live data centers). Some data loss is tolerable, but it will be ideal if I can avoid that.
>
> What is the best way to decommission entire data center? Should I issue decommission to the nodes one by one? Will that put additional load on remaining nodes in DC3? Or should I just shut them down and then run removetoken for these nodes? Or is there a better way?
>
> Thanks in advance,
> Hiren Shah
>
RE: Cassandra reconfiguration
Posted by Hiren Shah <hs...@dotomi.com>.
I just started decommissioning one of the nodes. It is streaming data to other nodes in the data center, though, which seems waste of time because all the nodes will be going away. Wouldn't it be better to just bring cassandra down on these nodes? And then run removetoken or change the topology and restart nodes on other two datacenters?
Does it make sense to issue decommission to other nodes before one is finished? That should also avoid the unnecessary streaming in theory.
Thanks,
Hiren
From: Hiren Shah [mailto:hshah@dotomi.com]
Sent: Thursday, September 22, 2011 8:18 AM
To: user@cassandra.apache.org
Subject: RE: Cassandra reconfiguration
Our RF is 2 and we are planning to keep that. Thanks for addressing the decommissioning puzzle. That is where I could not find any doc.
Hiren
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Wednesday, September 21, 2011 6:01 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra reconfiguration
What are the token assignments and what is the RF ?
Without knowing the details I would guess...
Make the RF changes you want for DC 1 and 2 and repair.
decomission the nodes in DC3 one at a time.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 22/09/2011, at 5:29 AM, Hiren Shah wrote:
Hi,
We have cassandra in production in three data centers with five nodes each. We are getting rid of one of the data centers, so I want to reconfig my cluster to two data centers, seven nodes each. I first need to shut down DC3. I have already shut down traffic to DC3 and am running repairs. My priority #1 is to maintain the stability of overall cluster (the two live data centers). Some data loss is tolerable, but it will be ideal if I can avoid that.
What is the best way to decommission entire data center? Should I issue decommission to the nodes one by one? Will that put additional load on remaining nodes in DC3? Or should I just shut them down and then run removetoken for these nodes? Or is there a better way?
Thanks in advance,
Hiren Shah
RE: Cassandra reconfiguration
Posted by Hiren Shah <hs...@dotomi.com>.
Our RF is 2 and we are planning to keep that. Thanks for addressing the decommissioning puzzle. That is where I could not find any doc.
Hiren
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Wednesday, September 21, 2011 6:01 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra reconfiguration
What are the token assignments and what is the RF ?
Without knowing the details I would guess...
Make the RF changes you want for DC 1 and 2 and repair.
decomission the nodes in DC3 one at a time.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 22/09/2011, at 5:29 AM, Hiren Shah wrote:
Hi,
We have cassandra in production in three data centers with five nodes each. We are getting rid of one of the data centers, so I want to reconfig my cluster to two data centers, seven nodes each. I first need to shut down DC3. I have already shut down traffic to DC3 and am running repairs. My priority #1 is to maintain the stability of overall cluster (the two live data centers). Some data loss is tolerable, but it will be ideal if I can avoid that.
What is the best way to decommission entire data center? Should I issue decommission to the nodes one by one? Will that put additional load on remaining nodes in DC3? Or should I just shut them down and then run removetoken for these nodes? Or is there a better way?
Thanks in advance,
Hiren Shah
Re: Cassandra reconfiguration
Posted by aaron morton <aa...@thelastpickle.com>.
What are the token assignments and what is the RF ?
Without knowing the details I would guess…
Make the RF changes you want for DC 1 and 2 and repair.
decomission the nodes in DC3 one at a time.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 22/09/2011, at 5:29 AM, Hiren Shah wrote:
> Hi,
>
> We have cassandra in production in three data centers with five nodes each. We are getting rid of one of the data centers, so I want to reconfig my cluster to two data centers, seven nodes each. I first need to shut down DC3. I have already shut down traffic to DC3 and am running repairs. My priority #1 is to maintain the stability of overall cluster (the two live data centers). Some data loss is tolerable, but it will be ideal if I can avoid that.
>
> What is the best way to decommission entire data center? Should I issue decommission to the nodes one by one? Will that put additional load on remaining nodes in DC3? Or should I just shut them down and then run removetoken for these nodes? Or is there a better way?
>
> Thanks in advance,
> Hiren Shah
>