You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Bryce Godfrey <Br...@azaleos.com> on 2012/08/22 22:43:44 UTC
Expanding cluster to include a new DR datacenter
We are in the process of building out a new DR system in another Data Center, and we want to mirror our Cassandra environment to that DR. I have a couple questions on the best way to do this after reading the documentation on the Datastax website. We didn't initially plan for this to be a DR setup when first deployed a while ago due to budgeting, but now we need to. So I'm just trying to nail down the order of doing this as well as any potential issues.
For the nodes, we don't plan on querying the servers in this DR until we fail over to this data center. We are going to have 5 similar nodes in the DR, should I join them into the ring at token+1?
All keyspaces are set to the replication strategy of SimpleStrategy. Can I change the replication strategy after joining the new nodes in the DR to NetworkTopologyStategy with the updated replication factor for each dr?
Lastly, is changing snitch from default of SimpleSnitch to RackInferringSnitch going to cause any issues? Since its in the Cassandra.yaml file I assume a rolling restart to pick up the value would be ok?
This is all on Cassandra 1.1.4, Thanks for any help!
Re: Expanding cluster to include a new DR datacenter
Posted by Tyler Hobbs <ty...@datastax.com>.
I believe the issue should be resolved by this ticket, which should get
into 1.1.5: https://issues.apache.org/jira/browse/CASSANDRA-4561
On Wed, Sep 5, 2012 at 12:33 PM, Bryce Godfrey <Br...@azaleos.com>wrote:
> I just wanted to raise this back up again, see if anyone can provide
> some help. Thanks.****
>
> ** **
>
> *From:* Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
> *Sent:* Friday, August 31, 2012 11:47 AM
>
> *To:* user@cassandra.apache.org
> *Subject:* RE: Expanding cluster to include a new DR datacenter****
>
> ** **
>
> Here is the log around the drop keyspace with debugging enabled.****
>
> ** **
>
> *From:* aaron morton [mailto:aaron@thelastpickle.com<aa...@thelastpickle.com>]
>
> *Sent:* Wednesday, August 29, 2012 10:22 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
> ** **
>
> Can you provide the server log ? ****
>
> ** **
>
> If you can turn the log level up to DEBUG that would be handy as well. ***
> *
>
> ** **
>
> Cheers****
>
> ** **
>
> -----------------****
>
> Aaron Morton****
>
> Freelance Developer****
>
> @aaronmorton****
>
> http://www.thelastpickle.com****
>
> ** **
>
> On 30/08/2012, at 8:22 AM, Bryce Godfrey <Br...@azaleos.com>
> wrote:****
>
> ** **
>
> Well I tried to drop the keyspace, but it’s still there. No errors in
> logs and Cassandra-cli showed the schema agreement after the command. I
> took a snapshot of the system keyspace first. Nothing is crashing in the
> clients yet either, still able to read/write to that keyspace.****
>
> ****
>
> [default@EBonding] drop keyspace EBonding;****
>
> 2eb11095-b8a8-31cd-80c3-c748d32a4208****
>
> Waiting for schema agreement...****
>
> ... schemas agree across the cluster****
>
> ****
>
> [default@unknown] use EBonding;****
>
> Authenticated to keyspace: EBonding****
>
> [default@EBonding] describe;****
>
> Keyspace: EBonding:****
>
> Replication Strategy: org.apache.cassandra.locator.SimpleStrategy****
>
> Durable Writes: true****
>
> Options: [replication_factor:2]****
>
> ****
>
> *From:* aaron morton [mailto:aaron@thelastpickle.com]
> *Sent:* Wednesday, August 29, 2012 2:36 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
> ****
>
> It would be handy to work out what the corruption is. Could you snapshot
> the system keyspace and store it somewhere, just incase we can look at it
> later ?****
>
> ****
>
> Is there a way I can confirm this ****
>
> Errors in the client and/or the server log is the the traditional way. **
> **
>
> ****
>
> go about cleaning up/restoring the proper schema?****
>
> If you need to get it back, and can handle the down time, the simple
> thing is drop the KS and re-create it. Remember to take a snapshot first.
> Drop keyspace takes one but it's the sort of thing I would do myself.****
>
> ****
>
> Or you can try _try_ nodetool resetlocalschema. Without knowing what the
> error is it's hard to say if it would work. ****
>
> ****
>
> Cheers****
>
> ****
>
> ****
>
> -----------------****
>
> Aaron Morton****
>
> Freelance Developer****
>
> @aaronmorton****
>
> http://www.thelastpickle.com****
>
> ****
>
> On 29/08/2012, at 9:10 AM, Bryce Godfrey <Br...@azaleos.com>
> wrote:****
>
>
>
> ****
>
> I believe what may be really going on is that my schema is in a bad or
> corrupt state. I also have one keyspace that I just cannot drop an
> existing column family from even though it shows no errors.****
>
> ****
>
> So right now I was able to get 4 of my 6 keyspaces over to Network
> Topology strategy.****
>
> ****
>
> I think I got into this bad state after pointing Opscenter at this cluster
> for the first time, as it started throwing errors after that and crashed a
> couple of my nodes until I stopped it and its agents.****
>
> ****
>
> Is there a way I can confirm this or go about cleaning up/restoring the
> proper schema?****
>
> ****
>
> *From:* Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
> *Sent:* Tuesday, August 28, 2012 11:09 AM
> *To:* user@cassandra.apache.org
> *Subject:* RE: Expanding cluster to include a new DR datacenter****
>
> ****
>
> So in an interesting turn of events, this works on my other 4 keyspaces
> but just not this ‘EBonding’ one which will not recognize the changes. I
> can probably get around this by dropping and re-creating this keyspace
> since its uptime is not too important for us.****
>
> ****
>
> [default@AlertStats] describe AlertStats;****
>
> Keyspace: AlertStats:****
>
> Replication Strategy:
> org.apache.cassandra.locator.NetworkTopologyStrategy****
>
> Durable Writes: true****
>
> Options: [Fisher:3]****
>
> ****
>
> *From:* Mohit Anchlia [mailto:mohitanchlia@gmail.com<mo...@gmail.com>
> ]
> *Sent:* Monday, August 27, 2012 3:50 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
> ****
>
> Can you describe your schema again with TierPoint in it?****
>
> On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey <Br...@azaleos.com>
> wrote:****
>
> Same results. I restarted the node also to see if it just wasn’t
> picking up the changes and it still shows Simple. ****
>
> ****
>
> When I specify the DC for strategy_options I should be using the DC name
> from properfy file snitch right? Ours is “Fisher” and “TierPoint” so
> that’s what I used.****
>
> ****
>
> *From:* Mohit Anchlia [mailto:mohitanchlia@gmail.com]
> *Sent:* Monday, August 27, 2012 1:21 PM****
>
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
> ****
>
> In your update command is it possible to specify RF for both DC? You could
> just do DC1:2, DC2:0.****
>
> On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey <Br...@azaleos.com>
> wrote:****
>
> Show schema output show the simple strategy still****
>
> [default@unknown] show schema EBonding;****
>
> create keyspace EBonding****
>
> with placement_strategy = 'SimpleStrategy'****
>
> and strategy_options = {replication_factor : 2}****
>
> and durable_writes = true;****
>
> ****
>
> This is the only thing I see in the system log at the time on all the
> nodes:****
>
> ****
>
> INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java
> (line 659) Enqueuing flush of Memtable-schema_keyspaces@1157216346(183/228
> serialized/live bytes, 4 ops)****
>
> INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264)
> Writing Memtable-schema_keyspaces@1157216346(183/228 serialized/live
> bytes, 4 ops)****
>
> INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305)
> Completed flushing
> /opt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-he-34817-Data.db
> (241 bytes) for commitlog p$****
>
> ****
>
> ****
>
> Should I turn the logging level up on something to see some more info
> maybe?****
>
> ****
>
> *From:* aaron morton [mailto:aaron@thelastpickle.com]
> *Sent:* Monday, August 27, 2012 1:35 AM****
>
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
> ****
>
> I did a quick test on a clean 1.1.4 and it worked ****
>
> ****
>
> Can you check the logs for errors ? Can you see your schema change in
> there ?****
>
> ****
>
> Also what is the output from show schema; in the cli ? ****
>
> ****
>
> Cheers****
>
> ****
>
> -----------------****
>
> Aaron Morton****
>
> Freelance Developer****
>
> @aaronmorton****
>
> http://www.thelastpickle.com****
>
> ****
>
> On 25/08/2012, at 6:53 PM, Bryce Godfrey <Br...@azaleos.com>
> wrote:****
>
> ****
>
> Yes****
>
> ****
>
> [default@unknown] describe cluster;****
>
> Cluster Information:****
>
> Snitch: org.apache.cassandra.locator.PropertyFileSnitch****
>
> Partitioner: org.apache.cassandra.dht.RandomPartitioner****
>
> Schema versions:****
>
> 9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5,
> 10.20.8.1, 10.20.8.2, 10.20.8.3]****
>
> ****
>
> *From:* Mohit Anchlia [mailto:mohitanchlia@gmail.com]
> *Sent:* Friday, August 24, 2012 1:55 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
> ****
>
> That's interesting can you do describe cluster?****
>
> On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <Br...@azaleos.com>
> wrote:****
>
> So I’m at the point of updating the keyspaces from Simple to
> NetworkTopology and I’m not sure if the changes are being accepted using
> Cassandra-cli.****
>
> ****
>
> I issue the change:****
>
> ****
>
> [default@EBonding] update keyspace EBonding****
>
> ... with placement_strategy =
> 'org.apache.cassandra.locator.NetworkTopologyStrategy'****
>
> ... and strategy_options={Fisher:2};****
>
> 9511e292-f1b6-3f78-b781-4c90aeb6b0f6****
>
> Waiting for schema agreement...****
>
> ... schemas agree across the cluster****
>
> ****
>
> Then I do a describe and it still shows the old strategy. Is there
> something else that I need to do? I’ve exited and restarted Cassandra-cli
> and it still shows the SimpleStrategy for that keyspace. Other nodes show
> the same information.****
>
> ****
>
> [default@EBonding] describe EBonding;****
>
> Keyspace: EBonding:****
>
> Replication Strategy: org.apache.cassandra.locator.SimpleStrategy****
>
> Durable Writes: true****
>
> Options: [replication_factor:2]****
>
> ****
>
> ****
>
> *From:* Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
> *Sent:* Thursday, August 23, 2012 11:06 AM
> *To:* user@cassandra.apache.org
> *Subject:* RE: Expanding cluster to include a new DR datacenter****
>
> ****
>
> Thanks for the information! Answers my questions.****
>
> ****
>
> *From:* Tyler Hobbs [mailto:tyler@datastax.com <ty...@datastax.com>]
> *Sent:* Wednesday, August 22, 2012 7:10 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
> ****
>
> If you didn't see this particular section, you may find it useful:
> http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
>
> Some comments inline:****
>
> On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Br...@azaleos.com>
> wrote:****
>
> We are in the process of building out a new DR system in another Data
> Center, and we want to mirror our Cassandra environment to that DR. I have
> a couple questions on the best way to do this after reading the
> documentation on the Datastax website. We didn’t initially plan for this
> to be a DR setup when first deployed a while ago due to budgeting, but now
> we need to. So I’m just trying to nail down the order of doing this as
> well as any potential issues.****
>
> ****
>
> For the nodes, we don’t plan on querying the servers in this DR until we
> fail over to this data center. We are going to have 5 similar nodes in
> the DR, should I join them into the ring at token+1?****
>
>
> Join them at token+10 just to leave a little space. Make sure you're
> using LOCAL_QUORUM for your queries instead of regular QUORUM.
> ****
>
> ****
>
> All keyspaces are set to the replication strategy of SimpleStrategy. Can
> I change the replication strategy after joining the new nodes in the DR to
> NetworkTopologyStategy with the updated replication factor for each dr?***
> *
>
>
> Switch your keyspaces over to NetworkTopologyStrategy before adding the
> new nodes. For the strategy options, just list the first dc until the
> second is up (e.g. {main_dc: 3}).
> ****
>
> ****
>
> Lastly, is changing snitch from default of SimpleSnitch to
> RackInferringSnitch going to cause any issues? Since its in the
> Cassandra.yaml file I assume a rolling restart to pick up the value would
> be ok?****
>
>
> This is the first thing you'll want to do. Unless your node IPs would
> naturally put all nodes in a DC in the same rack, I recommend using
> PropertyFileSnitch, explicitly using the same rack. (I tend to prefer
> PFSnitch regardless; it's harder to accidentally mess up.) A rolling
> restart is required to pick up the change. Make sure to fill out
> cassandra-topology.properties first if using PFSnitch.
> ****
>
> ****
>
> This is all on Cassandra 1.1.4, Thanks for any help!****
>
> ****
>
> ****
>
>
>
>
> --
> Tyler Hobbs
> DataStax <http://datastax.com/>****
>
> ****
>
> ****
>
> ** **
>
--
Tyler Hobbs
DataStax <http://datastax.com/>
RE: Expanding cluster to include a new DR datacenter
Posted by Bryce Godfrey <Br...@azaleos.com>.
I just wanted to raise this back up again, see if anyone can provide some help. Thanks.
From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
Sent: Friday, August 31, 2012 11:47 AM
To: user@cassandra.apache.org
Subject: RE: Expanding cluster to include a new DR datacenter
Here is the log around the drop keyspace with debugging enabled.
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Wednesday, August 29, 2012 10:22 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
Can you provide the server log ?
If you can turn the log level up to DEBUG that would be handy as well.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 30/08/2012, at 8:22 AM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Well I tried to drop the keyspace, but it's still there. No errors in logs and Cassandra-cli showed the schema agreement after the command. I took a snapshot of the system keyspace first. Nothing is crashing in the clients yet either, still able to read/write to that keyspace.
[default@EBonding] drop keyspace EBonding;
2eb11095-b8a8-31cd-80c3-c748d32a4208
Waiting for schema agreement...
... schemas agree across the cluster
[default@unknown] use EBonding;
Authenticated to keyspace: EBonding
[default@EBonding] describe;
Keyspace: EBonding:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Durable Writes: true
Options: [replication_factor:2]
From: aaron morton [mailto:aaron@thelastpickle.com<http://thelastpickle.com>]
Sent: Wednesday, August 29, 2012 2:36 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
It would be handy to work out what the corruption is. Could you snapshot the system keyspace and store it somewhere, just incase we can look at it later ?
Is there a way I can confirm this
Errors in the client and/or the server log is the the traditional way.
go about cleaning up/restoring the proper schema?
If you need to get it back, and can handle the down time, the simple thing is drop the KS and re-create it. Remember to take a snapshot first. Drop keyspace takes one but it's the sort of thing I would do myself.
Or you can try _try_ nodetool resetlocalschema. Without knowing what the error is it's hard to say if it would work.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 29/08/2012, at 9:10 AM, Bryce Godfrey <Br...@azaleos.com>> wrote:
I believe what may be really going on is that my schema is in a bad or corrupt state. I also have one keyspace that I just cannot drop an existing column family from even though it shows no errors.
So right now I was able to get 4 of my 6 keyspaces over to Network Topology strategy.
I think I got into this bad state after pointing Opscenter at this cluster for the first time, as it started throwing errors after that and crashed a couple of my nodes until I stopped it and its agents.
Is there a way I can confirm this or go about cleaning up/restoring the proper schema?
From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com<http://azaleos.com>]
Sent: Tuesday, August 28, 2012 11:09 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: RE: Expanding cluster to include a new DR datacenter
So in an interesting turn of events, this works on my other 4 keyspaces but just not this 'EBonding' one which will not recognize the changes. I can probably get around this by dropping and re-creating this keyspace since its uptime is not too important for us.
[default@AlertStats] describe AlertStats;
Keyspace: AlertStats:
Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
Durable Writes: true
Options: [Fisher:3]
From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
Sent: Monday, August 27, 2012 3:50 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
Can you describe your schema again with TierPoint in it?
On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Same results. I restarted the node also to see if it just wasn't picking up the changes and it still shows Simple.
When I specify the DC for strategy_options I should be using the DC name from properfy file snitch right? Ours is "Fisher" and "TierPoint" so that's what I used.
From: Mohit Anchlia [mailto:mohitanchlia@gmail.com<ma...@gmail.com>]
Sent: Monday, August 27, 2012 1:21 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
In your update command is it possible to specify RF for both DC? You could just do DC1:2, DC2:0.
On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Show schema output show the simple strategy still
[default@unknown] show schema EBonding;
create keyspace EBonding
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 2}
and durable_writes = true;
This is the only thing I see in the system log at the time on all the nodes:
INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264) Writing Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305) Completed flushing /opt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-he-34817-Data.db (241 bytes) for commitlog p$
Should I turn the logging level up on something to see some more info maybe?
From: aaron morton [mailto:aaron@thelastpickle.com<ma...@thelastpickle.com>]
Sent: Monday, August 27, 2012 1:35 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
I did a quick test on a clean 1.1.4 and it worked
Can you check the logs for errors ? Can you see your schema change in there ?
Also what is the output from show schema; in the cli ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com<http://www.thelastpickle.com/>
On 25/08/2012, at 6:53 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Yes
[default@unknown] describe cluster;
Cluster Information:
Snitch: org.apache.cassandra.locator.PropertyFileSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, 10.20.8.1, 10.20.8.2, 10.20.8.3]
From: Mohit Anchlia [mailto:mohitanchlia@<mailto:mohitanchlia@>gmail.com<http://gmail.com/>]
Sent: Friday, August 24, 2012 1:55 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
That's interesting can you do describe cluster?
On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
So I'm at the point of updating the keyspaces from Simple to NetworkTopology and I'm not sure if the changes are being accepted using Cassandra-cli.
I issue the change:
[default@EBonding] update keyspace EBonding
... with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
... and strategy_options={Fisher:2};
9511e292-f1b6-3f78-b781-4c90aeb6b0f6
Waiting for schema agreement...
... schemas agree across the cluster
Then I do a describe and it still shows the old strategy. Is there something else that I need to do? I've exited and restarted Cassandra-cli and it still shows the SimpleStrategy for that keyspace. Other nodes show the same information.
[default@EBonding] describe EBonding;
Keyspace: EBonding:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Durable Writes: true
Options: [replication_factor:2]
From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com<ma...@azaleos.com>]
Sent: Thursday, August 23, 2012 11:06 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: RE: Expanding cluster to include a new DR datacenter
Thanks for the information! Answers my questions.
From: Tyler Hobbs [mailto:tyler@datastax.com]
Sent: Wednesday, August 22, 2012 7:10 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
If you didn't see this particular section, you may find it useful:http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
Some comments inline:
On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
We are in the process of building out a new DR system in another Data Center, and we want to mirror our Cassandra environment to that DR. I have a couple questions on the best way to do this after reading the documentation on the Datastax website. We didn't initially plan for this to be a DR setup when first deployed a while ago due to budgeting, but now we need to. So I'm just trying to nail down the order of doing this as well as any potential issues.
For the nodes, we don't plan on querying the servers in this DR until we fail over to this data center. We are going to have 5 similar nodes in the DR, should I join them into the ring at token+1?
Join them at token+10 just to leave a little space. Make sure you're using LOCAL_QUORUM for your queries instead of regular QUORUM.
All keyspaces are set to the replication strategy of SimpleStrategy. Can I change the replication strategy after joining the new nodes in the DR to NetworkTopologyStategy with the updated replication factor for each dr?
Switch your keyspaces over to NetworkTopologyStrategy before adding the new nodes. For the strategy options, just list the first dc until the second is up (e.g. {main_dc: 3}).
Lastly, is changing snitch from default of SimpleSnitch to RackInferringSnitch going to cause any issues? Since its in the Cassandra.yaml file I assume a rolling restart to pick up the value would be ok?
This is the first thing you'll want to do. Unless your node IPs would naturally put all nodes in a DC in the same rack, I recommend using PropertyFileSnitch, explicitly using the same rack. (I tend to prefer PFSnitch regardless; it's harder to accidentally mess up.) A rolling restart is required to pick up the change. Make sure to fill out cassandra-topology.properties first if using PFSnitch.
This is all on Cassandra 1.1.4, Thanks for any help!
--
Tyler Hobbs
DataStax<http://datastax.com/>
RE: Expanding cluster to include a new DR datacenter
Posted by Bryce Godfrey <Br...@azaleos.com>.
Here is the log around the drop keyspace with debugging enabled.
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Wednesday, August 29, 2012 10:22 PM
To: user@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter
Can you provide the server log ?
If you can turn the log level up to DEBUG that would be handy as well.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 30/08/2012, at 8:22 AM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Well I tried to drop the keyspace, but it's still there. No errors in logs and Cassandra-cli showed the schema agreement after the command. I took a snapshot of the system keyspace first. Nothing is crashing in the clients yet either, still able to read/write to that keyspace.
[default@EBonding] drop keyspace EBonding;
2eb11095-b8a8-31cd-80c3-c748d32a4208
Waiting for schema agreement...
... schemas agree across the cluster
[default@unknown] use EBonding;
Authenticated to keyspace: EBonding
[default@EBonding] describe;
Keyspace: EBonding:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Durable Writes: true
Options: [replication_factor:2]
From: aaron morton [mailto:aaron@thelastpickle.com<http://thelastpickle.com>]
Sent: Wednesday, August 29, 2012 2:36 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
It would be handy to work out what the corruption is. Could you snapshot the system keyspace and store it somewhere, just incase we can look at it later ?
Is there a way I can confirm this
Errors in the client and/or the server log is the the traditional way.
go about cleaning up/restoring the proper schema?
If you need to get it back, and can handle the down time, the simple thing is drop the KS and re-create it. Remember to take a snapshot first. Drop keyspace takes one but it's the sort of thing I would do myself.
Or you can try _try_ nodetool resetlocalschema. Without knowing what the error is it's hard to say if it would work.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 29/08/2012, at 9:10 AM, Bryce Godfrey <Br...@azaleos.com>> wrote:
I believe what may be really going on is that my schema is in a bad or corrupt state. I also have one keyspace that I just cannot drop an existing column family from even though it shows no errors.
So right now I was able to get 4 of my 6 keyspaces over to Network Topology strategy.
I think I got into this bad state after pointing Opscenter at this cluster for the first time, as it started throwing errors after that and crashed a couple of my nodes until I stopped it and its agents.
Is there a way I can confirm this or go about cleaning up/restoring the proper schema?
From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com<http://azaleos.com>]
Sent: Tuesday, August 28, 2012 11:09 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: RE: Expanding cluster to include a new DR datacenter
So in an interesting turn of events, this works on my other 4 keyspaces but just not this 'EBonding' one which will not recognize the changes. I can probably get around this by dropping and re-creating this keyspace since its uptime is not too important for us.
[default@AlertStats] describe AlertStats;
Keyspace: AlertStats:
Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
Durable Writes: true
Options: [Fisher:3]
From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
Sent: Monday, August 27, 2012 3:50 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
Can you describe your schema again with TierPoint in it?
On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Same results. I restarted the node also to see if it just wasn't picking up the changes and it still shows Simple.
When I specify the DC for strategy_options I should be using the DC name from properfy file snitch right? Ours is "Fisher" and "TierPoint" so that's what I used.
From: Mohit Anchlia [mailto:mohitanchlia@gmail.com<ma...@gmail.com>]
Sent: Monday, August 27, 2012 1:21 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
In your update command is it possible to specify RF for both DC? You could just do DC1:2, DC2:0.
On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Show schema output show the simple strategy still
[default@unknown] show schema EBonding;
create keyspace EBonding
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 2}
and durable_writes = true;
This is the only thing I see in the system log at the time on all the nodes:
INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264) Writing Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305) Completed flushing /opt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-he-34817-Data.db (241 bytes) for commitlog p$
Should I turn the logging level up on something to see some more info maybe?
From: aaron morton [mailto:aaron@thelastpickle.com<ma...@thelastpickle.com>]
Sent: Monday, August 27, 2012 1:35 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
I did a quick test on a clean 1.1.4 and it worked
Can you check the logs for errors ? Can you see your schema change in there ?
Also what is the output from show schema; in the cli ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com<http://www.thelastpickle.com/>
On 25/08/2012, at 6:53 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Yes
[default@unknown] describe cluster;
Cluster Information:
Snitch: org.apache.cassandra.locator.PropertyFileSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, 10.20.8.1, 10.20.8.2, 10.20.8.3]
From: Mohit Anchlia [mailto:mohitanchlia@<mailto:mohitanchlia@>gmail.com<http://gmail.com/>]
Sent: Friday, August 24, 2012 1:55 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
That's interesting can you do describe cluster?
On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
So I'm at the point of updating the keyspaces from Simple to NetworkTopology and I'm not sure if the changes are being accepted using Cassandra-cli.
I issue the change:
[default@EBonding] update keyspace EBonding
... with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
... and strategy_options={Fisher:2};
9511e292-f1b6-3f78-b781-4c90aeb6b0f6
Waiting for schema agreement...
... schemas agree across the cluster
Then I do a describe and it still shows the old strategy. Is there something else that I need to do? I've exited and restarted Cassandra-cli and it still shows the SimpleStrategy for that keyspace. Other nodes show the same information.
[default@EBonding] describe EBonding;
Keyspace: EBonding:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Durable Writes: true
Options: [replication_factor:2]
From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com<ma...@azaleos.com>]
Sent: Thursday, August 23, 2012 11:06 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: RE: Expanding cluster to include a new DR datacenter
Thanks for the information! Answers my questions.
From: Tyler Hobbs [mailto:tyler@datastax.com]
Sent: Wednesday, August 22, 2012 7:10 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
If you didn't see this particular section, you may find it useful:http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
Some comments inline:
On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
We are in the process of building out a new DR system in another Data Center, and we want to mirror our Cassandra environment to that DR. I have a couple questions on the best way to do this after reading the documentation on the Datastax website. We didn't initially plan for this to be a DR setup when first deployed a while ago due to budgeting, but now we need to. So I'm just trying to nail down the order of doing this as well as any potential issues.
For the nodes, we don't plan on querying the servers in this DR until we fail over to this data center. We are going to have 5 similar nodes in the DR, should I join them into the ring at token+1?
Join them at token+10 just to leave a little space. Make sure you're using LOCAL_QUORUM for your queries instead of regular QUORUM.
All keyspaces are set to the replication strategy of SimpleStrategy. Can I change the replication strategy after joining the new nodes in the DR to NetworkTopologyStategy with the updated replication factor for each dr?
Switch your keyspaces over to NetworkTopologyStrategy before adding the new nodes. For the strategy options, just list the first dc until the second is up (e.g. {main_dc: 3}).
Lastly, is changing snitch from default of SimpleSnitch to RackInferringSnitch going to cause any issues? Since its in the Cassandra.yaml file I assume a rolling restart to pick up the value would be ok?
This is the first thing you'll want to do. Unless your node IPs would naturally put all nodes in a DC in the same rack, I recommend using PropertyFileSnitch, explicitly using the same rack. (I tend to prefer PFSnitch regardless; it's harder to accidentally mess up.) A rolling restart is required to pick up the change. Make sure to fill out cassandra-topology.properties first if using PFSnitch.
This is all on Cassandra 1.1.4, Thanks for any help!
--
Tyler Hobbs
DataStax<http://datastax.com/>
Re: Expanding cluster to include a new DR datacenter
Posted by aaron morton <aa...@thelastpickle.com>.
Can you provide the server log ?
If you can turn the log level up to DEBUG that would be handy as well.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 30/08/2012, at 8:22 AM, Bryce Godfrey <Br...@azaleos.com> wrote:
> Well I tried to drop the keyspace, but it’s still there. No errors in logs and Cassandra-cli showed the schema agreement after the command. I took a snapshot of the system keyspace first. Nothing is crashing in the clients yet either, still able to read/write to that keyspace.
>
> [default@EBonding] drop keyspace EBonding;
> 2eb11095-b8a8-31cd-80c3-c748d32a4208
> Waiting for schema agreement...
> ... schemas agree across the cluster
>
> [default@unknown] use EBonding;
> Authenticated to keyspace: EBonding
> [default@EBonding] describe;
> Keyspace: EBonding:
> Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
> Durable Writes: true
> Options: [replication_factor:2]
>
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: Wednesday, August 29, 2012 2:36 AM
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>
> It would be handy to work out what the corruption is. Could you snapshot the system keyspace and store it somewhere, just incase we can look at it later ?
>
> Is there a way I can confirm this
> Errors in the client and/or the server log is the the traditional way.
>
> go about cleaning up/restoring the proper schema?
> If you need to get it back, and can handle the down time, the simple thing is drop the KS and re-create it. Remember to take a snapshot first. Drop keyspace takes one but it's the sort of thing I would do myself.
>
> Or you can try _try_ nodetool resetlocalschema. Without knowing what the error is it's hard to say if it would work.
>
> Cheers
>
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 29/08/2012, at 9:10 AM, Bryce Godfrey <Br...@azaleos.com> wrote:
>
>
> I believe what may be really going on is that my schema is in a bad or corrupt state. I also have one keyspace that I just cannot drop an existing column family from even though it shows no errors.
>
> So right now I was able to get 4 of my 6 keyspaces over to Network Topology strategy.
>
> I think I got into this bad state after pointing Opscenter at this cluster for the first time, as it started throwing errors after that and crashed a couple of my nodes until I stopped it and its agents.
>
> Is there a way I can confirm this or go about cleaning up/restoring the proper schema?
>
> From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
> Sent: Tuesday, August 28, 2012 11:09 AM
> To: user@cassandra.apache.org
> Subject: RE: Expanding cluster to include a new DR datacenter
>
> So in an interesting turn of events, this works on my other 4 keyspaces but just not this ‘EBonding’ one which will not recognize the changes. I can probably get around this by dropping and re-creating this keyspace since its uptime is not too important for us.
>
> [default@AlertStats] describe AlertStats;
> Keyspace: AlertStats:
> Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
> Durable Writes: true
> Options: [Fisher:3]
>
> From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
> Sent: Monday, August 27, 2012 3:50 PM
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>
> Can you describe your schema again with TierPoint in it?
>
> On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey <Br...@azaleos.com> wrote:
> Same results. I restarted the node also to see if it just wasn’t picking up the changes and it still shows Simple.
>
> When I specify the DC for strategy_options I should be using the DC name from properfy file snitch right? Ours is “Fisher” and “TierPoint” so that’s what I used.
>
> From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
> Sent: Monday, August 27, 2012 1:21 PM
>
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>
> In your update command is it possible to specify RF for both DC? You could just do DC1:2, DC2:0.
>
> On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey <Br...@azaleos.com> wrote:
> Show schema output show the simple strategy still
> [default@unknown] show schema EBonding;
> create keyspace EBonding
> with placement_strategy = 'SimpleStrategy'
> and strategy_options = {replication_factor : 2}
> and durable_writes = true;
>
> This is the only thing I see in the system log at the time on all the nodes:
>
> INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
> INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264) Writing Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
> INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305) Completed flushing /opt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-he-34817-Data.db (241 bytes) for commitlog p$
>
>
> Should I turn the logging level up on something to see some more info maybe?
>
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: Monday, August 27, 2012 1:35 AM
>
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>
> I did a quick test on a clean 1.1.4 and it worked
>
> Can you check the logs for errors ? Can you see your schema change in there ?
>
> Also what is the output from show schema; in the cli ?
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 25/08/2012, at 6:53 PM, Bryce Godfrey <Br...@azaleos.com> wrote:
>
>
> Yes
>
> [default@unknown] describe cluster;
> Cluster Information:
> Snitch: org.apache.cassandra.locator.PropertyFileSnitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> 9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, 10.20.8.1, 10.20.8.2, 10.20.8.3]
>
> From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
> Sent: Friday, August 24, 2012 1:55 PM
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>
> That's interesting can you do describe cluster?
>
> On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <Br...@azaleos.com> wrote:
> So I’m at the point of updating the keyspaces from Simple to NetworkTopology and I’m not sure if the changes are being accepted using Cassandra-cli.
>
> I issue the change:
>
> [default@EBonding] update keyspace EBonding
> ... with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
> ... and strategy_options={Fisher:2};
> 9511e292-f1b6-3f78-b781-4c90aeb6b0f6
> Waiting for schema agreement...
> ... schemas agree across the cluster
>
> Then I do a describe and it still shows the old strategy. Is there something else that I need to do? I’ve exited and restarted Cassandra-cli and it still shows the SimpleStrategy for that keyspace. Other nodes show the same information.
>
> [default@EBonding] describe EBonding;
> Keyspace: EBonding:
> Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
> Durable Writes: true
> Options: [replication_factor:2]
>
>
> From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
> Sent: Thursday, August 23, 2012 11:06 AM
> To: user@cassandra.apache.org
> Subject: RE: Expanding cluster to include a new DR datacenter
>
> Thanks for the information! Answers my questions.
>
> From: Tyler Hobbs [mailto:tyler@datastax.com]
> Sent: Wednesday, August 22, 2012 7:10 PM
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>
> If you didn't see this particular section, you may find it useful:http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
>
> Some comments inline:
>
> On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Br...@azaleos.com> wrote:
> We are in the process of building out a new DR system in another Data Center, and we want to mirror our Cassandra environment to that DR. I have a couple questions on the best way to do this after reading the documentation on the Datastax website. We didn’t initially plan for this to be a DR setup when first deployed a while ago due to budgeting, but now we need to. So I’m just trying to nail down the order of doing this as well as any potential issues.
>
> For the nodes, we don’t plan on querying the servers in this DR until we fail over to this data center. We are going to have 5 similar nodes in the DR, should I join them into the ring at token+1?
>
> Join them at token+10 just to leave a little space. Make sure you're using LOCAL_QUORUM for your queries instead of regular QUORUM.
>
>
> All keyspaces are set to the replication strategy of SimpleStrategy. Can I change the replication strategy after joining the new nodes in the DR to NetworkTopologyStategy with the updated replication factor for each dr?
>
> Switch your keyspaces over to NetworkTopologyStrategy before adding the new nodes. For the strategy options, just list the first dc until the second is up (e.g. {main_dc: 3}).
>
>
> Lastly, is changing snitch from default of SimpleSnitch to RackInferringSnitch going to cause any issues? Since its in the Cassandra.yaml file I assume a rolling restart to pick up the value would be ok?
>
> This is the first thing you'll want to do. Unless your node IPs would naturally put all nodes in a DC in the same rack, I recommend using PropertyFileSnitch, explicitly using the same rack. (I tend to prefer PFSnitch regardless; it's harder to accidentally mess up.) A rolling restart is required to pick up the change. Make sure to fill out cassandra-topology.properties first if using PFSnitch.
>
>
> This is all on Cassandra 1.1.4, Thanks for any help!
>
>
>
>
>
> --
> Tyler Hobbs
> DataStax
>
>
>
RE: Expanding cluster to include a new DR datacenter
Posted by Bryce Godfrey <Br...@azaleos.com>.
Well I tried to drop the keyspace, but it's still there. No errors in logs and Cassandra-cli showed the schema agreement after the command. I took a snapshot of the system keyspace first. Nothing is crashing in the clients yet either, still able to read/write to that keyspace.
[default@EBonding] drop keyspace EBonding;
2eb11095-b8a8-31cd-80c3-c748d32a4208
Waiting for schema agreement...
... schemas agree across the cluster
[default@unknown] use EBonding;
Authenticated to keyspace: EBonding
[default@EBonding] describe;
Keyspace: EBonding:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Durable Writes: true
Options: [replication_factor:2]
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Wednesday, August 29, 2012 2:36 AM
To: user@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter
It would be handy to work out what the corruption is. Could you snapshot the system keyspace and store it somewhere, just incase we can look at it later ?
Is there a way I can confirm this
Errors in the client and/or the server log is the the traditional way.
go about cleaning up/restoring the proper schema?
If you need to get it back, and can handle the down time, the simple thing is drop the KS and re-create it. Remember to take a snapshot first. Drop keyspace takes one but it's the sort of thing I would do myself.
Or you can try _try_ nodetool resetlocalschema. Without knowing what the error is it's hard to say if it would work.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 29/08/2012, at 9:10 AM, Bryce Godfrey <Br...@azaleos.com>> wrote:
I believe what may be really going on is that my schema is in a bad or corrupt state. I also have one keyspace that I just cannot drop an existing column family from even though it shows no errors.
So right now I was able to get 4 of my 6 keyspaces over to Network Topology strategy.
I think I got into this bad state after pointing Opscenter at this cluster for the first time, as it started throwing errors after that and crashed a couple of my nodes until I stopped it and its agents.
Is there a way I can confirm this or go about cleaning up/restoring the proper schema?
From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com<http://azaleos.com>]
Sent: Tuesday, August 28, 2012 11:09 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: RE: Expanding cluster to include a new DR datacenter
So in an interesting turn of events, this works on my other 4 keyspaces but just not this 'EBonding' one which will not recognize the changes. I can probably get around this by dropping and re-creating this keyspace since its uptime is not too important for us.
[default@AlertStats] describe AlertStats;
Keyspace: AlertStats:
Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
Durable Writes: true
Options: [Fisher:3]
From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
Sent: Monday, August 27, 2012 3:50 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
Can you describe your schema again with TierPoint in it?
On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Same results. I restarted the node also to see if it just wasn't picking up the changes and it still shows Simple.
When I specify the DC for strategy_options I should be using the DC name from properfy file snitch right? Ours is "Fisher" and "TierPoint" so that's what I used.
From: Mohit Anchlia [mailto:mohitanchlia@gmail.com<ma...@gmail.com>]
Sent: Monday, August 27, 2012 1:21 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
In your update command is it possible to specify RF for both DC? You could just do DC1:2, DC2:0.
On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Show schema output show the simple strategy still
[default@unknown] show schema EBonding;
create keyspace EBonding
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 2}
and durable_writes = true;
This is the only thing I see in the system log at the time on all the nodes:
INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264) Writing Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305) Completed flushing /opt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-he-34817-Data.db (241 bytes) for commitlog p$
Should I turn the logging level up on something to see some more info maybe?
From: aaron morton [mailto:aaron@thelastpickle.com<ma...@thelastpickle.com>]
Sent: Monday, August 27, 2012 1:35 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
I did a quick test on a clean 1.1.4 and it worked
Can you check the logs for errors ? Can you see your schema change in there ?
Also what is the output from show schema; in the cli ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com<http://www.thelastpickle.com/>
On 25/08/2012, at 6:53 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Yes
[default@unknown] describe cluster;
Cluster Information:
Snitch: org.apache.cassandra.locator.PropertyFileSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, 10.20.8.1, 10.20.8.2, 10.20.8.3]
From: Mohit Anchlia [mailto:mohitanchlia@<mailto:mohitanchlia@>gmail.com<http://gmail.com/>]
Sent: Friday, August 24, 2012 1:55 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
That's interesting can you do describe cluster?
On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
So I'm at the point of updating the keyspaces from Simple to NetworkTopology and I'm not sure if the changes are being accepted using Cassandra-cli.
I issue the change:
[default@EBonding] update keyspace EBonding
... with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
... and strategy_options={Fisher:2};
9511e292-f1b6-3f78-b781-4c90aeb6b0f6
Waiting for schema agreement...
... schemas agree across the cluster
Then I do a describe and it still shows the old strategy. Is there something else that I need to do? I've exited and restarted Cassandra-cli and it still shows the SimpleStrategy for that keyspace. Other nodes show the same information.
[default@EBonding] describe EBonding;
Keyspace: EBonding:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Durable Writes: true
Options: [replication_factor:2]
From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com<ma...@azaleos.com>]
Sent: Thursday, August 23, 2012 11:06 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: RE: Expanding cluster to include a new DR datacenter
Thanks for the information! Answers my questions.
From: Tyler Hobbs [mailto:tyler@datastax.com]
Sent: Wednesday, August 22, 2012 7:10 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
If you didn't see this particular section, you may find it useful:http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
Some comments inline:
On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
We are in the process of building out a new DR system in another Data Center, and we want to mirror our Cassandra environment to that DR. I have a couple questions on the best way to do this after reading the documentation on the Datastax website. We didn't initially plan for this to be a DR setup when first deployed a while ago due to budgeting, but now we need to. So I'm just trying to nail down the order of doing this as well as any potential issues.
For the nodes, we don't plan on querying the servers in this DR until we fail over to this data center. We are going to have 5 similar nodes in the DR, should I join them into the ring at token+1?
Join them at token+10 just to leave a little space. Make sure you're using LOCAL_QUORUM for your queries instead of regular QUORUM.
All keyspaces are set to the replication strategy of SimpleStrategy. Can I change the replication strategy after joining the new nodes in the DR to NetworkTopologyStategy with the updated replication factor for each dr?
Switch your keyspaces over to NetworkTopologyStrategy before adding the new nodes. For the strategy options, just list the first dc until the second is up (e.g. {main_dc: 3}).
Lastly, is changing snitch from default of SimpleSnitch to RackInferringSnitch going to cause any issues? Since its in the Cassandra.yaml file I assume a rolling restart to pick up the value would be ok?
This is the first thing you'll want to do. Unless your node IPs would naturally put all nodes in a DC in the same rack, I recommend using PropertyFileSnitch, explicitly using the same rack. (I tend to prefer PFSnitch regardless; it's harder to accidentally mess up.) A rolling restart is required to pick up the change. Make sure to fill out cassandra-topology.properties first if using PFSnitch.
This is all on Cassandra 1.1.4, Thanks for any help!
--
Tyler Hobbs
DataStax<http://datastax.com/>
Re: Expanding cluster to include a new DR datacenter
Posted by aaron morton <aa...@thelastpickle.com>.
It would be handy to work out what the corruption is. Could you snapshot the system keyspace and store it somewhere, just incase we can look at it later ?
> Is there a way I can confirm this
Errors in the client and/or the server log is the the traditional way.
> go about cleaning up/restoring the proper schema?
If you need to get it back, and can handle the down time, the simple thing is drop the KS and re-create it. Remember to take a snapshot first. Drop keyspace takes one but it's the sort of thing I would do myself.
Or you can try _try_ nodetool resetlocalschema. Without knowing what the error is it's hard to say if it would work.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 29/08/2012, at 9:10 AM, Bryce Godfrey <Br...@azaleos.com> wrote:
> I believe what may be really going on is that my schema is in a bad or corrupt state. I also have one keyspace that I just cannot drop an existing column family from even though it shows no errors.
>
> So right now I was able to get 4 of my 6 keyspaces over to Network Topology strategy.
>
> I think I got into this bad state after pointing Opscenter at this cluster for the first time, as it started throwing errors after that and crashed a couple of my nodes until I stopped it and its agents.
>
> Is there a way I can confirm this or go about cleaning up/restoring the proper schema?
>
> From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
> Sent: Tuesday, August 28, 2012 11:09 AM
> To: user@cassandra.apache.org
> Subject: RE: Expanding cluster to include a new DR datacenter
>
> So in an interesting turn of events, this works on my other 4 keyspaces but just not this ‘EBonding’ one which will not recognize the changes. I can probably get around this by dropping and re-creating this keyspace since its uptime is not too important for us.
>
> [default@AlertStats] describe AlertStats;
> Keyspace: AlertStats:
> Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
> Durable Writes: true
> Options: [Fisher:3]
>
> From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
> Sent: Monday, August 27, 2012 3:50 PM
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>
> Can you describe your schema again with TierPoint in it?
>
> On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey <Br...@azaleos.com> wrote:
> Same results. I restarted the node also to see if it just wasn’t picking up the changes and it still shows Simple.
>
> When I specify the DC for strategy_options I should be using the DC name from properfy file snitch right? Ours is “Fisher” and “TierPoint” so that’s what I used.
>
> From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
> Sent: Monday, August 27, 2012 1:21 PM
>
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>
> In your update command is it possible to specify RF for both DC? You could just do DC1:2, DC2:0.
>
> On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey <Br...@azaleos.com> wrote:
> Show schema output show the simple strategy still
> [default@unknown] show schema EBonding;
> create keyspace EBonding
> with placement_strategy = 'SimpleStrategy'
> and strategy_options = {replication_factor : 2}
> and durable_writes = true;
>
> This is the only thing I see in the system log at the time on all the nodes:
>
> INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
> INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264) Writing Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
> INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305) Completed flushing /opt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-he-34817-Data.db (241 bytes) for commitlog p$
>
>
> Should I turn the logging level up on something to see some more info maybe?
>
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: Monday, August 27, 2012 1:35 AM
>
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>
> I did a quick test on a clean 1.1.4 and it worked
>
> Can you check the logs for errors ? Can you see your schema change in there ?
>
> Also what is the output from show schema; in the cli ?
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 25/08/2012, at 6:53 PM, Bryce Godfrey <Br...@azaleos.com> wrote:
>
>
> Yes
>
> [default@unknown] describe cluster;
> Cluster Information:
> Snitch: org.apache.cassandra.locator.PropertyFileSnitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> 9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, 10.20.8.1, 10.20.8.2, 10.20.8.3]
>
> From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
> Sent: Friday, August 24, 2012 1:55 PM
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>
> That's interesting can you do describe cluster?
>
> On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <Br...@azaleos.com> wrote:
> So I’m at the point of updating the keyspaces from Simple to NetworkTopology and I’m not sure if the changes are being accepted using Cassandra-cli.
>
> I issue the change:
>
> [default@EBonding] update keyspace EBonding
> ... with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
> ... and strategy_options={Fisher:2};
> 9511e292-f1b6-3f78-b781-4c90aeb6b0f6
> Waiting for schema agreement...
> ... schemas agree across the cluster
>
> Then I do a describe and it still shows the old strategy. Is there something else that I need to do? I’ve exited and restarted Cassandra-cli and it still shows the SimpleStrategy for that keyspace. Other nodes show the same information.
>
> [default@EBonding] describe EBonding;
> Keyspace: EBonding:
> Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
> Durable Writes: true
> Options: [replication_factor:2]
>
>
> From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
> Sent: Thursday, August 23, 2012 11:06 AM
> To: user@cassandra.apache.org
> Subject: RE: Expanding cluster to include a new DR datacenter
>
> Thanks for the information! Answers my questions.
>
> From: Tyler Hobbs [mailto:tyler@datastax.com]
> Sent: Wednesday, August 22, 2012 7:10 PM
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>
> If you didn't see this particular section, you may find it useful:http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
>
> Some comments inline:
>
> On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Br...@azaleos.com> wrote:
> We are in the process of building out a new DR system in another Data Center, and we want to mirror our Cassandra environment to that DR. I have a couple questions on the best way to do this after reading the documentation on the Datastax website. We didn’t initially plan for this to be a DR setup when first deployed a while ago due to budgeting, but now we need to. So I’m just trying to nail down the order of doing this as well as any potential issues.
>
> For the nodes, we don’t plan on querying the servers in this DR until we fail over to this data center. We are going to have 5 similar nodes in the DR, should I join them into the ring at token+1?
>
> Join them at token+10 just to leave a little space. Make sure you're using LOCAL_QUORUM for your queries instead of regular QUORUM.
>
>
> All keyspaces are set to the replication strategy of SimpleStrategy. Can I change the replication strategy after joining the new nodes in the DR to NetworkTopologyStategy with the updated replication factor for each dr?
>
> Switch your keyspaces over to NetworkTopologyStrategy before adding the new nodes. For the strategy options, just list the first dc until the second is up (e.g. {main_dc: 3}).
>
>
> Lastly, is changing snitch from default of SimpleSnitch to RackInferringSnitch going to cause any issues? Since its in the Cassandra.yaml file I assume a rolling restart to pick up the value would be ok?
>
> This is the first thing you'll want to do. Unless your node IPs would naturally put all nodes in a DC in the same rack, I recommend using PropertyFileSnitch, explicitly using the same rack. (I tend to prefer PFSnitch regardless; it's harder to accidentally mess up.) A rolling restart is required to pick up the change. Make sure to fill out cassandra-topology.properties first if using PFSnitch.
>
>
> This is all on Cassandra 1.1.4, Thanks for any help!
>
>
>
>
>
> --
> Tyler Hobbs
> DataStax
>
>
>
RE: Expanding cluster to include a new DR datacenter
Posted by Bryce Godfrey <Br...@azaleos.com>.
I believe what may be really going on is that my schema is in a bad or corrupt state. I also have one keyspace that I just cannot drop an existing column family from even though it shows no errors.
So right now I was able to get 4 of my 6 keyspaces over to Network Topology strategy.
I think I got into this bad state after pointing Opscenter at this cluster for the first time, as it started throwing errors after that and crashed a couple of my nodes until I stopped it and its agents.
Is there a way I can confirm this or go about cleaning up/restoring the proper schema?
From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
Sent: Tuesday, August 28, 2012 11:09 AM
To: user@cassandra.apache.org
Subject: RE: Expanding cluster to include a new DR datacenter
So in an interesting turn of events, this works on my other 4 keyspaces but just not this 'EBonding' one which will not recognize the changes. I can probably get around this by dropping and re-creating this keyspace since its uptime is not too important for us.
[default@AlertStats] describe AlertStats;
Keyspace: AlertStats:
Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
Durable Writes: true
Options: [Fisher:3]
From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
Sent: Monday, August 27, 2012 3:50 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
Can you describe your schema again with TierPoint in it?
On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Same results. I restarted the node also to see if it just wasn't picking up the changes and it still shows Simple.
When I specify the DC for strategy_options I should be using the DC name from properfy file snitch right? Ours is "Fisher" and "TierPoint" so that's what I used.
From: Mohit Anchlia [mailto:mohitanchlia@gmail.com<ma...@gmail.com>]
Sent: Monday, August 27, 2012 1:21 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
In your update command is it possible to specify RF for both DC? You could just do DC1:2, DC2:0.
On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Show schema output show the simple strategy still
[default@unknown] show schema EBonding;
create keyspace EBonding
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 2}
and durable_writes = true;
This is the only thing I see in the system log at the time on all the nodes:
INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264) Writing Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305) Completed flushing /opt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-he-34817-Data.db (241 bytes) for commitlog p$
Should I turn the logging level up on something to see some more info maybe?
From: aaron morton [mailto:aaron@thelastpickle.com<ma...@thelastpickle.com>]
Sent: Monday, August 27, 2012 1:35 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
I did a quick test on a clean 1.1.4 and it worked
Can you check the logs for errors ? Can you see your schema change in there ?
Also what is the output from show schema; in the cli ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com<http://www.thelastpickle.com/>
On 25/08/2012, at 6:53 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Yes
[default@unknown] describe cluster;
Cluster Information:
Snitch: org.apache.cassandra.locator.PropertyFileSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, 10.20.8.1, 10.20.8.2, 10.20.8.3]
From: Mohit Anchlia [mailto:mohitanchlia@<mailto:mohitanchlia@>gmail.com<http://gmail.com/>]
Sent: Friday, August 24, 2012 1:55 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
That's interesting can you do describe cluster?
On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
So I'm at the point of updating the keyspaces from Simple to NetworkTopology and I'm not sure if the changes are being accepted using Cassandra-cli.
I issue the change:
[default@EBonding] update keyspace EBonding
... with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
... and strategy_options={Fisher:2};
9511e292-f1b6-3f78-b781-4c90aeb6b0f6
Waiting for schema agreement...
... schemas agree across the cluster
Then I do a describe and it still shows the old strategy. Is there something else that I need to do? I've exited and restarted Cassandra-cli and it still shows the SimpleStrategy for that keyspace. Other nodes show the same information.
[default@EBonding] describe EBonding;
Keyspace: EBonding:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Durable Writes: true
Options: [replication_factor:2]
From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com<ma...@azaleos.com>]
Sent: Thursday, August 23, 2012 11:06 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: RE: Expanding cluster to include a new DR datacenter
Thanks for the information! Answers my questions.
From: Tyler Hobbs [mailto:tyler@datastax.com]
Sent: Wednesday, August 22, 2012 7:10 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
If you didn't see this particular section, you may find it useful:http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
Some comments inline:
On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
We are in the process of building out a new DR system in another Data Center, and we want to mirror our Cassandra environment to that DR. I have a couple questions on the best way to do this after reading the documentation on the Datastax website. We didn't initially plan for this to be a DR setup when first deployed a while ago due to budgeting, but now we need to. So I'm just trying to nail down the order of doing this as well as any potential issues.
For the nodes, we don't plan on querying the servers in this DR until we fail over to this data center. We are going to have 5 similar nodes in the DR, should I join them into the ring at token+1?
Join them at token+10 just to leave a little space. Make sure you're using LOCAL_QUORUM for your queries instead of regular QUORUM.
All keyspaces are set to the replication strategy of SimpleStrategy. Can I change the replication strategy after joining the new nodes in the DR to NetworkTopologyStategy with the updated replication factor for each dr?
Switch your keyspaces over to NetworkTopologyStrategy before adding the new nodes. For the strategy options, just list the first dc until the second is up (e.g. {main_dc: 3}).
Lastly, is changing snitch from default of SimpleSnitch to RackInferringSnitch going to cause any issues? Since its in the Cassandra.yaml file I assume a rolling restart to pick up the value would be ok?
This is the first thing you'll want to do. Unless your node IPs would naturally put all nodes in a DC in the same rack, I recommend using PropertyFileSnitch, explicitly using the same rack. (I tend to prefer PFSnitch regardless; it's harder to accidentally mess up.) A rolling restart is required to pick up the change. Make sure to fill out cassandra-topology.properties first if using PFSnitch.
This is all on Cassandra 1.1.4, Thanks for any help!
--
Tyler Hobbs
DataStax<http://datastax.com/>
RE: Expanding cluster to include a new DR datacenter
Posted by Bryce Godfrey <Br...@azaleos.com>.
So in an interesting turn of events, this works on my other 4 keyspaces but just not this 'EBonding' one which will not recognize the changes. I can probably get around this by dropping and re-creating this keyspace since its uptime is not too important for us.
[default@AlertStats] describe AlertStats;
Keyspace: AlertStats:
Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
Durable Writes: true
Options: [Fisher:3]
From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
Sent: Monday, August 27, 2012 3:50 PM
To: user@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter
Can you describe your schema again with TierPoint in it?
On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Same results. I restarted the node also to see if it just wasn't picking up the changes and it still shows Simple.
When I specify the DC for strategy_options I should be using the DC name from properfy file snitch right? Ours is "Fisher" and "TierPoint" so that's what I used.
From: Mohit Anchlia [mailto:mohitanchlia@gmail.com<ma...@gmail.com>]
Sent: Monday, August 27, 2012 1:21 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
In your update command is it possible to specify RF for both DC? You could just do DC1:2, DC2:0.
On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Show schema output show the simple strategy still
[default@unknown] show schema EBonding;
create keyspace EBonding
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 2}
and durable_writes = true;
This is the only thing I see in the system log at the time on all the nodes:
INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264) Writing Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305) Completed flushing /opt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-he-34817-Data.db (241 bytes) for commitlog p$
Should I turn the logging level up on something to see some more info maybe?
From: aaron morton [mailto:aaron@thelastpickle.com<ma...@thelastpickle.com>]
Sent: Monday, August 27, 2012 1:35 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
I did a quick test on a clean 1.1.4 and it worked
Can you check the logs for errors ? Can you see your schema change in there ?
Also what is the output from show schema; in the cli ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com<http://www.thelastpickle.com/>
On 25/08/2012, at 6:53 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Yes
[default@unknown] describe cluster;
Cluster Information:
Snitch: org.apache.cassandra.locator.PropertyFileSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, 10.20.8.1, 10.20.8.2, 10.20.8.3]
From: Mohit Anchlia [mailto:mohitanchlia@<mailto:mohitanchlia@>gmail.com<http://gmail.com/>]
Sent: Friday, August 24, 2012 1:55 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
That's interesting can you do describe cluster?
On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
So I'm at the point of updating the keyspaces from Simple to NetworkTopology and I'm not sure if the changes are being accepted using Cassandra-cli.
I issue the change:
[default@EBonding] update keyspace EBonding
... with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
... and strategy_options={Fisher:2};
9511e292-f1b6-3f78-b781-4c90aeb6b0f6
Waiting for schema agreement...
... schemas agree across the cluster
Then I do a describe and it still shows the old strategy. Is there something else that I need to do? I've exited and restarted Cassandra-cli and it still shows the SimpleStrategy for that keyspace. Other nodes show the same information.
[default@EBonding] describe EBonding;
Keyspace: EBonding:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Durable Writes: true
Options: [replication_factor:2]
From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com<ma...@azaleos.com>]
Sent: Thursday, August 23, 2012 11:06 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: RE: Expanding cluster to include a new DR datacenter
Thanks for the information! Answers my questions.
From: Tyler Hobbs [mailto:tyler@datastax.com]
Sent: Wednesday, August 22, 2012 7:10 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
If you didn't see this particular section, you may find it useful:http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
Some comments inline:
On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
We are in the process of building out a new DR system in another Data Center, and we want to mirror our Cassandra environment to that DR. I have a couple questions on the best way to do this after reading the documentation on the Datastax website. We didn't initially plan for this to be a DR setup when first deployed a while ago due to budgeting, but now we need to. So I'm just trying to nail down the order of doing this as well as any potential issues.
For the nodes, we don't plan on querying the servers in this DR until we fail over to this data center. We are going to have 5 similar nodes in the DR, should I join them into the ring at token+1?
Join them at token+10 just to leave a little space. Make sure you're using LOCAL_QUORUM for your queries instead of regular QUORUM.
All keyspaces are set to the replication strategy of SimpleStrategy. Can I change the replication strategy after joining the new nodes in the DR to NetworkTopologyStategy with the updated replication factor for each dr?
Switch your keyspaces over to NetworkTopologyStrategy before adding the new nodes. For the strategy options, just list the first dc until the second is up (e.g. {main_dc: 3}).
Lastly, is changing snitch from default of SimpleSnitch to RackInferringSnitch going to cause any issues? Since its in the Cassandra.yaml file I assume a rolling restart to pick up the value would be ok?
This is the first thing you'll want to do. Unless your node IPs would naturally put all nodes in a DC in the same rack, I recommend using PropertyFileSnitch, explicitly using the same rack. (I tend to prefer PFSnitch regardless; it's harder to accidentally mess up.) A rolling restart is required to pick up the change. Make sure to fill out cassandra-topology.properties first if using PFSnitch.
This is all on Cassandra 1.1.4, Thanks for any help!
--
Tyler Hobbs
DataStax<http://datastax.com/>
Re: Expanding cluster to include a new DR datacenter
Posted by Mohit Anchlia <mo...@gmail.com>.
Can you describe your schema again with TierPoint in it?
On Mon, Aug 27, 2012 at 3:22 PM, Bryce Godfrey <Br...@azaleos.com>wrote:
> Same results. I restarted the node also to see if it just wasn’t
> picking up the changes and it still shows Simple. ****
>
> ** **
>
> When I specify the DC for strategy_options I should be using the DC name
> from properfy file snitch right? Ours is “Fisher” and “TierPoint” so
> that’s what I used.****
>
> ** **
>
> *From:* Mohit Anchlia [mailto:mohitanchlia@gmail.com]
> *Sent:* Monday, August 27, 2012 1:21 PM
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
> ** **
>
> In your update command is it possible to specify RF for both DC? You could
> just do DC1:2, DC2:0.****
>
> On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey <Br...@azaleos.com>
> wrote:****
>
> Show schema output show the simple strategy still****
>
> [default@unknown] show schema EBonding;****
>
> create keyspace EBonding****
>
> with placement_strategy = 'SimpleStrategy'****
>
> and strategy_options = {replication_factor : 2}****
>
> and durable_writes = true;****
>
> ****
>
> This is the only thing I see in the system log at the time on all the
> nodes:****
>
> ****
>
> INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java
> (line 659) Enqueuing flush of Memtable-schema_keyspaces@1157216346(183/228
> serialized/live bytes, 4 ops)****
>
> INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264)
> Writing Memtable-schema_keyspaces@1157216346(183/228 serialized/live
> bytes, 4 ops)****
>
> INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305)
> Completed flushing
> /opt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-he-34817-Data.db
> (241 bytes) for commitlog p$****
>
> ****
>
> ****
>
> Should I turn the logging level up on something to see some more info
> maybe?****
>
> ****
>
> *From:* aaron morton [mailto:aaron@thelastpickle.com]
> *Sent:* Monday, August 27, 2012 1:35 AM ****
>
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
> ****
>
> I did a quick test on a clean 1.1.4 and it worked ****
>
> ****
>
> Can you check the logs for errors ? Can you see your schema change in
> there ?****
>
> ****
>
> Also what is the output from show schema; in the cli ? ****
>
> ****
>
> Cheers****
>
> ****
>
> -----------------****
>
> Aaron Morton****
>
> Freelance Developer****
>
> @aaronmorton****
>
> http://www.thelastpickle.com****
>
> ****
>
> On 25/08/2012, at 6:53 PM, Bryce Godfrey <Br...@azaleos.com>
> wrote:****
>
> ** **
>
> Yes****
>
> ****
>
> [default@unknown] describe cluster;****
>
> Cluster Information:****
>
> Snitch: org.apache.cassandra.locator.PropertyFileSnitch****
>
> Partitioner: org.apache.cassandra.dht.RandomPartitioner****
>
> Schema versions:****
>
> 9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5,
> 10.20.8.1, 10.20.8.2, 10.20.8.3]****
>
> ****
>
> *From:* Mohit Anchlia [mailto:mohitanchlia@gmail.com]
> *Sent:* Friday, August 24, 2012 1:55 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
> ****
>
> That's interesting can you do describe cluster?****
>
> On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <Br...@azaleos.com>
> wrote:****
>
> So I’m at the point of updating the keyspaces from Simple to
> NetworkTopology and I’m not sure if the changes are being accepted using
> Cassandra-cli.****
>
> ****
>
> I issue the change:****
>
> ****
>
> [default@EBonding] update keyspace EBonding****
>
> ... with placement_strategy =
> 'org.apache.cassandra.locator.NetworkTopologyStrategy'****
>
> ... and strategy_options={Fisher:2};****
>
> 9511e292-f1b6-3f78-b781-4c90aeb6b0f6****
>
> Waiting for schema agreement...****
>
> ... schemas agree across the cluster****
>
> ****
>
> Then I do a describe and it still shows the old strategy. Is there
> something else that I need to do? I’ve exited and restarted Cassandra-cli
> and it still shows the SimpleStrategy for that keyspace. Other nodes show
> the same information.****
>
> ****
>
> [default@EBonding] describe EBonding;****
>
> Keyspace: EBonding:****
>
> Replication Strategy: org.apache.cassandra.locator.SimpleStrategy****
>
> Durable Writes: true****
>
> Options: [replication_factor:2]****
>
> ****
>
> ****
>
> *From:* Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
> *Sent:* Thursday, August 23, 2012 11:06 AM
> *To:* user@cassandra.apache.org
> *Subject:* RE: Expanding cluster to include a new DR datacenter****
>
> ****
>
> Thanks for the information! Answers my questions.****
>
> ****
>
> *From:* Tyler Hobbs [mailto:tyler@datastax.com <ty...@datastax.com>]
> *Sent:* Wednesday, August 22, 2012 7:10 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
> ****
>
> If you didn't see this particular section, you may find it useful:
> http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
>
> Some comments inline:****
>
> On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Br...@azaleos.com>
> wrote:****
>
> We are in the process of building out a new DR system in another Data
> Center, and we want to mirror our Cassandra environment to that DR. I have
> a couple questions on the best way to do this after reading the
> documentation on the Datastax website. We didn’t initially plan for this
> to be a DR setup when first deployed a while ago due to budgeting, but now
> we need to. So I’m just trying to nail down the order of doing this as
> well as any potential issues.****
>
> ****
>
> For the nodes, we don’t plan on querying the servers in this DR until we
> fail over to this data center. We are going to have 5 similar nodes in
> the DR, should I join them into the ring at token+1?****
>
>
> Join them at token+10 just to leave a little space. Make sure you're
> using LOCAL_QUORUM for your queries instead of regular QUORUM.
> ****
>
> ****
>
> All keyspaces are set to the replication strategy of SimpleStrategy. Can
> I change the replication strategy after joining the new nodes in the DR to
> NetworkTopologyStategy with the updated replication factor for each dr?***
> *
>
>
> Switch your keyspaces over to NetworkTopologyStrategy before adding the
> new nodes. For the strategy options, just list the first dc until the
> second is up (e.g. {main_dc: 3}).
> ****
>
> ****
>
> Lastly, is changing snitch from default of SimpleSnitch to
> RackInferringSnitch going to cause any issues? Since its in the
> Cassandra.yaml file I assume a rolling restart to pick up the value would
> be ok?****
>
>
> This is the first thing you'll want to do. Unless your node IPs would
> naturally put all nodes in a DC in the same rack, I recommend using
> PropertyFileSnitch, explicitly using the same rack. (I tend to prefer
> PFSnitch regardless; it's harder to accidentally mess up.) A rolling
> restart is required to pick up the change. Make sure to fill out
> cassandra-topology.properties first if using PFSnitch.
> ****
>
> ****
>
> This is all on Cassandra 1.1.4, Thanks for any help!****
>
> ****
>
> ****
>
>
>
>
> --
> Tyler Hobbs
> DataStax <http://datastax.com/>****
>
> ****
>
> ** **
>
RE: Expanding cluster to include a new DR datacenter
Posted by Bryce Godfrey <Br...@azaleos.com>.
Same results. I restarted the node also to see if it just wasn't picking up the changes and it still shows Simple.
When I specify the DC for strategy_options I should be using the DC name from properfy file snitch right? Ours is "Fisher" and "TierPoint" so that's what I used.
From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
Sent: Monday, August 27, 2012 1:21 PM
To: user@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter
In your update command is it possible to specify RF for both DC? You could just do DC1:2, DC2:0.
On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Show schema output show the simple strategy still
[default@unknown] show schema EBonding;
create keyspace EBonding
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 2}
and durable_writes = true;
This is the only thing I see in the system log at the time on all the nodes:
INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264) Writing Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305) Completed flushing /opt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-he-34817-Data.db (241 bytes) for commitlog p$
Should I turn the logging level up on something to see some more info maybe?
From: aaron morton [mailto:aaron@thelastpickle.com<ma...@thelastpickle.com>]
Sent: Monday, August 27, 2012 1:35 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
I did a quick test on a clean 1.1.4 and it worked
Can you check the logs for errors ? Can you see your schema change in there ?
Also what is the output from show schema; in the cli ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com<http://www.thelastpickle.com/>
On 25/08/2012, at 6:53 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Yes
[default@unknown] describe cluster;
Cluster Information:
Snitch: org.apache.cassandra.locator.PropertyFileSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, 10.20.8.1, 10.20.8.2, 10.20.8.3]
From: Mohit Anchlia [mailto:mohitanchlia@<mailto:mohitanchlia@>gmail.com<http://gmail.com/>]
Sent: Friday, August 24, 2012 1:55 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
That's interesting can you do describe cluster?
On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
So I'm at the point of updating the keyspaces from Simple to NetworkTopology and I'm not sure if the changes are being accepted using Cassandra-cli.
I issue the change:
[default@EBonding] update keyspace EBonding
... with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
... and strategy_options={Fisher:2};
9511e292-f1b6-3f78-b781-4c90aeb6b0f6
Waiting for schema agreement...
... schemas agree across the cluster
Then I do a describe and it still shows the old strategy. Is there something else that I need to do? I've exited and restarted Cassandra-cli and it still shows the SimpleStrategy for that keyspace. Other nodes show the same information.
[default@EBonding] describe EBonding;
Keyspace: EBonding:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Durable Writes: true
Options: [replication_factor:2]
From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com<ma...@azaleos.com>]
Sent: Thursday, August 23, 2012 11:06 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: RE: Expanding cluster to include a new DR datacenter
Thanks for the information! Answers my questions.
From: Tyler Hobbs [mailto:tyler@datastax.com]
Sent: Wednesday, August 22, 2012 7:10 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
If you didn't see this particular section, you may find it useful:http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
Some comments inline:
On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
We are in the process of building out a new DR system in another Data Center, and we want to mirror our Cassandra environment to that DR. I have a couple questions on the best way to do this after reading the documentation on the Datastax website. We didn't initially plan for this to be a DR setup when first deployed a while ago due to budgeting, but now we need to. So I'm just trying to nail down the order of doing this as well as any potential issues.
For the nodes, we don't plan on querying the servers in this DR until we fail over to this data center. We are going to have 5 similar nodes in the DR, should I join them into the ring at token+1?
Join them at token+10 just to leave a little space. Make sure you're using LOCAL_QUORUM for your queries instead of regular QUORUM.
All keyspaces are set to the replication strategy of SimpleStrategy. Can I change the replication strategy after joining the new nodes in the DR to NetworkTopologyStategy with the updated replication factor for each dr?
Switch your keyspaces over to NetworkTopologyStrategy before adding the new nodes. For the strategy options, just list the first dc until the second is up (e.g. {main_dc: 3}).
Lastly, is changing snitch from default of SimpleSnitch to RackInferringSnitch going to cause any issues? Since its in the Cassandra.yaml file I assume a rolling restart to pick up the value would be ok?
This is the first thing you'll want to do. Unless your node IPs would naturally put all nodes in a DC in the same rack, I recommend using PropertyFileSnitch, explicitly using the same rack. (I tend to prefer PFSnitch regardless; it's harder to accidentally mess up.) A rolling restart is required to pick up the change. Make sure to fill out cassandra-topology.properties first if using PFSnitch.
This is all on Cassandra 1.1.4, Thanks for any help!
--
Tyler Hobbs
DataStax<http://datastax.com/>
Re: Expanding cluster to include a new DR datacenter
Posted by Mohit Anchlia <mo...@gmail.com>.
In your update command is it possible to specify RF for both DC? You could
just do DC1:2, DC2:0.
On Mon, Aug 27, 2012 at 11:16 AM, Bryce Godfrey
<Br...@azaleos.com>wrote:
> Show schema output show the simple strategy still****
>
> [default@unknown] show schema EBonding;****
>
> create keyspace EBonding****
>
> with placement_strategy = 'SimpleStrategy'****
>
> and strategy_options = {replication_factor : 2}****
>
> and durable_writes = true;****
>
> ** **
>
> This is the only thing I see in the system log at the time on all the
> nodes:****
>
> ** **
>
> INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java
> (line 659) Enqueuing flush of Memtable-schema_keyspaces@1157216346(183/228
> serialized/live bytes, 4 ops)****
>
> INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264)
> Writing Memtable-schema_keyspaces@1157216346(183/228 serialized/live
> bytes, 4 ops)****
>
> INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305)
> Completed flushing
> /opt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-he-34817-Data.db
> (241 bytes) for commitlog p$****
>
> ** **
>
> ** **
>
> Should I turn the logging level up on something to see some more info
> maybe?****
>
> ** **
>
> *From:* aaron morton [mailto:aaron@thelastpickle.com]
> *Sent:* Monday, August 27, 2012 1:35 AM
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
> ** **
>
> I did a quick test on a clean 1.1.4 and it worked ****
>
> ** **
>
> Can you check the logs for errors ? Can you see your schema change in
> there ?****
>
> ** **
>
> Also what is the output from show schema; in the cli ? ****
>
> ** **
>
> Cheers****
>
> ** **
>
> -----------------****
>
> Aaron Morton****
>
> Freelance Developer****
>
> @aaronmorton****
>
> http://www.thelastpickle.com****
>
> ** **
>
> On 25/08/2012, at 6:53 PM, Bryce Godfrey <Br...@azaleos.com>
> wrote:****
>
>
>
> ****
>
> Yes****
>
> ****
>
> [default@unknown] describe cluster;****
>
> Cluster Information:****
>
> Snitch: org.apache.cassandra.locator.PropertyFileSnitch****
>
> Partitioner: org.apache.cassandra.dht.RandomPartitioner****
>
> Schema versions:****
>
> 9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5,
> 10.20.8.1, 10.20.8.2, 10.20.8.3]****
>
> ****
>
> *From:* Mohit Anchlia [mailto:mohitanchlia@gmail.com]
> *Sent:* Friday, August 24, 2012 1:55 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
> ****
>
> That's interesting can you do describe cluster?****
>
> On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <Br...@azaleos.com>
> wrote:****
>
> So I’m at the point of updating the keyspaces from Simple to
> NetworkTopology and I’m not sure if the changes are being accepted using
> Cassandra-cli.****
>
> ****
>
> I issue the change:****
>
> ****
>
> [default@EBonding] update keyspace EBonding****
>
> ... with placement_strategy =
> 'org.apache.cassandra.locator.NetworkTopologyStrategy'****
>
> ... and strategy_options={Fisher:2};****
>
> 9511e292-f1b6-3f78-b781-4c90aeb6b0f6****
>
> Waiting for schema agreement...****
>
> ... schemas agree across the cluster****
>
> ****
>
> Then I do a describe and it still shows the old strategy. Is there
> something else that I need to do? I’ve exited and restarted Cassandra-cli
> and it still shows the SimpleStrategy for that keyspace. Other nodes show
> the same information.****
>
> ****
>
> [default@EBonding] describe EBonding;****
>
> Keyspace: EBonding:****
>
> Replication Strategy: org.apache.cassandra.locator.SimpleStrategy****
>
> Durable Writes: true****
>
> Options: [replication_factor:2]****
>
> ****
>
> ****
>
> *From:* Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
> *Sent:* Thursday, August 23, 2012 11:06 AM
> *To:* user@cassandra.apache.org
> *Subject:* RE: Expanding cluster to include a new DR datacenter****
>
> ****
>
> Thanks for the information! Answers my questions.****
>
> ****
>
> *From:* Tyler Hobbs [mailto:tyler@datastax.com <ty...@datastax.com>]
> *Sent:* Wednesday, August 22, 2012 7:10 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
> ****
>
> If you didn't see this particular section, you may find it useful:
> http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
>
> Some comments inline:****
>
> On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Br...@azaleos.com>
> wrote:****
>
> We are in the process of building out a new DR system in another Data
> Center, and we want to mirror our Cassandra environment to that DR. I have
> a couple questions on the best way to do this after reading the
> documentation on the Datastax website. We didn’t initially plan for this
> to be a DR setup when first deployed a while ago due to budgeting, but now
> we need to. So I’m just trying to nail down the order of doing this as
> well as any potential issues.****
>
> ****
>
> For the nodes, we don’t plan on querying the servers in this DR until we
> fail over to this data center. We are going to have 5 similar nodes in
> the DR, should I join them into the ring at token+1?****
>
>
> Join them at token+10 just to leave a little space. Make sure you're
> using LOCAL_QUORUM for your queries instead of regular QUORUM.
> ****
>
> ****
>
> All keyspaces are set to the replication strategy of SimpleStrategy. Can
> I change the replication strategy after joining the new nodes in the DR to
> NetworkTopologyStategy with the updated replication factor for each dr?***
> *
>
>
> Switch your keyspaces over to NetworkTopologyStrategy before adding the
> new nodes. For the strategy options, just list the first dc until the
> second is up (e.g. {main_dc: 3}).
> ****
>
> ****
>
> Lastly, is changing snitch from default of SimpleSnitch to
> RackInferringSnitch going to cause any issues? Since its in the
> Cassandra.yaml file I assume a rolling restart to pick up the value would
> be ok?****
>
>
> This is the first thing you'll want to do. Unless your node IPs would
> naturally put all nodes in a DC in the same rack, I recommend using
> PropertyFileSnitch, explicitly using the same rack. (I tend to prefer
> PFSnitch regardless; it's harder to accidentally mess up.) A rolling
> restart is required to pick up the change. Make sure to fill out
> cassandra-topology.properties first if using PFSnitch.
> ****
>
> ****
>
> This is all on Cassandra 1.1.4, Thanks for any help!****
>
> ****
>
> ****
>
>
>
>
> --
> Tyler Hobbs
> DataStax <http://datastax.com/>****
>
> ** **
>
RE: Expanding cluster to include a new DR datacenter
Posted by Bryce Godfrey <Br...@azaleos.com>.
Show schema output show the simple strategy still
[default@unknown] show schema EBonding;
create keyspace EBonding
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 2}
and durable_writes = true;
This is the only thing I see in the system log at the time on all the nodes:
INFO [MigrationStage:1] 2012-08-27 10:54:18,608 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
INFO [FlushWriter:765] 2012-08-27 10:54:18,612 Memtable.java (line 264) Writing Memtable-schema_keyspaces@1157216346(183/228 serialized/live bytes, 4 ops)
INFO [FlushWriter:765] 2012-08-27 10:54:18,627 Memtable.java (line 305) Completed flushing /opt/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-he-34817-Data.db (241 bytes) for commitlog p$
Should I turn the logging level up on something to see some more info maybe?
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Monday, August 27, 2012 1:35 AM
To: user@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter
I did a quick test on a clean 1.1.4 and it worked
Can you check the logs for errors ? Can you see your schema change in there ?
Also what is the output from show schema; in the cli ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 25/08/2012, at 6:53 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
Yes
[default@unknown] describe cluster;
Cluster Information:
Snitch: org.apache.cassandra.locator.PropertyFileSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, 10.20.8.1, 10.20.8.2, 10.20.8.3]
From: Mohit Anchlia [mailto:mohitanchlia@gmail.com<http://gmail.com>]
Sent: Friday, August 24, 2012 1:55 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
That's interesting can you do describe cluster?
On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
So I'm at the point of updating the keyspaces from Simple to NetworkTopology and I'm not sure if the changes are being accepted using Cassandra-cli.
I issue the change:
[default@EBonding] update keyspace EBonding
... with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
... and strategy_options={Fisher:2};
9511e292-f1b6-3f78-b781-4c90aeb6b0f6
Waiting for schema agreement...
... schemas agree across the cluster
Then I do a describe and it still shows the old strategy. Is there something else that I need to do? I've exited and restarted Cassandra-cli and it still shows the SimpleStrategy for that keyspace. Other nodes show the same information.
[default@EBonding] describe EBonding;
Keyspace: EBonding:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Durable Writes: true
Options: [replication_factor:2]
From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com<ma...@azaleos.com>]
Sent: Thursday, August 23, 2012 11:06 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: RE: Expanding cluster to include a new DR datacenter
Thanks for the information! Answers my questions.
From: Tyler Hobbs [mailto:tyler@datastax.com]
Sent: Wednesday, August 22, 2012 7:10 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
If you didn't see this particular section, you may find it useful:http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
Some comments inline:
On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
We are in the process of building out a new DR system in another Data Center, and we want to mirror our Cassandra environment to that DR. I have a couple questions on the best way to do this after reading the documentation on the Datastax website. We didn't initially plan for this to be a DR setup when first deployed a while ago due to budgeting, but now we need to. So I'm just trying to nail down the order of doing this as well as any potential issues.
For the nodes, we don't plan on querying the servers in this DR until we fail over to this data center. We are going to have 5 similar nodes in the DR, should I join them into the ring at token+1?
Join them at token+10 just to leave a little space. Make sure you're using LOCAL_QUORUM for your queries instead of regular QUORUM.
All keyspaces are set to the replication strategy of SimpleStrategy. Can I change the replication strategy after joining the new nodes in the DR to NetworkTopologyStategy with the updated replication factor for each dr?
Switch your keyspaces over to NetworkTopologyStrategy before adding the new nodes. For the strategy options, just list the first dc until the second is up (e.g. {main_dc: 3}).
Lastly, is changing snitch from default of SimpleSnitch to RackInferringSnitch going to cause any issues? Since its in the Cassandra.yaml file I assume a rolling restart to pick up the value would be ok?
This is the first thing you'll want to do. Unless your node IPs would naturally put all nodes in a DC in the same rack, I recommend using PropertyFileSnitch, explicitly using the same rack. (I tend to prefer PFSnitch regardless; it's harder to accidentally mess up.) A rolling restart is required to pick up the change. Make sure to fill out cassandra-topology.properties first if using PFSnitch.
This is all on Cassandra 1.1.4, Thanks for any help!
--
Tyler Hobbs
DataStax<http://datastax.com/>
Re: Expanding cluster to include a new DR datacenter
Posted by aaron morton <aa...@thelastpickle.com>.
I did a quick test on a clean 1.1.4 and it worked
Can you check the logs for errors ? Can you see your schema change in there ?
Also what is the output from show schema; in the cli ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 25/08/2012, at 6:53 PM, Bryce Godfrey <Br...@azaleos.com> wrote:
> Yes
>
> [default@unknown] describe cluster;
> Cluster Information:
> Snitch: org.apache.cassandra.locator.PropertyFileSnitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> 9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, 10.20.8.1, 10.20.8.2, 10.20.8.3]
>
> From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
> Sent: Friday, August 24, 2012 1:55 PM
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>
> That's interesting can you do describe cluster?
>
> On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <Br...@azaleos.com> wrote:
> So I’m at the point of updating the keyspaces from Simple to NetworkTopology and I’m not sure if the changes are being accepted using Cassandra-cli.
>
> I issue the change:
>
> [default@EBonding] update keyspace EBonding
> ... with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
> ... and strategy_options={Fisher:2};
> 9511e292-f1b6-3f78-b781-4c90aeb6b0f6
> Waiting for schema agreement...
> ... schemas agree across the cluster
>
> Then I do a describe and it still shows the old strategy. Is there something else that I need to do? I’ve exited and restarted Cassandra-cli and it still shows the SimpleStrategy for that keyspace. Other nodes show the same information.
>
> [default@EBonding] describe EBonding;
> Keyspace: EBonding:
> Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
> Durable Writes: true
> Options: [replication_factor:2]
>
>
> From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
> Sent: Thursday, August 23, 2012 11:06 AM
> To: user@cassandra.apache.org
> Subject: RE: Expanding cluster to include a new DR datacenter
>
> Thanks for the information! Answers my questions.
>
> From: Tyler Hobbs [mailto:tyler@datastax.com]
> Sent: Wednesday, August 22, 2012 7:10 PM
> To: user@cassandra.apache.org
> Subject: Re: Expanding cluster to include a new DR datacenter
>
> If you didn't see this particular section, you may find it useful:http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
>
> Some comments inline:
>
> On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Br...@azaleos.com> wrote:
> We are in the process of building out a new DR system in another Data Center, and we want to mirror our Cassandra environment to that DR. I have a couple questions on the best way to do this after reading the documentation on the Datastax website. We didn’t initially plan for this to be a DR setup when first deployed a while ago due to budgeting, but now we need to. So I’m just trying to nail down the order of doing this as well as any potential issues.
>
> For the nodes, we don’t plan on querying the servers in this DR until we fail over to this data center. We are going to have 5 similar nodes in the DR, should I join them into the ring at token+1?
>
> Join them at token+10 just to leave a little space. Make sure you're using LOCAL_QUORUM for your queries instead of regular QUORUM.
>
>
> All keyspaces are set to the replication strategy of SimpleStrategy. Can I change the replication strategy after joining the new nodes in the DR to NetworkTopologyStategy with the updated replication factor for each dr?
>
> Switch your keyspaces over to NetworkTopologyStrategy before adding the new nodes. For the strategy options, just list the first dc until the second is up (e.g. {main_dc: 3}).
>
>
> Lastly, is changing snitch from default of SimpleSnitch to RackInferringSnitch going to cause any issues? Since its in the Cassandra.yaml file I assume a rolling restart to pick up the value would be ok?
>
> This is the first thing you'll want to do. Unless your node IPs would naturally put all nodes in a DC in the same rack, I recommend using PropertyFileSnitch, explicitly using the same rack. (I tend to prefer PFSnitch regardless; it's harder to accidentally mess up.) A rolling restart is required to pick up the change. Make sure to fill out cassandra-topology.properties first if using PFSnitch.
>
>
> This is all on Cassandra 1.1.4, Thanks for any help!
>
>
>
>
>
> --
> Tyler Hobbs
> DataStax
>
RE: Expanding cluster to include a new DR datacenter
Posted by Bryce Godfrey <Br...@azaleos.com>.
Yes
[default@unknown] describe cluster;
Cluster Information:
Snitch: org.apache.cassandra.locator.PropertyFileSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
9511e292-f1b6-3f78-b781-4c90aeb6b0f6: [10.20.8.4, 10.20.8.5, 10.20.8.1, 10.20.8.2, 10.20.8.3]
From: Mohit Anchlia [mailto:mohitanchlia@gmail.com]
Sent: Friday, August 24, 2012 1:55 PM
To: user@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter
That's interesting can you do describe cluster?
On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
So I'm at the point of updating the keyspaces from Simple to NetworkTopology and I'm not sure if the changes are being accepted using Cassandra-cli.
I issue the change:
[default@EBonding] update keyspace EBonding
... with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
... and strategy_options={Fisher:2};
9511e292-f1b6-3f78-b781-4c90aeb6b0f6
Waiting for schema agreement...
... schemas agree across the cluster
Then I do a describe and it still shows the old strategy. Is there something else that I need to do? I've exited and restarted Cassandra-cli and it still shows the SimpleStrategy for that keyspace. Other nodes show the same information.
[default@EBonding] describe EBonding;
Keyspace: EBonding:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Durable Writes: true
Options: [replication_factor:2]
From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com<ma...@azaleos.com>]
Sent: Thursday, August 23, 2012 11:06 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: RE: Expanding cluster to include a new DR datacenter
Thanks for the information! Answers my questions.
From: Tyler Hobbs [mailto:tyler@datastax.com]
Sent: Wednesday, August 22, 2012 7:10 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
If you didn't see this particular section, you may find it useful: http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
Some comments inline:
On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
We are in the process of building out a new DR system in another Data Center, and we want to mirror our Cassandra environment to that DR. I have a couple questions on the best way to do this after reading the documentation on the Datastax website. We didn't initially plan for this to be a DR setup when first deployed a while ago due to budgeting, but now we need to. So I'm just trying to nail down the order of doing this as well as any potential issues.
For the nodes, we don't plan on querying the servers in this DR until we fail over to this data center. We are going to have 5 similar nodes in the DR, should I join them into the ring at token+1?
Join them at token+10 just to leave a little space. Make sure you're using LOCAL_QUORUM for your queries instead of regular QUORUM.
All keyspaces are set to the replication strategy of SimpleStrategy. Can I change the replication strategy after joining the new nodes in the DR to NetworkTopologyStategy with the updated replication factor for each dr?
Switch your keyspaces over to NetworkTopologyStrategy before adding the new nodes. For the strategy options, just list the first dc until the second is up (e.g. {main_dc: 3}).
Lastly, is changing snitch from default of SimpleSnitch to RackInferringSnitch going to cause any issues? Since its in the Cassandra.yaml file I assume a rolling restart to pick up the value would be ok?
This is the first thing you'll want to do. Unless your node IPs would naturally put all nodes in a DC in the same rack, I recommend using PropertyFileSnitch, explicitly using the same rack. (I tend to prefer PFSnitch regardless; it's harder to accidentally mess up.) A rolling restart is required to pick up the change. Make sure to fill out cassandra-topology.properties first if using PFSnitch.
This is all on Cassandra 1.1.4, Thanks for any help!
--
Tyler Hobbs
DataStax<http://datastax.com/>
Re: Expanding cluster to include a new DR datacenter
Posted by Mohit Anchlia <mo...@gmail.com>.
That's interesting can you do describe cluster?
On Fri, Aug 24, 2012 at 12:11 PM, Bryce Godfrey
<Br...@azaleos.com>wrote:
> So I’m at the point of updating the keyspaces from Simple to
> NetworkTopology and I’m not sure if the changes are being accepted using
> Cassandra-cli.****
>
> ** **
>
> I issue the change:****
>
> ** **
>
> [default@EBonding] update keyspace EBonding****
>
> ... with placement_strategy =
> 'org.apache.cassandra.locator.NetworkTopologyStrategy'****
>
> ... and strategy_options={Fisher:2};****
>
> 9511e292-f1b6-3f78-b781-4c90aeb6b0f6****
>
> Waiting for schema agreement...****
>
> ... schemas agree across the cluster****
>
> ** **
>
> Then I do a describe and it still shows the old strategy. Is there
> something else that I need to do? I’ve exited and restarted Cassandra-cli
> and it still shows the SimpleStrategy for that keyspace. Other nodes show
> the same information.****
>
> ** **
>
> [default@EBonding] describe EBonding;****
>
> Keyspace: EBonding:****
>
> Replication Strategy: org.apache.cassandra.locator.SimpleStrategy****
>
> Durable Writes: true****
>
> Options: [replication_factor:2]****
>
> ** **
>
> ** **
>
> *From:* Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
> *Sent:* Thursday, August 23, 2012 11:06 AM
> *To:* user@cassandra.apache.org
> *Subject:* RE: Expanding cluster to include a new DR datacenter****
>
> ** **
>
> Thanks for the information! Answers my questions.****
>
> ** **
>
> *From:* Tyler Hobbs [mailto:tyler@datastax.com <ty...@datastax.com>]
> *Sent:* Wednesday, August 22, 2012 7:10 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Expanding cluster to include a new DR datacenter****
>
> ** **
>
> If you didn't see this particular section, you may find it useful:
> http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
>
> Some comments inline:****
>
> On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Br...@azaleos.com>
> wrote:****
>
> We are in the process of building out a new DR system in another Data
> Center, and we want to mirror our Cassandra environment to that DR. I have
> a couple questions on the best way to do this after reading the
> documentation on the Datastax website. We didn’t initially plan for this
> to be a DR setup when first deployed a while ago due to budgeting, but now
> we need to. So I’m just trying to nail down the order of doing this as
> well as any potential issues.****
>
> ****
>
> For the nodes, we don’t plan on querying the servers in this DR until we
> fail over to this data center. We are going to have 5 similar nodes in
> the DR, should I join them into the ring at token+1?****
>
>
> Join them at token+10 just to leave a little space. Make sure you're
> using LOCAL_QUORUM for your queries instead of regular QUORUM.
> ****
>
> ****
>
> All keyspaces are set to the replication strategy of SimpleStrategy. Can
> I change the replication strategy after joining the new nodes in the DR to
> NetworkTopologyStategy with the updated replication factor for each dr?***
> *
>
>
> Switch your keyspaces over to NetworkTopologyStrategy before adding the
> new nodes. For the strategy options, just list the first dc until the
> second is up (e.g. {main_dc: 3}).
> ****
>
> ****
>
> Lastly, is changing snitch from default of SimpleSnitch to
> RackInferringSnitch going to cause any issues? Since its in the
> Cassandra.yaml file I assume a rolling restart to pick up the value would
> be ok?****
>
>
> This is the first thing you'll want to do. Unless your node IPs would
> naturally put all nodes in a DC in the same rack, I recommend using
> PropertyFileSnitch, explicitly using the same rack. (I tend to prefer
> PFSnitch regardless; it's harder to accidentally mess up.) A rolling
> restart is required to pick up the change. Make sure to fill out
> cassandra-topology.properties first if using PFSnitch.
> ****
>
> ****
>
> This is all on Cassandra 1.1.4, Thanks for any help!****
>
> ****
>
> ****
>
>
>
>
> --
> Tyler Hobbs
> DataStax <http://datastax.com/>****
>
RE: Expanding cluster to include a new DR datacenter
Posted by Bryce Godfrey <Br...@azaleos.com>.
So I'm at the point of updating the keyspaces from Simple to NetworkTopology and I'm not sure if the changes are being accepted using Cassandra-cli.
I issue the change:
[default@EBonding] update keyspace EBonding
... with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
... and strategy_options={Fisher:2};
9511e292-f1b6-3f78-b781-4c90aeb6b0f6
Waiting for schema agreement...
... schemas agree across the cluster
Then I do a describe and it still shows the old strategy. Is there something else that I need to do? I've exited and restarted Cassandra-cli and it still shows the SimpleStrategy for that keyspace. Other nodes show the same information.
[default@EBonding] describe EBonding;
Keyspace: EBonding:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Durable Writes: true
Options: [replication_factor:2]
From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
Sent: Thursday, August 23, 2012 11:06 AM
To: user@cassandra.apache.org
Subject: RE: Expanding cluster to include a new DR datacenter
Thanks for the information! Answers my questions.
From: Tyler Hobbs [mailto:tyler@datastax.com]
Sent: Wednesday, August 22, 2012 7:10 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Expanding cluster to include a new DR datacenter
If you didn't see this particular section, you may find it useful: http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
Some comments inline:
On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
We are in the process of building out a new DR system in another Data Center, and we want to mirror our Cassandra environment to that DR. I have a couple questions on the best way to do this after reading the documentation on the Datastax website. We didn't initially plan for this to be a DR setup when first deployed a while ago due to budgeting, but now we need to. So I'm just trying to nail down the order of doing this as well as any potential issues.
For the nodes, we don't plan on querying the servers in this DR until we fail over to this data center. We are going to have 5 similar nodes in the DR, should I join them into the ring at token+1?
Join them at token+10 just to leave a little space. Make sure you're using LOCAL_QUORUM for your queries instead of regular QUORUM.
All keyspaces are set to the replication strategy of SimpleStrategy. Can I change the replication strategy after joining the new nodes in the DR to NetworkTopologyStategy with the updated replication factor for each dr?
Switch your keyspaces over to NetworkTopologyStrategy before adding the new nodes. For the strategy options, just list the first dc until the second is up (e.g. {main_dc: 3}).
Lastly, is changing snitch from default of SimpleSnitch to RackInferringSnitch going to cause any issues? Since its in the Cassandra.yaml file I assume a rolling restart to pick up the value would be ok?
This is the first thing you'll want to do. Unless your node IPs would naturally put all nodes in a DC in the same rack, I recommend using PropertyFileSnitch, explicitly using the same rack. (I tend to prefer PFSnitch regardless; it's harder to accidentally mess up.) A rolling restart is required to pick up the change. Make sure to fill out cassandra-topology.properties first if using PFSnitch.
This is all on Cassandra 1.1.4, Thanks for any help!
--
Tyler Hobbs
DataStax<http://datastax.com/>
RE: Expanding cluster to include a new DR datacenter
Posted by Bryce Godfrey <Br...@azaleos.com>.
Thanks for the information! Answers my questions.
From: Tyler Hobbs [mailto:tyler@datastax.com]
Sent: Wednesday, August 22, 2012 7:10 PM
To: user@cassandra.apache.org
Subject: Re: Expanding cluster to include a new DR datacenter
If you didn't see this particular section, you may find it useful: http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
Some comments inline:
On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Br...@azaleos.com>> wrote:
We are in the process of building out a new DR system in another Data Center, and we want to mirror our Cassandra environment to that DR. I have a couple questions on the best way to do this after reading the documentation on the Datastax website. We didn't initially plan for this to be a DR setup when first deployed a while ago due to budgeting, but now we need to. So I'm just trying to nail down the order of doing this as well as any potential issues.
For the nodes, we don't plan on querying the servers in this DR until we fail over to this data center. We are going to have 5 similar nodes in the DR, should I join them into the ring at token+1?
Join them at token+10 just to leave a little space. Make sure you're using LOCAL_QUORUM for your queries instead of regular QUORUM.
All keyspaces are set to the replication strategy of SimpleStrategy. Can I change the replication strategy after joining the new nodes in the DR to NetworkTopologyStategy with the updated replication factor for each dr?
Switch your keyspaces over to NetworkTopologyStrategy before adding the new nodes. For the strategy options, just list the first dc until the second is up (e.g. {main_dc: 3}).
Lastly, is changing snitch from default of SimpleSnitch to RackInferringSnitch going to cause any issues? Since its in the Cassandra.yaml file I assume a rolling restart to pick up the value would be ok?
This is the first thing you'll want to do. Unless your node IPs would naturally put all nodes in a DC in the same rack, I recommend using PropertyFileSnitch, explicitly using the same rack. (I tend to prefer PFSnitch regardless; it's harder to accidentally mess up.) A rolling restart is required to pick up the change. Make sure to fill out cassandra-topology.properties first if using PFSnitch.
This is all on Cassandra 1.1.4, Thanks for any help!
--
Tyler Hobbs
DataStax<http://datastax.com/>
Re: Expanding cluster to include a new DR datacenter
Posted by Tyler Hobbs <ty...@datastax.com>.
If you didn't see this particular section, you may find it useful:
http://www.datastax.com/docs/1.1/operations/cluster_management#adding-a-data-center-to-a-cluster
Some comments inline:
On Wed, Aug 22, 2012 at 3:43 PM, Bryce Godfrey <Br...@azaleos.com>wrote:
> We are in the process of building out a new DR system in another Data
> Center, and we want to mirror our Cassandra environment to that DR. I have
> a couple questions on the best way to do this after reading the
> documentation on the Datastax website. We didn’t initially plan for this
> to be a DR setup when first deployed a while ago due to budgeting, but now
> we need to. So I’m just trying to nail down the order of doing this as
> well as any potential issues.****
>
> ** **
>
> For the nodes, we don’t plan on querying the servers in this DR until we
> fail over to this data center. We are going to have 5 similar nodes in
> the DR, should I join them into the ring at token+1?
>
Join them at token+10 just to leave a little space. Make sure you're using
LOCAL_QUORUM for your queries instead of regular QUORUM.
> ****
>
> ** **
>
> All keyspaces are set to the replication strategy of SimpleStrategy. Can
> I change the replication strategy after joining the new nodes in the DR to
> NetworkTopologyStategy with the updated replication factor for each dr?
>
Switch your keyspaces over to NetworkTopologyStrategy before adding the new
nodes. For the strategy options, just list the first dc until the second
is up (e.g. {main_dc: 3}).
> ****
>
> ** **
>
> Lastly, is changing snitch from default of SimpleSnitch to
> RackInferringSnitch going to cause any issues? Since its in the
> Cassandra.yaml file I assume a rolling restart to pick up the value would
> be ok?
>
This is the first thing you'll want to do. Unless your node IPs would
naturally put all nodes in a DC in the same rack, I recommend using
PropertyFileSnitch, explicitly using the same rack. (I tend to prefer
PFSnitch regardless; it's harder to accidentally mess up.) A rolling
restart is required to pick up the change. Make sure to fill out
cassandra-topology.properties first if using PFSnitch.
> ****
>
> ** **
>
> This is all on Cassandra 1.1.4, Thanks for any help!****
>
> ** **
>
> ** **
>
--
Tyler Hobbs
DataStax <http://datastax.com/>