Keyspace Replication changes not synchronized after adding Datacenter

[Thunder Stumpges <th...@gmail.com>]

Hi guys,

We have recently added two datacenters to our existing 2.0.6 cluster. We
followed the process here pretty much exactly:
http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html

We are using GossipingPropertyFileSnitch and NetworkTopologyStrategy across
the board. All property files are identical in each of the three
datacenters, and we use two nodes from each DC in the seed list.

However when we came to step 7.a. we ran the ALTER KEYSPACE command on one
of the new datacenters (to add it as a replica). This change was reflected
on the new datacenter where it ran as returned by DESCRIBE KEYSPACE.
However the change was NOT propagated to either of the other two
datacenters. We effectively had to run the ALTER KEYSPACE command 3 times,
one in each datacenter. Is this expected? I could find no documentation
stating that this needed to be done, nor any documentation around how the
system keyspace was kept in sync across datacenters in general.

If this is indicative of a larger problem with our installation, how would
we go about troubleshooting it?

Thanks in advance!
Thunder

Re: Keyspace Replication changes not synchronized after adding Datacenter

[Paul Leddy <pa...@c3energy.com>]

Hello,

No, that is not expected at all, to run the alter statement in each DC. 
Yes, it indicates a larger problem, for sure.

Check that ports are open between all nodes, especially 7000, if I 
recall correctly. We use a simple telnet check.

Paul

On 04/13/2015 10:22 AM, Thunder Stumpges wrote:
> Hi guys,
>
> We have recently added two datacenters to our existing 2.0.6 cluster. 
> We followed the process here pretty much exactly:
> http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html
>
> We are using GossipingPropertyFileSnitch and NetworkTopologyStrategy 
> across the board. All property files are identical in each of the 
> three datacenters, and we use two nodes from each DC in the seed list.
>
> However when we came to step 7.a. we ran the ALTER KEYSPACE command on 
> one of the new datacenters (to add it as a replica). This change was 
> reflected on the new datacenter where it ran as returned by DESCRIBE 
> KEYSPACE. However the change was NOT propagated to either of the other 
> two datacenters. We effectively had to run the ALTER KEYSPACE command 
> 3 times, one in each datacenter. Is this expected? I could find no 
> documentation stating that this needed to be done, nor any 
> documentation around how the system keyspace was kept in sync across 
> datacenters in general.
>
> If this is indicative of a larger problem with our installation, how 
> would we go about troubleshooting it?
>
> Thanks in advance!
> Thunder
>
>