You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by suraj pasuparthy <su...@gmail.com> on 2017/05/16 20:19:26 UTC
Replication issue with Multi DC setup in cassandra
Hello,
I am tying to find a way to PREVENT just one of my keyspaces to not sync to
the other datacenter.
I have 2 datacenters setup this way :
Datacenter: DC:4.4.4.4
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 4.4.4.4 189.42 KiB 32 100.0%
939f5965-f9d5-4673-b1a4-29fa5ecae0f9 rack1
Datacenter: DC:4.4.4.5
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 4.4.4.5 218.29 KiB 32 100.0%
c0d1d859-7ae9-4ce9-a50f-ea316963dbb1 rack1
all my keyspaces have 1 copy in each DC and it works like a charm.
however, I have ONE keyspace that i do not want to sync. and i define the
keyspace this way:
CREATE KEYSPACE nosync WITH replication = {'class':
'NetworkTopologyStrategy', 'DC:4.4.4.4': '1', 'DC:4.4.4.5': '0'} AND
durable_writes = true;
and i still see that keyspace show up in DC:4.4.4.5
i even tried
CREATE KEYSPACE nosync WITH replication = {'class':
'NetworkTopologyStrategy', 'DC:4.4.4.4': '1'} AND durable_writes = true;
Same issue, i still see the keyspace show up in the DC:4.4.4.5
Could anyone help me figure this out?
Cheers
-Suraj
Re: Replication issue with Multi DC setup in cassandra
Posted by daemeon reiydelle <da...@gmail.com>.
Cqlsh looks at the cluster, not node
“All men dream, but not equally. Those who dream by night in the dusty
recesses of their minds wake up in the day to find it was vanity, but the
dreamers of the day are dangerous men, for they may act their dreams with
open eyes, to make it possible.” — T.E. Lawrence
sent from my mobile
Daemeon Reiydelle
skype daemeon.c.m.reiydelle
USA 415.501.0198
On May 16, 2017 2:42 PM, "suraj pasuparthy" <su...@gmail.com>
wrote:
> So i though the same,
> I see the data via the CQLSH in both the datacenters. consistency is set
> to LQ
>
> thanks
> -Suraj
>
> On Tue, May 16, 2017 at 2:19 PM, Nitan Kainth <ni...@bamlabs.com> wrote:
>
>> Do you see data on other DC or just directory structure? Directory
>> structure would populate because it is DDL but inserts shouldn’t populate,
>> ideally.
>>
>> On May 16, 2017, at 3:19 PM, suraj pasuparthy <su...@gmail.com>
>> wrote:
>>
>> elp me fig
>>
>>
>>
>
>
> --
> Suraj Pasuparthy
>
> cisco systems
> Software Engineer
> San Jose CA
>
Re: Replication issue with Multi DC setup in cassandra
Posted by Arvydas Jonusonis <ar...@gmail.com>.
Run *nodetool cleanup* on the *4.4.4.5* DC node(s). Changing network
topology does not *remove* data - it's a manual task.
But it should prevent it from replicating over to the undesired DC.
Also make sure your LoadBalancingStrategy is set to DCAwareRoundRobinPolicy,
with *4.4.4.4* DC set as the *local* DC.
Arvydas
On Wed, May 24, 2017 at 9:46 PM, daemeon reiydelle <da...@gmail.com>
wrote:
> May I inquire if your configuration is actually data center aware? Do you
> understand the difference between LQ and replication?
>
>
>
>
>
> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <(415)%20501-0198>London
> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>
>
> *“All men dream, but not equally. Those who dream by night in the dusty
> recesses of their minds wake up in the day to find it was vanity, but the
> dreamers of the day are dangerous men, for they may act their dreams with
> open eyes, to make it possible.” — T.E. Lawrence*
>
>
> On Wed, May 24, 2017 at 12:03 PM, Igor Leão <ig...@ubee.in> wrote:
>
>> Did you run `nodetool repair` after changing the keyspace? (not sure if
>> it makes sense though)
>>
>> 2017-05-16 19:52 GMT-03:00 Nitan Kainth <ni...@bamlabs.com>:
>>
>>> Strange. Anybody else might share something more important.
>>>
>>> Sent from my iPhone
>>>
>>> On May 16, 2017, at 5:23 PM, suraj pasuparthy <
>>> suraj.pasuparthy@gmail.com> wrote:
>>>
>>> Yes is see them in the datacenter's data directories.. infact i see then
>>> even after i bring down the interface between the 2 DC's which further
>>> confirms that a local copy is maintained in the DC that was not configured
>>> in the strategy ..
>>> its quite important that we block the info for this keyspace from
>>> replicating :(.. not sure why this does not work
>>>
>>> Thanks
>>> Suraj
>>>
>>> On Tue, May 16, 2017 at 3:06 PM Nitan Kainth <ni...@bamlabs.com> wrote:
>>>
>>>> check for datafiles on filesystem in both DCs.
>>>>
>>>> On May 16, 2017, at 4:42 PM, suraj pasuparthy <
>>>> suraj.pasuparthy@gmail.com> wrote:
>>>>
>>>> So i though the same,
>>>> I see the data via the CQLSH in both the datacenters. consistency is
>>>> set to LQ
>>>>
>>>> thanks
>>>> -Suraj
>>>>
>>>> On Tue, May 16, 2017 at 2:19 PM, Nitan Kainth <ni...@bamlabs.com>
>>>> wrote:
>>>>
>>>>> Do you see data on other DC or just directory structure? Directory
>>>>> structure would populate because it is DDL but inserts shouldn’t populate,
>>>>> ideally.
>>>>>
>>>>> On May 16, 2017, at 3:19 PM, suraj pasuparthy <
>>>>> suraj.pasuparthy@gmail.com> wrote:
>>>>>
>>>>> elp me fig
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Suraj Pasuparthy
>>>>
>>>> cisco systems
>>>> Software Engineer
>>>> San Jose CA
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>>
>> --
>> Igor Leão Site Reliability Engineer
>>
>> Mobile: +55 81 99727-1083 <callto://+5581997271083>
>> Skype: *igorvpcleao*
>> Office: +55 81 4042-9757 <callto://+558140429757>
>> Website: inlocomedia.com <http://www.inlocomedia.com/>
>> [image: inlocomedia]
>> <http://t.sidekickopen29.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XX45w6CwnN7dSpvzQZpw8W8pTc_456dVQFdQm8LT02?t=http%3A%2F%2Fwww.inlocomedia.com%2F&si=4991638468296704&pi=9266b53b-57c9-4b38-d81a-d2f8f01ed355>
>> [image: LinkedIn]
>> <http://t.sidekickopen29.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XX45w6CwnN7dSpvzQZpw8W8pTc_456dVQFdQm8LT02?t=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fin-loco-media&si=4991638468296704&pi=9266b53b-57c9-4b38-d81a-d2f8f01ed355>
>> [image: Facebook] <https://www.facebook.com/inlocomedia> [image:
>> Twitter]
>> <http://t.sidekickopen29.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XX45w6CwnN7dSpvzQZpw8W8pTc_456dVQFdQm8LT02?t=https%3A%2F%2Ftwitter.com%2Finlocomedia&si=4991638468296704&pi=9266b53b-57c9-4b38-d81a-d2f8f01ed355>
>>
>>
>>
>>
>>
>>
>>
>
Re: Replication issue with Multi DC setup in cassandra
Posted by daemeon reiydelle <da...@gmail.com>.
May I inquire if your configuration is actually data center aware? Do you
understand the difference between LQ and replication?
*Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872*
*“All men dream, but not equally. Those who dream by night in the dusty
recesses of their minds wake up in the day to find it was vanity, but the
dreamers of the day are dangerous men, for they may act their dreams with
open eyes, to make it possible.” — T.E. Lawrence*
On Wed, May 24, 2017 at 12:03 PM, Igor Leão <ig...@ubee.in> wrote:
> Did you run `nodetool repair` after changing the keyspace? (not sure if it
> makes sense though)
>
> 2017-05-16 19:52 GMT-03:00 Nitan Kainth <ni...@bamlabs.com>:
>
>> Strange. Anybody else might share something more important.
>>
>> Sent from my iPhone
>>
>> On May 16, 2017, at 5:23 PM, suraj pasuparthy <su...@gmail.com>
>> wrote:
>>
>> Yes is see them in the datacenter's data directories.. infact i see then
>> even after i bring down the interface between the 2 DC's which further
>> confirms that a local copy is maintained in the DC that was not configured
>> in the strategy ..
>> its quite important that we block the info for this keyspace from
>> replicating :(.. not sure why this does not work
>>
>> Thanks
>> Suraj
>>
>> On Tue, May 16, 2017 at 3:06 PM Nitan Kainth <ni...@bamlabs.com> wrote:
>>
>>> check for datafiles on filesystem in both DCs.
>>>
>>> On May 16, 2017, at 4:42 PM, suraj pasuparthy <
>>> suraj.pasuparthy@gmail.com> wrote:
>>>
>>> So i though the same,
>>> I see the data via the CQLSH in both the datacenters. consistency is set
>>> to LQ
>>>
>>> thanks
>>> -Suraj
>>>
>>> On Tue, May 16, 2017 at 2:19 PM, Nitan Kainth <ni...@bamlabs.com> wrote:
>>>
>>>> Do you see data on other DC or just directory structure? Directory
>>>> structure would populate because it is DDL but inserts shouldn’t populate,
>>>> ideally.
>>>>
>>>> On May 16, 2017, at 3:19 PM, suraj pasuparthy <
>>>> suraj.pasuparthy@gmail.com> wrote:
>>>>
>>>> elp me fig
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Suraj Pasuparthy
>>>
>>> cisco systems
>>> Software Engineer
>>> San Jose CA
>>>
>>>
>>>
>>>
>>>
>>>
>
>
> --
> Igor Leão Site Reliability Engineer
>
> Mobile: +55 81 99727-1083 <callto://+5581997271083>
> Skype: *igorvpcleao*
> Office: +55 81 4042-9757 <callto://+558140429757>
> Website: inlocomedia.com <http://www.inlocomedia.com/>
> [image: inlocomedia]
> <http://t.sidekickopen29.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XX45w6CwnN7dSpvzQZpw8W8pTc_456dVQFdQm8LT02?t=http%3A%2F%2Fwww.inlocomedia.com%2F&si=4991638468296704&pi=9266b53b-57c9-4b38-d81a-d2f8f01ed355>
> [image: LinkedIn]
> <http://t.sidekickopen29.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XX45w6CwnN7dSpvzQZpw8W8pTc_456dVQFdQm8LT02?t=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fin-loco-media&si=4991638468296704&pi=9266b53b-57c9-4b38-d81a-d2f8f01ed355>
> [image: Facebook] <https://www.facebook.com/inlocomedia> [image: Twitter]
> <http://t.sidekickopen29.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XX45w6CwnN7dSpvzQZpw8W8pTc_456dVQFdQm8LT02?t=https%3A%2F%2Ftwitter.com%2Finlocomedia&si=4991638468296704&pi=9266b53b-57c9-4b38-d81a-d2f8f01ed355>
>
>
>
>
>
>
>
Re: Replication issue with Multi DC setup in cassandra
Posted by Igor Leão <ig...@ubee.in>.
Did you run `nodetool repair` after changing the keyspace? (not sure if it
makes sense though)
2017-05-16 19:52 GMT-03:00 Nitan Kainth <ni...@bamlabs.com>:
> Strange. Anybody else might share something more important.
>
> Sent from my iPhone
>
> On May 16, 2017, at 5:23 PM, suraj pasuparthy <su...@gmail.com>
> wrote:
>
> Yes is see them in the datacenter's data directories.. infact i see then
> even after i bring down the interface between the 2 DC's which further
> confirms that a local copy is maintained in the DC that was not configured
> in the strategy ..
> its quite important that we block the info for this keyspace from
> replicating :(.. not sure why this does not work
>
> Thanks
> Suraj
>
> On Tue, May 16, 2017 at 3:06 PM Nitan Kainth <ni...@bamlabs.com> wrote:
>
>> check for datafiles on filesystem in both DCs.
>>
>> On May 16, 2017, at 4:42 PM, suraj pasuparthy <su...@gmail.com>
>> wrote:
>>
>> So i though the same,
>> I see the data via the CQLSH in both the datacenters. consistency is set
>> to LQ
>>
>> thanks
>> -Suraj
>>
>> On Tue, May 16, 2017 at 2:19 PM, Nitan Kainth <ni...@bamlabs.com> wrote:
>>
>>> Do you see data on other DC or just directory structure? Directory
>>> structure would populate because it is DDL but inserts shouldn’t populate,
>>> ideally.
>>>
>>> On May 16, 2017, at 3:19 PM, suraj pasuparthy <
>>> suraj.pasuparthy@gmail.com> wrote:
>>>
>>> elp me fig
>>>
>>>
>>>
>>
>>
>> --
>> Suraj Pasuparthy
>>
>> cisco systems
>> Software Engineer
>> San Jose CA
>>
>>
>>
>>
>>
>>
--
Igor Leão Site Reliability Engineer
Mobile: +55 81 99727-1083 <callto://+5581997271083>
Skype: *igorvpcleao*
Office: +55 81 4042-9757 <callto://+558140429757>
Website: inlocomedia.com <http://www.inlocomedia.com/>
[image: inlocomedia]
<http://t.sidekickopen29.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XX45w6CwnN7dSpvzQZpw8W8pTc_456dVQFdQm8LT02?t=http%3A%2F%2Fwww.inlocomedia.com%2F&si=4991638468296704&pi=9266b53b-57c9-4b38-d81a-d2f8f01ed355>
[image: LinkedIn]
<http://t.sidekickopen29.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XX45w6CwnN7dSpvzQZpw8W8pTc_456dVQFdQm8LT02?t=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fin-loco-media&si=4991638468296704&pi=9266b53b-57c9-4b38-d81a-d2f8f01ed355>
[image: Facebook] <https://www.facebook.com/inlocomedia> [image: Twitter]
<http://t.sidekickopen29.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XX45w6CwnN7dSpvzQZpw8W8pTc_456dVQFdQm8LT02?t=https%3A%2F%2Ftwitter.com%2Finlocomedia&si=4991638468296704&pi=9266b53b-57c9-4b38-d81a-d2f8f01ed355>
Re: Replication issue with Multi DC setup in cassandra
Posted by Nitan Kainth <ni...@bamlabs.com>.
Strange. Anybody else might share something more important.
Sent from my iPhone
> On May 16, 2017, at 5:23 PM, suraj pasuparthy <su...@gmail.com> wrote:
>
> Yes is see them in the datacenter's data directories.. infact i see then even after i bring down the interface between the 2 DC's which further confirms that a local copy is maintained in the DC that was not configured in the strategy ..
> its quite important that we block the info for this keyspace from replicating :(.. not sure why this does not work
>
> Thanks
> Suraj
>
>> On Tue, May 16, 2017 at 3:06 PM Nitan Kainth <ni...@bamlabs.com> wrote:
>> check for datafiles on filesystem in both DCs.
>>
>>> On May 16, 2017, at 4:42 PM, suraj pasuparthy <su...@gmail.com> wrote:
>>>
>>> So i though the same,
>>> I see the data via the CQLSH in both the datacenters. consistency is set to LQ
>>>
>>> thanks
>>> -Suraj
>>>
>>>> On Tue, May 16, 2017 at 2:19 PM, Nitan Kainth <ni...@bamlabs.com> wrote:
>>>> Do you see data on other DC or just directory structure? Directory structure would populate because it is DDL but inserts shouldn’t populate, ideally.
>>>>
>>>>> On May 16, 2017, at 3:19 PM, suraj pasuparthy <su...@gmail.com> wrote:
>>>>>
>>>>> elp me fig
>>>>
>>>
>>>
>>>
>>> --
>>> Suraj Pasuparthy
>>>
>>> cisco systems
>>> Software Engineer
>>> San Jose CA
>>>
>>>
>>>
>>>
>>
Re: Replication issue with Multi DC setup in cassandra
Posted by suraj pasuparthy <su...@gmail.com>.
Yes is see them in the datacenter's data directories.. infact i see then
even after i bring down the interface between the 2 DC's which further
confirms that a local copy is maintained in the DC that was not configured
in the strategy ..
its quite important that we block the info for this keyspace from
replicating :(.. not sure why this does not work
Thanks
Suraj
On Tue, May 16, 2017 at 3:06 PM Nitan Kainth <ni...@bamlabs.com> wrote:
> check for datafiles on filesystem in both DCs.
>
> On May 16, 2017, at 4:42 PM, suraj pasuparthy <su...@gmail.com>
> wrote:
>
> So i though the same,
> I see the data via the CQLSH in both the datacenters. consistency is set
> to LQ
>
> thanks
> -Suraj
>
> On Tue, May 16, 2017 at 2:19 PM, Nitan Kainth <ni...@bamlabs.com> wrote:
>
>> Do you see data on other DC or just directory structure? Directory
>> structure would populate because it is DDL but inserts shouldn’t populate,
>> ideally.
>>
>> On May 16, 2017, at 3:19 PM, suraj pasuparthy <su...@gmail.com>
>> wrote:
>>
>> elp me fig
>>
>>
>>
>
>
> --
> Suraj Pasuparthy
>
> cisco systems
> Software Engineer
> San Jose CA
>
>
>
>
>
>
Re: Replication issue with Multi DC setup in cassandra
Posted by Nitan Kainth <ni...@bamlabs.com>.
check for datafiles on filesystem in both DCs.
> On May 16, 2017, at 4:42 PM, suraj pasuparthy <su...@gmail.com> wrote:
>
> So i though the same,
> I see the data via the CQLSH in both the datacenters. consistency is set to LQ
>
> thanks
> -Suraj
>
> On Tue, May 16, 2017 at 2:19 PM, Nitan Kainth <nitan@bamlabs.com <ma...@bamlabs.com>> wrote:
> Do you see data on other DC or just directory structure? Directory structure would populate because it is DDL but inserts shouldn’t populate, ideally.
>
>> On May 16, 2017, at 3:19 PM, suraj pasuparthy <suraj.pasuparthy@gmail.com <ma...@gmail.com>> wrote:
>>
>> elp me fig
>
>
>
>
> --
> Suraj Pasuparthy
>
> cisco systems
> Software Engineer
> San Jose CA
Re: Replication issue with Multi DC setup in cassandra
Posted by suraj pasuparthy <su...@gmail.com>.
So i though the same,
I see the data via the CQLSH in both the datacenters. consistency is set to
LQ
thanks
-Suraj
On Tue, May 16, 2017 at 2:19 PM, Nitan Kainth <ni...@bamlabs.com> wrote:
> Do you see data on other DC or just directory structure? Directory
> structure would populate because it is DDL but inserts shouldn’t populate,
> ideally.
>
> On May 16, 2017, at 3:19 PM, suraj pasuparthy <su...@gmail.com>
> wrote:
>
> elp me fig
>
>
>
--
Suraj Pasuparthy
cisco systems
Software Engineer
San Jose CA
Re: Replication issue with Multi DC setup in cassandra
Posted by Nitan Kainth <ni...@bamlabs.com>.
Do you see data on other DC or just directory structure? Directory structure would populate because it is DDL but inserts shouldn’t populate, ideally.
> On May 16, 2017, at 3:19 PM, suraj pasuparthy <su...@gmail.com> wrote:
>
> elp me fig