You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Tom van den Berge <to...@gmail.com> on 2015/09/03 20:27:37 UTC
How to prevent queries being routed to new DC?
I want to start using vnodes in my cluster. To do so, I've set up a new
data center with the same number of nodes as the existing one, as described
in
http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configVnodesProduction_t.html.
The new DC is in the same physical location as the old one.
The problem I'm running into is that as soon as the nodes in the new data
center are started, the application that is using the nodes in the old data
center is frequently getting error messages because queries don't return
the expected data. I'm pretty sure this is because somehow these queries
are routed to the new, empty data center. The application is not connecting
to the nodes in the new DC.
I've tried two different things to prevent this:
1) Ensure that all queries use either LOCAL_ONE or LOCAL_QUORUM
consistency. Nevertheless, I'm still seeing failed queries.
2) Start the new nodes with -Dcassandra.join_ring=false, to prevent them
from participating in the cluster. Although they don't show up in nodetool
ring, I'm still seeing failed queries.
If I understand it correctly, both measures should prevent queries from
ending up in the new DC, but somehow they don't in my situation.
How is it possible that queries are routed to the new, emtpy data center?
And more importantly, how can I prevent it?
Thanks,
Tom
Re: How to prevent queries being routed to new DC?
Posted by Tom van den Berge <to...@gmail.com>.
Hi Anuj,
That could indeed explain reads on my new DC. However, what I'm seeing in
my client application is that every now and then, a read query does not
produce any result, while I'm sure that it should. If I understand the read
repair process correctly, it will never cause a read query fail to find a
replica, right?
On Tue, Sep 8, 2015 at 4:40 AM, Anuj Wadehra <an...@yahoo.co.in> wrote:
> Hi Tom,
>
> While reading data ( even at CL LOCAL_QUORUM), if data in different nodes
> required to meet CL in your local cluster doesnt match, data will be read
> from remote dc for read repair if read_repair_chance is not 0.
>
> Imp points:
> 1.If you are reading and writing at local_quorum you can set
> read_repair_chance to 0 to prevent cross dc read repair.
> 2. For enabling dc local read repairs you can use
> dclocal_read_repair_chance and set read_repair_chance to 0.
> 3. If you are experiencing frequent requests being routed due to digest
> mismatch you may need to investigate mutation drops in your cluster using
> tpstats.
>
> Refer to similar issue raised by us :
> https://issues.apache.org/jira/browse/CASSANDRA-8479
>
> Thanks
> Anuj
>
> Sent from Yahoo Mail on Android
> <https://overview.mail.yahoo.com/mobile/?.src=Android>
> ------------------------------
> *From*:"Tom van den Berge" <to...@drillster.com>
> *Date*:Tue, 8 Sep, 2015 at 1:31 am
> *Subject*:Re: How to prevent queries being routed to new DC?
>
> NetworkTopologyStrategy
>
> On Mon, Sep 7, 2015 at 4:39 PM, Ryan Svihla <rs...@foundev.pro> wrote:
>
>> What's your keyspace replication strategy?
>>
>> On Thu, Sep 3, 2015 at 3:16 PM Tom van den Berge <
>> tom.vandenberge@gmail.com> wrote:
>>
>>> Thanks for your help so far!
>>>
>>> I have some problems trying to understand the jira mentioned by Rob :(
>>>
>>> I'm currently trying to set up the first node in the new DC with
>>> auto_bootstrap = true. The node then becomes visible with status "joining",
>>> which (hopefully) prevents other DCs from sending queries to it.
>>>
>>> Do you think this will work?
>>>
>>>
>>>
>>> On Thu, Sep 3, 2015 at 9:46 PM, Robert Coli <rc...@eventbrite.com>
>>> wrote:
>>>
>>>> On Thu, Sep 3, 2015 at 12:25 PM, Bryan Cheng <br...@blockcypher.com>
>>>> wrote:
>>>>
>>>>> I'd recommend you enable tracing and do a few queries in a controlled
>>>>> environment to verify that queries are being routed to your new nodes.
>>>>> Provided you have followed the procedure outlined above (specifically, have
>>>>> set auto_bootstrap to false in your new cluster), rebuild has not been run,
>>>>> the application is not connecting to the new cluster, and all your queries
>>>>> are run at LOCAL_* quorum levels, I do not believe those queries should be
>>>>> routed to the new dc.
>>>>>
>>>>
>>>> Other than CASSANDRA-9753, this is true.
>>>>
>>>> https://issues.apache.org/jira/browse/CASSANDRA-9753 (Unresolved; ):
>>>> "LOCAL_QUORUM reads can block cross-DC if there is a digest mismatch"
>>>>
>>>> =Rob
>>>>
>>>>
>>> --
>> Regards,
>>
>> Ryan Svihla
>
>
>
>
> --
> Tom van den Berge
> Lead Software Engineer
> [image: Drillster] Middenburcht 136
> 3452 MT Vleuten
> Netherlands+31 30 755 53 30
> www.drillster.com [image: Follow us on Facebook] Follow us on Facebook
> <https://www.facebook.com/Drillster>
>
Re: How to prevent queries being routed to new DC?
Posted by Anuj Wadehra <an...@yahoo.co.in>.
Hi Tom,
While reading data ( even at CL LOCAL_QUORUM), if data in different nodes required to meet CL in your local cluster doesnt match, data will be read from remote dc for read repair if read_repair_chance is not 0.
Imp points:
1.If you are reading and writing at local_quorum you can set read_repair_chance to 0 to prevent cross dc read repair.
2. For enabling dc local read repairs you can use dclocal_read_repair_chance and set read_repair_chance to 0.
3. If you are experiencing frequent requests being routed due to digest mismatch you may need to investigate mutation drops in your cluster using tpstats.
Refer to similar issue raised by us :https://issues.apache.org/jira/browse/CASSANDRA-8479
Thanks
Anuj
Sent from Yahoo Mail on Android
From:"Tom van den Berge" <to...@drillster.com>
Date:Tue, 8 Sep, 2015 at 1:31 am
Subject:Re: How to prevent queries being routed to new DC?
NetworkTopologyStrategy
On Mon, Sep 7, 2015 at 4:39 PM, Ryan Svihla <rs...@foundev.pro> wrote:
What's your keyspace replication strategy?
On Thu, Sep 3, 2015 at 3:16 PM Tom van den Berge <to...@gmail.com> wrote:
Thanks for your help so far!
I have some problems trying to understand the jira mentioned by Rob :(
I'm currently trying to set up the first node in the new DC with auto_bootstrap = true. The node then becomes visible with status "joining", which (hopefully) prevents other DCs from sending queries to it.
Do you think this will work?
On Thu, Sep 3, 2015 at 9:46 PM, Robert Coli <rc...@eventbrite.com> wrote:
On Thu, Sep 3, 2015 at 12:25 PM, Bryan Cheng <br...@blockcypher.com> wrote:
I'd recommend you enable tracing and do a few queries in a controlled environment to verify that queries are being routed to your new nodes. Provided you have followed the procedure outlined above (specifically, have set auto_bootstrap to false in your new cluster), rebuild has not been run, the application is not connecting to the new cluster, and all your queries are run at LOCAL_* quorum levels, I do not believe those queries should be routed to the new dc.
Other than CASSANDRA-9753, this is true.
https://issues.apache.org/jira/browse/CASSANDRA-9753 (Unresolved; ): "LOCAL_QUORUM reads can block cross-DC if there is a digest mismatch"
=Rob
--
Regards,
Ryan Svihla
--
Tom van den Berge
Lead Software Engineer
 Middenburcht 136
3452 MT Vleuten
Netherlands+31 30 755 53 30
www.drillster.com  Follow us on Facebook
Re: How to prevent queries being routed to new DC?
Posted by Tom van den Berge <to...@drillster.com>.
NetworkTopologyStrategy
On Mon, Sep 7, 2015 at 4:39 PM, Ryan Svihla <rs...@foundev.pro> wrote:
> What's your keyspace replication strategy?
>
> On Thu, Sep 3, 2015 at 3:16 PM Tom van den Berge <
> tom.vandenberge@gmail.com> wrote:
>
>> Thanks for your help so far!
>>
>> I have some problems trying to understand the jira mentioned by Rob :(
>>
>> I'm currently trying to set up the first node in the new DC with
>> auto_bootstrap = true. The node then becomes visible with status "joining",
>> which (hopefully) prevents other DCs from sending queries to it.
>>
>> Do you think this will work?
>>
>>
>>
>> On Thu, Sep 3, 2015 at 9:46 PM, Robert Coli <rc...@eventbrite.com> wrote:
>>
>>> On Thu, Sep 3, 2015 at 12:25 PM, Bryan Cheng <br...@blockcypher.com>
>>> wrote:
>>>
>>>> I'd recommend you enable tracing and do a few queries in a controlled
>>>> environment to verify that queries are being routed to your new nodes.
>>>> Provided you have followed the procedure outlined above (specifically, have
>>>> set auto_bootstrap to false in your new cluster), rebuild has not been run,
>>>> the application is not connecting to the new cluster, and all your queries
>>>> are run at LOCAL_* quorum levels, I do not believe those queries should be
>>>> routed to the new dc.
>>>>
>>>
>>> Other than CASSANDRA-9753, this is true.
>>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-9753 (Unresolved; ):
>>> "LOCAL_QUORUM reads can block cross-DC if there is a digest mismatch"
>>>
>>> =Rob
>>>
>>>
>> --
> Regards,
>
> Ryan Svihla
--
Tom van den Berge
Lead Software Engineer
[image: Drillster] Middenburcht 136
3452 MT Vleuten
Netherlands+31 30 755 53 30
www.drillster.com [image: Follow us on Facebook] Follow us on Facebook
<https://www.facebook.com/Drillster>
Re: How to prevent queries being routed to new DC?
Posted by Ryan Svihla <rs...@foundev.pro>.
What's your keyspace replication strategy?
On Thu, Sep 3, 2015 at 3:16 PM Tom van den Berge <to...@gmail.com>
wrote:
> Thanks for your help so far!
>
> I have some problems trying to understand the jira mentioned by Rob :(
>
> I'm currently trying to set up the first node in the new DC with
> auto_bootstrap = true. The node then becomes visible with status "joining",
> which (hopefully) prevents other DCs from sending queries to it.
>
> Do you think this will work?
>
>
>
> On Thu, Sep 3, 2015 at 9:46 PM, Robert Coli <rc...@eventbrite.com> wrote:
>
>> On Thu, Sep 3, 2015 at 12:25 PM, Bryan Cheng <br...@blockcypher.com>
>> wrote:
>>
>>> I'd recommend you enable tracing and do a few queries in a controlled
>>> environment to verify that queries are being routed to your new nodes.
>>> Provided you have followed the procedure outlined above (specifically, have
>>> set auto_bootstrap to false in your new cluster), rebuild has not been run,
>>> the application is not connecting to the new cluster, and all your queries
>>> are run at LOCAL_* quorum levels, I do not believe those queries should be
>>> routed to the new dc.
>>>
>>
>> Other than CASSANDRA-9753, this is true.
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-9753 (Unresolved; ):
>> "LOCAL_QUORUM reads can block cross-DC if there is a digest mismatch"
>>
>> =Rob
>>
>>
> --
Regards,
Ryan Svihla
Re: How to prevent queries being routed to new DC?
Posted by Tom van den Berge <to...@gmail.com>.
Thanks for your help so far!
I have some problems trying to understand the jira mentioned by Rob :(
I'm currently trying to set up the first node in the new DC with
auto_bootstrap = true. The node then becomes visible with status "joining",
which (hopefully) prevents other DCs from sending queries to it.
Do you think this will work?
On Thu, Sep 3, 2015 at 9:46 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Thu, Sep 3, 2015 at 12:25 PM, Bryan Cheng <br...@blockcypher.com>
> wrote:
>
>> I'd recommend you enable tracing and do a few queries in a controlled
>> environment to verify that queries are being routed to your new nodes.
>> Provided you have followed the procedure outlined above (specifically, have
>> set auto_bootstrap to false in your new cluster), rebuild has not been run,
>> the application is not connecting to the new cluster, and all your queries
>> are run at LOCAL_* quorum levels, I do not believe those queries should be
>> routed to the new dc.
>>
>
> Other than CASSANDRA-9753, this is true.
>
> https://issues.apache.org/jira/browse/CASSANDRA-9753 (Unresolved; ):
> "LOCAL_QUORUM reads can block cross-DC if there is a digest mismatch"
>
> =Rob
>
>
Re: How to prevent queries being routed to new DC?
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Sep 3, 2015 at 12:25 PM, Bryan Cheng <br...@blockcypher.com> wrote:
> I'd recommend you enable tracing and do a few queries in a controlled
> environment to verify that queries are being routed to your new nodes.
> Provided you have followed the procedure outlined above (specifically, have
> set auto_bootstrap to false in your new cluster), rebuild has not been run,
> the application is not connecting to the new cluster, and all your queries
> are run at LOCAL_* quorum levels, I do not believe those queries should be
> routed to the new dc.
>
Other than CASSANDRA-9753, this is true.
https://issues.apache.org/jira/browse/CASSANDRA-9753 (Unresolved; ):
"LOCAL_QUORUM reads can block cross-DC if there is a digest mismatch"
=Rob
Re: How to prevent queries being routed to new DC?
Posted by Bryan Cheng <br...@blockcypher.com>.
Hey Tom,
I'd recommend you enable tracing and do a few queries in a controlled
environment to verify that queries are being routed to your new nodes.
Provided you have followed the procedure outlined above (specifically, have
set auto_bootstrap to false in your new cluster), rebuild has not been run,
the application is not connecting to the new cluster, and all your queries
are run at LOCAL_* quorum levels, I do not believe those queries should be
routed to the new dc.
On Thu, Sep 3, 2015 at 12:14 PM, Tom van den Berge <
tom.vandenberge@gmail.com> wrote:
> Hi Bryan,
>
> It does not generate any errors. A query for a specific row simply does
> not return the row if it is sent to a node in the new DC. This makes sense,
> because the node is still empty.
>
> On Thu, Sep 3, 2015 at 9:03 PM, Bryan Cheng <br...@blockcypher.com> wrote:
>
>> This all seems fine so far. Are you able to see what errors are being
>> returned?
>>
>> We had a similar issue where one of our secondary, less used keyspaces
>> was on a replication strategy that was not DC-aware, which was causing
>> errors about being unable to satisfy LOCAL_ONE and LOCAL_QUORUM quoroum
>> levels.
>>
>>
>> On Thu, Sep 3, 2015 at 11:53 AM, Tom van den Berge <
>> tom.vandenberge@gmail.com> wrote:
>>
>>> Hi Bryan,
>>>
>>> I'm using the PropertyFileSnitch, and it contains entries for all nodes
>>> in the old DC, and all nodes in the new DC. The replication factor for both
>>> DCs is 1.
>>>
>>> With the first approach I described, the new nodes join the cluster, and
>>> show up correctly under the new DC, so all seems to be fine.
>>> With the second approach (join_ring=false), they don't show up at all,
>>> which is also what I expected.
>>>
>>>
>>> On Thu, Sep 3, 2015 at 8:44 PM, Bryan Cheng <br...@blockcypher.com>
>>> wrote:
>>>
>>>> Hey Tom,
>>>>
>>>> What's your replication strategy look like? When your new nodes join
>>>> the ring, can you verify that they show up under a new DC and not as part
>>>> of the old?
>>>>
>>>> --Bryan
>>>>
>>>> On Thu, Sep 3, 2015 at 11:27 AM, Tom van den Berge <
>>>> tom.vandenberge@gmail.com> wrote:
>>>>
>>>>> I want to start using vnodes in my cluster. To do so, I've set up a
>>>>> new data center with the same number of nodes as the existing one, as
>>>>> described in
>>>>> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configVnodesProduction_t.html.
>>>>> The new DC is in the same physical location as the old one.
>>>>>
>>>>> The problem I'm running into is that as soon as the nodes in the new
>>>>> data center are started, the application that is using the nodes in the old
>>>>> data center is frequently getting error messages because queries don't
>>>>> return the expected data. I'm pretty sure this is because somehow these
>>>>> queries are routed to the new, empty data center. The application is not
>>>>> connecting to the nodes in the new DC.
>>>>>
>>>>> I've tried two different things to prevent this:
>>>>>
>>>>> 1) Ensure that all queries use either LOCAL_ONE or LOCAL_QUORUM
>>>>> consistency. Nevertheless, I'm still seeing failed queries.
>>>>> 2) Start the new nodes with -Dcassandra.join_ring=false, to prevent
>>>>> them from participating in the cluster. Although they don't show up in
>>>>> nodetool ring, I'm still seeing failed queries.
>>>>>
>>>>> If I understand it correctly, both measures should prevent queries
>>>>> from ending up in the new DC, but somehow they don't in my situation.
>>>>>
>>>>> How is it possible that queries are routed to the new, emtpy data
>>>>> center? And more importantly, how can I prevent it?
>>>>>
>>>>> Thanks,
>>>>> Tom
>>>>>
>>>>
>>>>
>>>
>>
>
Re: How to prevent queries being routed to new DC?
Posted by Tom van den Berge <to...@gmail.com>.
Hi Bryan,
It does not generate any errors. A query for a specific row simply does not
return the row if it is sent to a node in the new DC. This makes sense,
because the node is still empty.
On Thu, Sep 3, 2015 at 9:03 PM, Bryan Cheng <br...@blockcypher.com> wrote:
> This all seems fine so far. Are you able to see what errors are being
> returned?
>
> We had a similar issue where one of our secondary, less used keyspaces was
> on a replication strategy that was not DC-aware, which was causing errors
> about being unable to satisfy LOCAL_ONE and LOCAL_QUORUM quoroum levels.
>
>
> On Thu, Sep 3, 2015 at 11:53 AM, Tom van den Berge <
> tom.vandenberge@gmail.com> wrote:
>
>> Hi Bryan,
>>
>> I'm using the PropertyFileSnitch, and it contains entries for all nodes
>> in the old DC, and all nodes in the new DC. The replication factor for both
>> DCs is 1.
>>
>> With the first approach I described, the new nodes join the cluster, and
>> show up correctly under the new DC, so all seems to be fine.
>> With the second approach (join_ring=false), they don't show up at all,
>> which is also what I expected.
>>
>>
>> On Thu, Sep 3, 2015 at 8:44 PM, Bryan Cheng <br...@blockcypher.com>
>> wrote:
>>
>>> Hey Tom,
>>>
>>> What's your replication strategy look like? When your new nodes join the
>>> ring, can you verify that they show up under a new DC and not as part of
>>> the old?
>>>
>>> --Bryan
>>>
>>> On Thu, Sep 3, 2015 at 11:27 AM, Tom van den Berge <
>>> tom.vandenberge@gmail.com> wrote:
>>>
>>>> I want to start using vnodes in my cluster. To do so, I've set up a new
>>>> data center with the same number of nodes as the existing one, as described
>>>> in
>>>> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configVnodesProduction_t.html.
>>>> The new DC is in the same physical location as the old one.
>>>>
>>>> The problem I'm running into is that as soon as the nodes in the new
>>>> data center are started, the application that is using the nodes in the old
>>>> data center is frequently getting error messages because queries don't
>>>> return the expected data. I'm pretty sure this is because somehow these
>>>> queries are routed to the new, empty data center. The application is not
>>>> connecting to the nodes in the new DC.
>>>>
>>>> I've tried two different things to prevent this:
>>>>
>>>> 1) Ensure that all queries use either LOCAL_ONE or LOCAL_QUORUM
>>>> consistency. Nevertheless, I'm still seeing failed queries.
>>>> 2) Start the new nodes with -Dcassandra.join_ring=false, to prevent
>>>> them from participating in the cluster. Although they don't show up in
>>>> nodetool ring, I'm still seeing failed queries.
>>>>
>>>> If I understand it correctly, both measures should prevent queries from
>>>> ending up in the new DC, but somehow they don't in my situation.
>>>>
>>>> How is it possible that queries are routed to the new, emtpy data
>>>> center? And more importantly, how can I prevent it?
>>>>
>>>> Thanks,
>>>> Tom
>>>>
>>>
>>>
>>
>
Re: How to prevent queries being routed to new DC?
Posted by Bryan Cheng <br...@blockcypher.com>.
This all seems fine so far. Are you able to see what errors are being
returned?
We had a similar issue where one of our secondary, less used keyspaces was
on a replication strategy that was not DC-aware, which was causing errors
about being unable to satisfy LOCAL_ONE and LOCAL_QUORUM quoroum levels.
On Thu, Sep 3, 2015 at 11:53 AM, Tom van den Berge <
tom.vandenberge@gmail.com> wrote:
> Hi Bryan,
>
> I'm using the PropertyFileSnitch, and it contains entries for all nodes in
> the old DC, and all nodes in the new DC. The replication factor for both
> DCs is 1.
>
> With the first approach I described, the new nodes join the cluster, and
> show up correctly under the new DC, so all seems to be fine.
> With the second approach (join_ring=false), they don't show up at all,
> which is also what I expected.
>
>
> On Thu, Sep 3, 2015 at 8:44 PM, Bryan Cheng <br...@blockcypher.com> wrote:
>
>> Hey Tom,
>>
>> What's your replication strategy look like? When your new nodes join the
>> ring, can you verify that they show up under a new DC and not as part of
>> the old?
>>
>> --Bryan
>>
>> On Thu, Sep 3, 2015 at 11:27 AM, Tom van den Berge <
>> tom.vandenberge@gmail.com> wrote:
>>
>>> I want to start using vnodes in my cluster. To do so, I've set up a new
>>> data center with the same number of nodes as the existing one, as described
>>> in
>>> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configVnodesProduction_t.html.
>>> The new DC is in the same physical location as the old one.
>>>
>>> The problem I'm running into is that as soon as the nodes in the new
>>> data center are started, the application that is using the nodes in the old
>>> data center is frequently getting error messages because queries don't
>>> return the expected data. I'm pretty sure this is because somehow these
>>> queries are routed to the new, empty data center. The application is not
>>> connecting to the nodes in the new DC.
>>>
>>> I've tried two different things to prevent this:
>>>
>>> 1) Ensure that all queries use either LOCAL_ONE or LOCAL_QUORUM
>>> consistency. Nevertheless, I'm still seeing failed queries.
>>> 2) Start the new nodes with -Dcassandra.join_ring=false, to prevent them
>>> from participating in the cluster. Although they don't show up in nodetool
>>> ring, I'm still seeing failed queries.
>>>
>>> If I understand it correctly, both measures should prevent queries from
>>> ending up in the new DC, but somehow they don't in my situation.
>>>
>>> How is it possible that queries are routed to the new, emtpy data
>>> center? And more importantly, how can I prevent it?
>>>
>>> Thanks,
>>> Tom
>>>
>>
>>
>
Re: How to prevent queries being routed to new DC?
Posted by Tom van den Berge <to...@gmail.com>.
Hi Bryan,
I'm using the PropertyFileSnitch, and it contains entries for all nodes in
the old DC, and all nodes in the new DC. The replication factor for both
DCs is 1.
With the first approach I described, the new nodes join the cluster, and
show up correctly under the new DC, so all seems to be fine.
With the second approach (join_ring=false), they don't show up at all,
which is also what I expected.
On Thu, Sep 3, 2015 at 8:44 PM, Bryan Cheng <br...@blockcypher.com> wrote:
> Hey Tom,
>
> What's your replication strategy look like? When your new nodes join the
> ring, can you verify that they show up under a new DC and not as part of
> the old?
>
> --Bryan
>
> On Thu, Sep 3, 2015 at 11:27 AM, Tom van den Berge <
> tom.vandenberge@gmail.com> wrote:
>
>> I want to start using vnodes in my cluster. To do so, I've set up a new
>> data center with the same number of nodes as the existing one, as described
>> in
>> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configVnodesProduction_t.html.
>> The new DC is in the same physical location as the old one.
>>
>> The problem I'm running into is that as soon as the nodes in the new data
>> center are started, the application that is using the nodes in the old data
>> center is frequently getting error messages because queries don't return
>> the expected data. I'm pretty sure this is because somehow these queries
>> are routed to the new, empty data center. The application is not connecting
>> to the nodes in the new DC.
>>
>> I've tried two different things to prevent this:
>>
>> 1) Ensure that all queries use either LOCAL_ONE or LOCAL_QUORUM
>> consistency. Nevertheless, I'm still seeing failed queries.
>> 2) Start the new nodes with -Dcassandra.join_ring=false, to prevent them
>> from participating in the cluster. Although they don't show up in nodetool
>> ring, I'm still seeing failed queries.
>>
>> If I understand it correctly, both measures should prevent queries from
>> ending up in the new DC, but somehow they don't in my situation.
>>
>> How is it possible that queries are routed to the new, emtpy data center?
>> And more importantly, how can I prevent it?
>>
>> Thanks,
>> Tom
>>
>
>
Re: How to prevent queries being routed to new DC?
Posted by Bryan Cheng <br...@blockcypher.com>.
Hey Tom,
What's your replication strategy look like? When your new nodes join the
ring, can you verify that they show up under a new DC and not as part of
the old?
--Bryan
On Thu, Sep 3, 2015 at 11:27 AM, Tom van den Berge <
tom.vandenberge@gmail.com> wrote:
> I want to start using vnodes in my cluster. To do so, I've set up a new
> data center with the same number of nodes as the existing one, as described
> in
> http://docs.datastax.com/en/cassandra/2.0/cassandra/configuration/configVnodesProduction_t.html.
> The new DC is in the same physical location as the old one.
>
> The problem I'm running into is that as soon as the nodes in the new data
> center are started, the application that is using the nodes in the old data
> center is frequently getting error messages because queries don't return
> the expected data. I'm pretty sure this is because somehow these queries
> are routed to the new, empty data center. The application is not connecting
> to the nodes in the new DC.
>
> I've tried two different things to prevent this:
>
> 1) Ensure that all queries use either LOCAL_ONE or LOCAL_QUORUM
> consistency. Nevertheless, I'm still seeing failed queries.
> 2) Start the new nodes with -Dcassandra.join_ring=false, to prevent them
> from participating in the cluster. Although they don't show up in nodetool
> ring, I'm still seeing failed queries.
>
> If I understand it correctly, both measures should prevent queries from
> ending up in the new DC, but somehow they don't in my situation.
>
> How is it possible that queries are routed to the new, emtpy data center?
> And more importantly, how can I prevent it?
>
> Thanks,
> Tom
>