You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by srmore <co...@gmail.com> on 2013/06/03 22:09:38 UTC
Consistency level for multi-datacenter setup
I am a bit confused when using the consistency level for multi datacenter
setup. Following is my setup:
I have 4 nodes the way these are set up are
Node 1 DC 1 - N1DC1
Node 2 DC 1 - N2DC1
Node 1 DC 2 - N1DC2
Node 2 DC 2 - N2DC2
I setup a delay in between two datacenters (DC1 and DC2 around 1 sec one
way)
I am observing that when I use consistency level 2 for some reason the
coordinate node is picking up the nodes from other datacenter. My
understanding was that Cassandra picks up nodes which are close by (from
local datacenter), determined by Gossip but looks like that's not the case.
I found the following comment on Datastax website :
"If using a consistency level of ONE or LOCAL_QUORUM, only the nodes in the
same data center as the coordinator node must respond to the client request
in order for the request to succeed."
Does this mean that for multi datacenter we can only use ONE or
LOCAL_QUORUM if we want to use the local datacenter to avoid cross
datacenter latency.
I am using the GossipingPropertyFileSnitch.
Thanks !
Re: Consistency level for multi-datacenter setup
Posted by srmore <co...@gmail.com>.
Yup, RF is 2 for both the datacenters.
On Mon, Jun 3, 2013 at 3:36 PM, Sylvain Lebresne <sy...@datastax.com>wrote:
> What's your replication factor? Do you have RF=2 on both datacenters?
>
>
> On Mon, Jun 3, 2013 at 10:09 PM, srmore <co...@gmail.com> wrote:
>
>> I am a bit confused when using the consistency level for multi datacenter
>> setup. Following is my setup:
>>
>> I have 4 nodes the way these are set up are
>> Node 1 DC 1 - N1DC1
>> Node 2 DC 1 - N2DC1
>>
>> Node 1 DC 2 - N1DC2
>> Node 2 DC 2 - N2DC2
>>
>> I setup a delay in between two datacenters (DC1 and DC2 around 1 sec one
>> way)
>>
>> I am observing that when I use consistency level 2 for some reason the
>> coordinate node is picking up the nodes from other datacenter. My
>> understanding was that Cassandra picks up nodes which are close by (from
>> local datacenter), determined by Gossip but looks like that's not the case.
>>
>> I found the following comment on Datastax website :
>>
>> "If using a consistency level of ONE or LOCAL_QUORUM, only the nodes in
>> the same data center as the coordinator node must respond to the client
>> request in order for the request to succeed."
>>
>> Does this mean that for multi datacenter we can only use ONE or
>> LOCAL_QUORUM if we want to use the local datacenter to avoid cross
>> datacenter latency.
>>
>> I am using the GossipingPropertyFileSnitch.
>>
>> Thanks !
>>
>>
>
Re: Consistency level for multi-datacenter setup
Posted by Sylvain Lebresne <sy...@datastax.com>.
What's your replication factor? Do you have RF=2 on both datacenters?
On Mon, Jun 3, 2013 at 10:09 PM, srmore <co...@gmail.com> wrote:
> I am a bit confused when using the consistency level for multi datacenter
> setup. Following is my setup:
>
> I have 4 nodes the way these are set up are
> Node 1 DC 1 - N1DC1
> Node 2 DC 1 - N2DC1
>
> Node 1 DC 2 - N1DC2
> Node 2 DC 2 - N2DC2
>
> I setup a delay in between two datacenters (DC1 and DC2 around 1 sec one
> way)
>
> I am observing that when I use consistency level 2 for some reason the
> coordinate node is picking up the nodes from other datacenter. My
> understanding was that Cassandra picks up nodes which are close by (from
> local datacenter), determined by Gossip but looks like that's not the case.
>
> I found the following comment on Datastax website :
>
> "If using a consistency level of ONE or LOCAL_QUORUM, only the nodes in
> the same data center as the coordinator node must respond to the client
> request in order for the request to succeed."
>
> Does this mean that for multi datacenter we can only use ONE or
> LOCAL_QUORUM if we want to use the local datacenter to avoid cross
> datacenter latency.
>
> I am using the GossipingPropertyFileSnitch.
>
> Thanks !
>
>
Re: Consistency level for multi-datacenter setup
Posted by srmore <co...@gmail.com>.
With CL=TWO it appears that one node randomly picks the node from other
datacenter to get the data. i.e. one node in the datacenter consistently
underperforms.
On Mon, Jun 3, 2013 at 3:21 PM, Hiller, Dean <De...@nrel.gov> wrote:
> What happens when you use CL=TWO.
>
> Dean
>
> From: srmore <co...@gmail.com>>
> Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <
> user@cassandra.apache.org<ma...@cassandra.apache.org>>
> Date: Monday, June 3, 2013 2:09 PM
> To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <
> user@cassandra.apache.org<ma...@cassandra.apache.org>>
> Subject: Consistency level for multi-datacenter setup
>
> I am a bit confused when using the consistency level for multi datacenter
> setup. Following is my setup:
>
> I have 4 nodes the way these are set up are
> Node 1 DC 1 - N1DC1
> Node 2 DC 1 - N2DC1
>
> Node 1 DC 2 - N1DC2
> Node 2 DC 2 - N2DC2
>
> I setup a delay in between two datacenters (DC1 and DC2 around 1 sec one
> way)
>
> I am observing that when I use consistency level 2 for some reason the
> coordinate node is picking up the nodes from other datacenter. My
> understanding was that Cassandra picks up nodes which are close by (from
> local datacenter), determined by Gossip but looks like that's not the case.
>
> I found the following comment on Datastax website :
>
> "If using a consistency level of ONE or LOCAL_QUORUM, only the nodes in
> the same data center as the coordinator node must respond to the client
> request in order for the request to succeed."
>
> Does this mean that for multi datacenter we can only use ONE or
> LOCAL_QUORUM if we want to use the local datacenter to avoid cross
> datacenter latency.
>
> I am using the GossipingPropertyFileSnitch.
>
> Thanks !
>
>
Re: Consistency level for multi-datacenter setup
Posted by srmore <co...@gmail.com>.
After some more investigation it does not appear to be the CL issue. Every
time I am starting up the node in other datacenter with 1sec delay my
throughput starts degrading, even with CL=ONE and CL=LOCAL_QUORUM.
I will put the logs on debug and investigate more and report back the
findings.
On Mon, Jun 3, 2013 at 3:37 PM, Hiller, Dean <De...@nrel.gov> wrote:
> Our badness threshold is 0.1 currently(just checked). Our website used to
> get slow during a slow node time until we rolled our own patch out.
>
> Dean
>
> From: srmore <co...@gmail.com>>
> Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <
> user@cassandra.apache.org<ma...@cassandra.apache.org>>
> Date: Monday, June 3, 2013 2:31 PM
> To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <
> user@cassandra.apache.org<ma...@cassandra.apache.org>>
> Subject: Re: Consistency level for multi-datacenter setup
>
> We observed that as well, please let us know what you find out it would be
> extremely helpful. There is also this property that you can play with to
> take care of slow nodes
> dynamic_snitch_badness_threshold.
>
>
> http://www.datastax.com/docs/1.1/configuration/node_configuration#dynamic-snitch-badness-threshold
>
> Thanks !
>
>
> On Mon, Jun 3, 2013 at 3:24 PM, Hiller, Dean <Dean.Hiller@nrel.gov<mailto:
> Dean.Hiller@nrel.gov>> wrote:
> Also, we had to put a fix into cassandra so it removed "slow nodes" from
> the list of nodes to read from. With that fix our QUOROM(not local quorom)
> started working again and would easily take the other DC nodes out of the
> list of reading from for you as well. I need to circle back to with my
> teammate to check if he got his fix posted to the dev list or not.
>
> Later,
> Dean
>
> From: srmore <co...@gmail.com><mailto:
> comomore@gmail.com<ma...@gmail.com>>>
> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org
> ><ma...@cassandra.apache.org>>" <
> user@cassandra.apache.org<ma...@cassandra.apache.org><mailto:
> user@cassandra.apache.org<ma...@cassandra.apache.org>>>
> Date: Monday, June 3, 2013 2:09 PM
> To: "user@cassandra.apache.org<ma...@cassandra.apache.org><mailto:
> user@cassandra.apache.org<ma...@cassandra.apache.org>>" <
> user@cassandra.apache.org<ma...@cassandra.apache.org><mailto:
> user@cassandra.apache.org<ma...@cassandra.apache.org>>>
> Subject: Consistency level for multi-datacenter setup
>
> I am a bit confused when using the consistency level for multi datacenter
> setup. Following is my setup:
>
> I have 4 nodes the way these are set up are
> Node 1 DC 1 - N1DC1
> Node 2 DC 1 - N2DC1
>
> Node 1 DC 2 - N1DC2
> Node 2 DC 2 - N2DC2
>
> I setup a delay in between two datacenters (DC1 and DC2 around 1 sec one
> way)
>
> I am observing that when I use consistency level 2 for some reason the
> coordinate node is picking up the nodes from other datacenter. My
> understanding was that Cassandra picks up nodes which are close by (from
> local datacenter), determined by Gossip but looks like that's not the case.
>
> I found the following comment on Datastax website :
>
> "If using a consistency level of ONE or LOCAL_QUORUM, only the nodes in
> the same data center as the coordinator node must respond to the client
> request in order for the request to succeed."
>
> Does this mean that for multi datacenter we can only use ONE or
> LOCAL_QUORUM if we want to use the local datacenter to avoid cross
> datacenter latency.
>
> I am using the GossipingPropertyFileSnitch.
>
> Thanks !
>
>
>
Re: Consistency level for multi-datacenter setup
Posted by "Hiller, Dean" <De...@nrel.gov>.
Our badness threshold is 0.1 currently(just checked). Our website used to get slow during a slow node time until we rolled our own patch out.
Dean
From: srmore <co...@gmail.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Monday, June 3, 2013 2:31 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Consistency level for multi-datacenter setup
We observed that as well, please let us know what you find out it would be extremely helpful. There is also this property that you can play with to take care of slow nodes
dynamic_snitch_badness_threshold.
http://www.datastax.com/docs/1.1/configuration/node_configuration#dynamic-snitch-badness-threshold
Thanks !
On Mon, Jun 3, 2013 at 3:24 PM, Hiller, Dean <De...@nrel.gov>> wrote:
Also, we had to put a fix into cassandra so it removed "slow nodes" from the list of nodes to read from. With that fix our QUOROM(not local quorom) started working again and would easily take the other DC nodes out of the list of reading from for you as well. I need to circle back to with my teammate to check if he got his fix posted to the dev list or not.
Later,
Dean
From: srmore <co...@gmail.com>>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>>" <us...@cassandra.apache.org>>>
Date: Monday, June 3, 2013 2:09 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>>" <us...@cassandra.apache.org>>>
Subject: Consistency level for multi-datacenter setup
I am a bit confused when using the consistency level for multi datacenter setup. Following is my setup:
I have 4 nodes the way these are set up are
Node 1 DC 1 - N1DC1
Node 2 DC 1 - N2DC1
Node 1 DC 2 - N1DC2
Node 2 DC 2 - N2DC2
I setup a delay in between two datacenters (DC1 and DC2 around 1 sec one way)
I am observing that when I use consistency level 2 for some reason the coordinate node is picking up the nodes from other datacenter. My understanding was that Cassandra picks up nodes which are close by (from local datacenter), determined by Gossip but looks like that's not the case.
I found the following comment on Datastax website :
"If using a consistency level of ONE or LOCAL_QUORUM, only the nodes in the same data center as the coordinator node must respond to the client request in order for the request to succeed."
Does this mean that for multi datacenter we can only use ONE or LOCAL_QUORUM if we want to use the local datacenter to avoid cross datacenter latency.
I am using the GossipingPropertyFileSnitch.
Thanks !
Re: Consistency level for multi-datacenter setup
Posted by srmore <co...@gmail.com>.
We observed that as well, please let us know what you find out it would be
extremely helpful. There is also this property that you can play with to
take care of slow nodes
*dynamic_snitch_badness_threshold*.
http://www.datastax.com/docs/1.1/configuration/node_configuration#dynamic-snitch-badness-threshold
Thanks !
On Mon, Jun 3, 2013 at 3:24 PM, Hiller, Dean <De...@nrel.gov> wrote:
> Also, we had to put a fix into cassandra so it removed "slow nodes" from
> the list of nodes to read from. With that fix our QUOROM(not local quorom)
> started working again and would easily take the other DC nodes out of the
> list of reading from for you as well. I need to circle back to with my
> teammate to check if he got his fix posted to the dev list or not.
>
> Later,
> Dean
>
> From: srmore <co...@gmail.com>>
> Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <
> user@cassandra.apache.org<ma...@cassandra.apache.org>>
> Date: Monday, June 3, 2013 2:09 PM
> To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <
> user@cassandra.apache.org<ma...@cassandra.apache.org>>
> Subject: Consistency level for multi-datacenter setup
>
> I am a bit confused when using the consistency level for multi datacenter
> setup. Following is my setup:
>
> I have 4 nodes the way these are set up are
> Node 1 DC 1 - N1DC1
> Node 2 DC 1 - N2DC1
>
> Node 1 DC 2 - N1DC2
> Node 2 DC 2 - N2DC2
>
> I setup a delay in between two datacenters (DC1 and DC2 around 1 sec one
> way)
>
> I am observing that when I use consistency level 2 for some reason the
> coordinate node is picking up the nodes from other datacenter. My
> understanding was that Cassandra picks up nodes which are close by (from
> local datacenter), determined by Gossip but looks like that's not the case.
>
> I found the following comment on Datastax website :
>
> "If using a consistency level of ONE or LOCAL_QUORUM, only the nodes in
> the same data center as the coordinator node must respond to the client
> request in order for the request to succeed."
>
> Does this mean that for multi datacenter we can only use ONE or
> LOCAL_QUORUM if we want to use the local datacenter to avoid cross
> datacenter latency.
>
> I am using the GossipingPropertyFileSnitch.
>
> Thanks !
>
>
Re: Consistency level for multi-datacenter setup
Posted by "Hiller, Dean" <De...@nrel.gov>.
Also, we had to put a fix into cassandra so it removed "slow nodes" from the list of nodes to read from. With that fix our QUOROM(not local quorom) started working again and would easily take the other DC nodes out of the list of reading from for you as well. I need to circle back to with my teammate to check if he got his fix posted to the dev list or not.
Later,
Dean
From: srmore <co...@gmail.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Monday, June 3, 2013 2:09 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Consistency level for multi-datacenter setup
I am a bit confused when using the consistency level for multi datacenter setup. Following is my setup:
I have 4 nodes the way these are set up are
Node 1 DC 1 - N1DC1
Node 2 DC 1 - N2DC1
Node 1 DC 2 - N1DC2
Node 2 DC 2 - N2DC2
I setup a delay in between two datacenters (DC1 and DC2 around 1 sec one way)
I am observing that when I use consistency level 2 for some reason the coordinate node is picking up the nodes from other datacenter. My understanding was that Cassandra picks up nodes which are close by (from local datacenter), determined by Gossip but looks like that's not the case.
I found the following comment on Datastax website :
"If using a consistency level of ONE or LOCAL_QUORUM, only the nodes in the same data center as the coordinator node must respond to the client request in order for the request to succeed."
Does this mean that for multi datacenter we can only use ONE or LOCAL_QUORUM if we want to use the local datacenter to avoid cross datacenter latency.
I am using the GossipingPropertyFileSnitch.
Thanks !
Re: Consistency level for multi-datacenter setup
Posted by "Hiller, Dean" <De...@nrel.gov>.
What happens when you use CL=TWO.
Dean
From: srmore <co...@gmail.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Monday, June 3, 2013 2:09 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Consistency level for multi-datacenter setup
I am a bit confused when using the consistency level for multi datacenter setup. Following is my setup:
I have 4 nodes the way these are set up are
Node 1 DC 1 - N1DC1
Node 2 DC 1 - N2DC1
Node 1 DC 2 - N1DC2
Node 2 DC 2 - N2DC2
I setup a delay in between two datacenters (DC1 and DC2 around 1 sec one way)
I am observing that when I use consistency level 2 for some reason the coordinate node is picking up the nodes from other datacenter. My understanding was that Cassandra picks up nodes which are close by (from local datacenter), determined by Gossip but looks like that's not the case.
I found the following comment on Datastax website :
"If using a consistency level of ONE or LOCAL_QUORUM, only the nodes in the same data center as the coordinator node must respond to the client request in order for the request to succeed."
Does this mean that for multi datacenter we can only use ONE or LOCAL_QUORUM if we want to use the local datacenter to avoid cross datacenter latency.
I am using the GossipingPropertyFileSnitch.
Thanks !