You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by AJ <aj...@dude.podzone.net> on 2011/06/15 00:47:01 UTC

Docs: Token Selection

This http://wiki.apache.org/cassandra/Operations#Token_selection  says:

"With NetworkTopologyStrategy, you should calculate the tokens the nodes 
in each DC independantly."

and gives the example:

DC1
node 1 = 0
node 2 = 85070591730234615865843651857942052864

DC2
node 3 = 1
node 4 = 85070591730234615865843651857942052865


So, according to the above, the token ranges would be (abbreviated nums):

DC1
node 1 = 0	Range: (8..4, 16], (0, 0]
node 2 = 8..4	Range: (0, 8..4]

DC2
node 3 = 1	Range: (8..5, 16], (0, 1]
node 4 = 8..5	Range: (1, 8..5]


If the above is correct, then I would be surprised as this paragraph is 
the only place were one would discover this and may be easy to miss... 
unless there's a doc buried somewhere in plain view that I missed.

So, have I interpreted this paragraph correctly?  Was this design to 
help keep data somewhat localized if that was important, such as a 
geographically dispersed DC?

Thanks!

Re: Docs: Token Selection

Posted by AJ <aj...@dude.podzone.net>.
LOL, I feel Eric's pain.  This double-ring thing can throw you for a 
loop since, like I said, there is only one place it is documented and it 
is only *implied*, so one is not sure he is interpreting it correctly.  
Even the source for NTS doesn't mention this.

Thanks for everyone's help on this.

On 6/16/2011 5:43 AM, aaron morton wrote:
> See this thread for background 
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Replica-data-distributing-between-racks-td6324819.html 
>
>
> In a multi DC environment, if you calculate the initial tokens for the 
> entire cluster data will not be evenly distributed.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>


Re: Docs: Token Selection

Posted by AJ <aj...@dude.podzone.net>.
On 6/17/2011 12:33 PM, Eric tamme wrote:
>
> As i said previously, trying to build make cassandra treat things
> differently based on some kind of persistent locality set it maintains
> in memory .. or whatever .. sounds like you will be absolutely
> undermining the core principles of how cassandra works.
>
> -Eric
>

That sounds so funny because figuring out how Cassandra works is the 
hard part! ;o)  But, thanks for your explanation; it helps.

Re: Docs: Token Selection

Posted by Eric tamme <et...@gmail.com>.
> Yes.  But, the more I think about it, the more I see issues.  Here is what I
> envision (Issues marked with *):
>
> Three or more dc's, each serving as fail-overs for the others with 1 maximum
> unavailable dc supported at a time.
> Each dc is a production dc serving users that I choose.
> Each dc also stores 0-1 replicas from the other dc's.
> Direct customers to their "home" dc of my choice.
> Data coming from the client local to the dc is replicated X times in the
> local dc and 1 time in any other dc (randomly).

There is no "random" placement with NTS.  Make sure each DC has a
complete, evenly distributed token range coverage for the nodes in
that DC. Use propertyFileSnitch and specify replication for each
{DC0:2,DC1:2,DC2:2}

> In the even a dc becomes unreachable by users, an arbitrary fail-over dc can
> serve their requests albeit with increased latency.
> *There will only be 1 replica left amongst the remaining fail-over dc's, so
> this could be a problem depending on the CL used other than CL.ONE.

So ... increase the RF per DC.. either you want 1 replica or you want
more .. you cant have it both ways.

> *During the fail-over state, the cluster needs to know that the real "home"
> of the replicas belongs to the currently unavailable dc.

You are introducing concepts that dont exist in cassandra.  There is
no "real home" and by trying to make cassandra aware of where home is
for some set of clients, you are subverting the completely distributed
nature.

>But, as of now, I
> don't think that's possible and so new writes will start to be replicated in
> the current dc as if the currently-used fail-over dc is the home dc.

It will behave exactly as it did when clients were in the other data
center, and will insert data locally if possible, and replicate
according to your placement strategy.

> Maybe these goals can be achieve with a kind of ordered asymmetrical
> replication strategy like you illustrated above.  The hard part will be to
> figure out a simple and elegant way to do this w/o undermining C*.

As i said previously, trying to build make cassandra treat things
differently based on some kind of persistent locality set it maintains
in memory .. or whatever .. sounds like you will be absolutely
undermining the core principles of how cassandra works.

-Eric

Re: Docs: Token Selection

Posted by AJ <aj...@dude.podzone.net>.
On 6/17/2011 10:31 AM, Eric tamme wrote:
>> What I don't like about NTS is I would have to have more replicas than I
>> need.  {DC1=2, DC2=2}, RF=4 would be the minimum.  If I felt that 2 local
>> replicas was insufficient, I'd have to move up to RF=6 which seems like a
>> waste... I'm predicting data in the TB range so I'm trying to keep replicas
>> to a minimum.
>>
>> My goal is to have 2-3 replicas in a local data center and 1 replica in
>> another dc.  I think that would be enough barring a major catastrophe.  But,
>> I'm not sure this is possible.  I define "local" as in the same data center
>> as the client doing the insert/update.
> Yes, not being able to configure the replication factor differently
> for each data center is a bit annoying.  Im assuming you basically
> want DC1 to have a replication factor of {DC1:2, DC2:1} and DC2 to
> have {DC1:1,DC2:2}.

Yes.  But, the more I think about it, the more I see issues.  Here is 
what I envision (Issues marked with *):

Three or more dc's, each serving as fail-overs for the others with 1 
maximum unavailable dc supported at a time.
Each dc is a production dc serving users that I choose.
Each dc also stores 0-1 replicas from the other dc's.
Direct customers to their "home" dc of my choice.
Data coming from the client local to the dc is replicated X times in the 
local dc and 1 time in any other dc (randomly).
In the even a dc becomes unreachable by users, an arbitrary fail-over dc 
can serve their requests albeit with increased latency.
*There will only be 1 replica left amongst the remaining fail-over dc's, 
so this could be a problem depending on the CL used other than CL.ONE.
*During the fail-over state, the cluster needs to know that the real 
"home" of the replicas belongs to the currently unavailable dc.  But, as 
of now, I don't think that's possible and so new writes will start to be 
replicated in the current dc as if the currently-used fail-over dc is 
the home dc.

Maybe these goals can be achieve with a kind of ordered asymmetrical 
replication strategy like you illustrated above.  The hard part will be 
to figure out a simple and elegant way to do this w/o undermining C*.


> I would very much like that feature as well, but I dont know the
> feasibility of it.
>
> -Eric
>


Re: Docs: Token Selection

Posted by AJ <aj...@dude.podzone.net>.
On 6/17/2011 1:27 PM, Sasha Dolgy wrote:
> Replication factor is defined per keyspace if i'm not mistaken.  Can't
> remember if NTS is per keyspace or per cluster ... if it's per
> keyspace, that would be a way around it ... without having to maintain
> multiple clusters .... just have multiple keyspaces ...

Hey!  I think you're on to something Sasha!  Placement strat and RF are 
both defined per ks.  I'll let that stew for a while in my brain.

Thanks!

> On Fri, Jun 17, 2011 at 9:23 PM, AJ<aj...@dude.podzone.net>  wrote:
>> On 6/17/2011 12:32 PM, Jeremiah Jordan wrote:
>>>         Run two clusters, one which has {DC1:2, DC2:1} and one which is
>>> {DC1:1,DC2:2}.  You can't have both in the same cluster, otherwise it
>>> isn't possible to tell where the data got written when you want to read
>>> it.  For a given key "XYZ" you must be able to compute which nodes it is
>>> stored on just using "XYZ", so a strategy where it is on nodes
>>> DC1_1,DC1_2, and DC2_1 when a node in DC1 is the coordinator, and to
>>> DC1_1, DC2_1 and DC2_2 when a node in DC2 is the coordinator won't work.
>>> Given just "XYZ" I don't know where to look for the data.
>>>         But, from the way you describe what you want to happen, clients
>>> from DC1 aren't using data inserted by clients from DC2, so you should
>>> just make two different Cassandra clusters.  Once for the DC1 guys which
>>> is {DC1:2, DC2:1} and one for the DC2 guys which is {DC1:1,DC2:2}.
>>>
>> Interesting.  Thx.
>>
>>
>
>


Re: Docs: Token Selection

Posted by Sasha Dolgy <sd...@gmail.com>.
Replication factor is defined per keyspace if i'm not mistaken.  Can't
remember if NTS is per keyspace or per cluster ... if it's per
keyspace, that would be a way around it ... without having to maintain
multiple clusters .... just have multiple keyspaces ...

On Fri, Jun 17, 2011 at 9:23 PM, AJ <aj...@dude.podzone.net> wrote:
> On 6/17/2011 12:32 PM, Jeremiah Jordan wrote:
>>
>>        Run two clusters, one which has {DC1:2, DC2:1} and one which is
>> {DC1:1,DC2:2}.  You can't have both in the same cluster, otherwise it
>> isn't possible to tell where the data got written when you want to read
>> it.  For a given key "XYZ" you must be able to compute which nodes it is
>> stored on just using "XYZ", so a strategy where it is on nodes
>> DC1_1,DC1_2, and DC2_1 when a node in DC1 is the coordinator, and to
>> DC1_1, DC2_1 and DC2_2 when a node in DC2 is the coordinator won't work.
>> Given just "XYZ" I don't know where to look for the data.
>>        But, from the way you describe what you want to happen, clients
>> from DC1 aren't using data inserted by clients from DC2, so you should
>> just make two different Cassandra clusters.  Once for the DC1 guys which
>> is {DC1:2, DC2:1} and one for the DC2 guys which is {DC1:1,DC2:2}.
>>
>
> Interesting.  Thx.
>
>



-- 
Sasha Dolgy
sasha.dolgy@gmail.com

Re: Docs: Token Selection

Posted by AJ <aj...@dude.podzone.net>.
On 6/17/2011 12:32 PM, Jeremiah Jordan wrote:
> 	Run two clusters, one which has {DC1:2, DC2:1} and one which is
> {DC1:1,DC2:2}.  You can't have both in the same cluster, otherwise it
> isn't possible to tell where the data got written when you want to read
> it.  For a given key "XYZ" you must be able to compute which nodes it is
> stored on just using "XYZ", so a strategy where it is on nodes
> DC1_1,DC1_2, and DC2_1 when a node in DC1 is the coordinator, and to
> DC1_1, DC2_1 and DC2_2 when a node in DC2 is the coordinator won't work.
> Given just "XYZ" I don't know where to look for the data.
> 	But, from the way you describe what you want to happen, clients
> from DC1 aren't using data inserted by clients from DC2, so you should
> just make two different Cassandra clusters.  Once for the DC1 guys which
> is {DC1:2, DC2:1} and one for the DC2 guys which is {DC1:1,DC2:2}.
>

Interesting.  Thx.


RE: Docs: Token Selection

Posted by Jeremiah Jordan <JE...@morningstar.com>.
	Run two clusters, one which has {DC1:2, DC2:1} and one which is
{DC1:1,DC2:2}.  You can't have both in the same cluster, otherwise it
isn't possible to tell where the data got written when you want to read
it.  For a given key "XYZ" you must be able to compute which nodes it is
stored on just using "XYZ", so a strategy where it is on nodes
DC1_1,DC1_2, and DC2_1 when a node in DC1 is the coordinator, and to
DC1_1, DC2_1 and DC2_2 when a node in DC2 is the coordinator won't work.
Given just "XYZ" I don't know where to look for the data.
	But, from the way you describe what you want to happen, clients
from DC1 aren't using data inserted by clients from DC2, so you should
just make two different Cassandra clusters.  Once for the DC1 guys which
is {DC1:2, DC2:1} and one for the DC2 guys which is {DC1:1,DC2:2}.

-----Original Message-----
From: AJ [mailto:aj@dude.podzone.net] 
Sent: Friday, June 17, 2011 1:02 PM
To: user@cassandra.apache.org
Subject: Re: Docs: Token Selection

Hi Jeremiah, can you give more details?

Thanks

On 6/17/2011 10:49 AM, Jeremiah Jordan wrote:
> Run two Cassandra clusters...
>
> -----Original Message-----
> From: Eric tamme [mailto:etamme@gmail.com]
> Sent: Friday, June 17, 2011 11:31 AM
> To: user@cassandra.apache.org
> Subject: Re: Docs: Token Selection
>
>> What I don't like about NTS is I would have to have more replicas 
>> than I need.  {DC1=2, DC2=2}, RF=4 would be the minimum.  If I felt 
>> that 2 local replicas was insufficient, I'd have to move up to RF=6 
>> which seems like a waste... I'm predicting data in the TB range so 
>> I'm trying to keep replicas to a minimum.
>>
>> My goal is to have 2-3 replicas in a local data center and 1 replica 
>> in another dc.  I think that would be enough barring a major 
>> catastrophe.  But, I'm not sure this is possible.  I define "local" 
>> as in the same data center as the client doing the insert/update.
> Yes, not being able to configure the replication factor differently
for each data center is a bit annoying.  Im assuming you basically want
DC1 to have a replication factor of {DC1:2, DC2:1} and DC2 to have
{DC1:1,DC2:2}.
>
> I would very much like that feature as well, but I dont know the
feasibility of it.
>
> -Eric
>


Re: Docs: Token Selection

Posted by AJ <aj...@dude.podzone.net>.
Hi Jeremiah, can you give more details?

Thanks

On 6/17/2011 10:49 AM, Jeremiah Jordan wrote:
> Run two Cassandra clusters...
>
> -----Original Message-----
> From: Eric tamme [mailto:etamme@gmail.com]
> Sent: Friday, June 17, 2011 11:31 AM
> To: user@cassandra.apache.org
> Subject: Re: Docs: Token Selection
>
>> What I don't like about NTS is I would have to have more replicas than
>> I need.  {DC1=2, DC2=2}, RF=4 would be the minimum.  If I felt that 2
>> local replicas was insufficient, I'd have to move up to RF=6 which
>> seems like a waste... I'm predicting data in the TB range so I'm
>> trying to keep replicas to a minimum.
>>
>> My goal is to have 2-3 replicas in a local data center and 1 replica
>> in another dc.  I think that would be enough barring a major
>> catastrophe.  But, I'm not sure this is possible.  I define "local" as
>> in the same data center as the client doing the insert/update.
> Yes, not being able to configure the replication factor differently for each data center is a bit annoying.  Im assuming you basically want DC1 to have a replication factor of {DC1:2, DC2:1} and DC2 to have {DC1:1,DC2:2}.
>
> I would very much like that feature as well, but I dont know the feasibility of it.
>
> -Eric
>


RE: Docs: Token Selection

Posted by Jeremiah Jordan <JE...@morningstar.com>.
Run two Cassandra clusters... 

-----Original Message-----
From: Eric tamme [mailto:etamme@gmail.com] 
Sent: Friday, June 17, 2011 11:31 AM
To: user@cassandra.apache.org
Subject: Re: Docs: Token Selection

> What I don't like about NTS is I would have to have more replicas than 
> I need.  {DC1=2, DC2=2}, RF=4 would be the minimum.  If I felt that 2 
> local replicas was insufficient, I'd have to move up to RF=6 which 
> seems like a waste... I'm predicting data in the TB range so I'm 
> trying to keep replicas to a minimum.
>
> My goal is to have 2-3 replicas in a local data center and 1 replica 
> in another dc.  I think that would be enough barring a major 
> catastrophe.  But, I'm not sure this is possible.  I define "local" as 
> in the same data center as the client doing the insert/update.

Yes, not being able to configure the replication factor differently for each data center is a bit annoying.  Im assuming you basically want DC1 to have a replication factor of {DC1:2, DC2:1} and DC2 to have {DC1:1,DC2:2}.

I would very much like that feature as well, but I dont know the feasibility of it.

-Eric

Re: Docs: Token Selection

Posted by AJ <aj...@dude.podzone.net>.
+1  Yes, that is what I'm talking about Eric.  Maybe I could write my 
own strategy, I dunno.  I'll have to understand more first.

On 6/17/2011 10:37 AM, Sasha Dolgy wrote:
> +1 for this if it is possible...
>
> On Fri, Jun 17, 2011 at 6:31 PM, Eric tamme<et...@gmail.com>  wrote:
>>> What I don't like about NTS is I would have to have more replicas than I
>>> need.  {DC1=2, DC2=2}, RF=4 would be the minimum.  If I felt that 2 local
>>> replicas was insufficient, I'd have to move up to RF=6 which seems like a
>>> waste... I'm predicting data in the TB range so I'm trying to keep replicas
>>> to a minimum.
>>>
>>> My goal is to have 2-3 replicas in a local data center and 1 replica in
>>> another dc.  I think that would be enough barring a major catastrophe.  But,
>>> I'm not sure this is possible.  I define "local" as in the same data center
>>> as the client doing the insert/update.
>> Yes, not being able to configure the replication factor differently
>> for each data center is a bit annoying.  Im assuming you basically
>> want DC1 to have a replication factor of {DC1:2, DC2:1} and DC2 to
>> have {DC1:1,DC2:2}.
>>
>> I would very much like that feature as well, but I dont know the
>> feasibility of it.
>>
>> -Eric


Re: Docs: Token Selection

Posted by Sasha Dolgy <sd...@gmail.com>.
+1 for this if it is possible...

On Fri, Jun 17, 2011 at 6:31 PM, Eric tamme <et...@gmail.com> wrote:
>> What I don't like about NTS is I would have to have more replicas than I
>> need.  {DC1=2, DC2=2}, RF=4 would be the minimum.  If I felt that 2 local
>> replicas was insufficient, I'd have to move up to RF=6 which seems like a
>> waste... I'm predicting data in the TB range so I'm trying to keep replicas
>> to a minimum.
>>
>> My goal is to have 2-3 replicas in a local data center and 1 replica in
>> another dc.  I think that would be enough barring a major catastrophe.  But,
>> I'm not sure this is possible.  I define "local" as in the same data center
>> as the client doing the insert/update.
>
> Yes, not being able to configure the replication factor differently
> for each data center is a bit annoying.  Im assuming you basically
> want DC1 to have a replication factor of {DC1:2, DC2:1} and DC2 to
> have {DC1:1,DC2:2}.
>
> I would very much like that feature as well, but I dont know the
> feasibility of it.
>
> -Eric

Re: Docs: Token Selection

Posted by Eric tamme <et...@gmail.com>.
> What I don't like about NTS is I would have to have more replicas than I
> need.  {DC1=2, DC2=2}, RF=4 would be the minimum.  If I felt that 2 local
> replicas was insufficient, I'd have to move up to RF=6 which seems like a
> waste... I'm predicting data in the TB range so I'm trying to keep replicas
> to a minimum.
>
> My goal is to have 2-3 replicas in a local data center and 1 replica in
> another dc.  I think that would be enough barring a major catastrophe.  But,
> I'm not sure this is possible.  I define "local" as in the same data center
> as the client doing the insert/update.

Yes, not being able to configure the replication factor differently
for each data center is a bit annoying.  Im assuming you basically
want DC1 to have a replication factor of {DC1:2, DC2:1} and DC2 to
have {DC1:1,DC2:2}.

I would very much like that feature as well, but I dont know the
feasibility of it.

-Eric

Re: Docs: Token Selection

Posted by AJ <aj...@dude.podzone.net>.
On 6/17/2011 7:26 AM, William Oberman wrote:
> I haven't done it yet, but when I researched how to make 
> geo-diverse/failover DCs, I figured I'd have to do something like 
> RF=6, strategy = {DC1=3, DC2=3}, and LOCAL_QUORUM for reads/writes. 
>  This gives you an "ack" after 2 local nodes do the read/write, but 
> the data eventually gets distributed to the other DC for a full 
> failover.  No "ying-yang", but I believe accomplishes the same goal?
>
> will

What I don't like about NTS is I would have to have more replicas than I 
need.  {DC1=2, DC2=2}, RF=4 would be the minimum.  If I felt that 2 
local replicas was insufficient, I'd have to move up to RF=6 which seems 
like a waste... I'm predicting data in the TB range so I'm trying to 
keep replicas to a minimum.

My goal is to have 2-3 replicas in a local data center and 1 replica in 
another dc.  I think that would be enough barring a major catastrophe.  
But, I'm not sure this is possible.  I define "local" as in the same 
data center as the client doing the insert/update.


Re: Docs: Token Selection

Posted by William Oberman <ob...@civicscience.com>.
I haven't done it yet, but when I researched how to make
geo-diverse/failover DCs, I figured I'd have to do something like RF=6,
strategy = {DC1=3, DC2=3}, and LOCAL_QUORUM for reads/writes.  This gives
you an "ack" after 2 local nodes do the read/write, but the data eventually
gets distributed to the other DC for a full failover.  No "ying-yang", but I
believe accomplishes the same goal?

will

On Fri, Jun 17, 2011 at 2:20 AM, Jonathan Ellis <jb...@gmail.com> wrote:

> Replication location is determined by the row key, not the location of
> the client that inserted it.  (Otherwise, without knowing what DC a
> row was inserted in, you couldn't look it up to read it!)
>
> On Fri, Jun 17, 2011 at 12:20 AM, AJ <aj...@dude.podzone.net> wrote:
> > On 6/16/2011 9:45 PM, aaron morton wrote:
> >>>
> >>> But, I'm thinking about using OldNetworkTopStrat.
> >>
> >> NetworkTopologyStrategy is where it's at.
> >
> > Oh yeah?  It didn't look like it would serve my requirements.  I want 2
> full
> > production geo-diverse data centers with each serving as a failover for
> the
> > other.  Random Partitioner.  Each dc holds 2 replicas from the local
> clients
> > and 1 replica goes to the other dc.  It doesn't look like I can do a
> > ying-yang setup like that with NTS.  Am I wrong?
> >
> >> A
> >> -----------------
> >> Aaron Morton
> >> Freelance Cassandra Developer
> >> @aaronmorton
> >> http://www.thelastpickle.com
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>



-- 
Will Oberman
Civic Science, Inc.
3030 Penn Avenue., First Floor
Pittsburgh, PA 15201
(M) 412-480-7835
(E) oberman@civicscience.com

Re: Docs: Token Selection

Posted by Eric tamme <et...@gmail.com>.
On Fri, Jun 17, 2011 at 12:07 PM, AJ <aj...@dude.podzone.net> wrote:
> Thanks Jonathan.  I assumed since each data center owned the full key space
> that the first replica would be stored in the dc of the coordinating node,
> the 2nd in another dc, and the 3rd+ back in the 1st dc.  But, are you saying
> that the first endpoint is selected regardless of the location of the
> coordinating node?  Are you saying that the starting endpoint is the one
> closest to the row token regardless of the dc?

Cassandra will write to the node whose Initial Token is closest
without being larger then the data's token.  It doesn't take into
consideration what node was the recipient of the request (which i
guess is what you are calling the coordinating node).  The proximity
of a node is determined by your endpoint snitch - so it is a
determining factor in figuring out who gets a write in the condition
of overlapping token ranges.

> So, it is possible that a
> replica might not even get stored in the dc of the coordinator at all
> depending on how many dc's there are, rf, starting token assignments, etc.?

Cassandra determines replica writes the same way it would do for
primary insertion, but it is subject to the constraints of your
replica placement strategy, and how it determines locality based on
your endpoint snitch. So yes, It is entirely possible that the DC that
the "coordinator" is in will not get any replica, its really dependent
on your initial tokens, endpoint snitch, and placement strategy.

-Eric

Re: Docs: Token Selection

Posted by AJ <aj...@dude.podzone.net>.
Thanks Jonathan.  I assumed since each data center owned the full key 
space that the first replica would be stored in the dc of the 
coordinating node, the 2nd in another dc, and the 3rd+ back in the 1st 
dc.  But, are you saying that the first endpoint is selected regardless 
of the location of the coordinating node?  Are you saying that the 
starting endpoint is the one closest to the row token regardless of the 
dc?  So, it is possible that a replica might not even get stored in the 
dc of the coordinator at all depending on how many dc's there are, rf, 
starting token assignments, etc.?


On 6/17/2011 12:20 AM, Jonathan Ellis wrote:
> Replication location is determined by the row key, not the location of
> the client that inserted it.  (Otherwise, without knowing what DC a
> row was inserted in, you couldn't look it up to read it!)
>
> On Fri, Jun 17, 2011 at 12:20 AM, AJ<aj...@dude.podzone.net>  wrote:
>> On 6/16/2011 9:45 PM, aaron morton wrote:
>>>> But, I'm thinking about using OldNetworkTopStrat.
>>> NetworkTopologyStrategy is where it's at.
>> Oh yeah?  It didn't look like it would serve my requirements.  I want 2 full
>> production geo-diverse data centers with each serving as a failover for the
>> other.  Random Partitioner.  Each dc holds 2 replicas from the local clients
>> and 1 replica goes to the other dc.  It doesn't look like I can do a
>> ying-yang setup like that with NTS.  Am I wrong?
>>
>>> A
>>> -----------------
>>> Aaron Morton
>>> Freelance Cassandra Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>
>
>


Re: Docs: Token Selection

Posted by Jonathan Ellis <jb...@gmail.com>.
Replication location is determined by the row key, not the location of
the client that inserted it.  (Otherwise, without knowing what DC a
row was inserted in, you couldn't look it up to read it!)

On Fri, Jun 17, 2011 at 12:20 AM, AJ <aj...@dude.podzone.net> wrote:
> On 6/16/2011 9:45 PM, aaron morton wrote:
>>>
>>> But, I'm thinking about using OldNetworkTopStrat.
>>
>> NetworkTopologyStrategy is where it's at.
>
> Oh yeah?  It didn't look like it would serve my requirements.  I want 2 full
> production geo-diverse data centers with each serving as a failover for the
> other.  Random Partitioner.  Each dc holds 2 replicas from the local clients
> and 1 replica goes to the other dc.  It doesn't look like I can do a
> ying-yang setup like that with NTS.  Am I wrong?
>
>> A
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Docs: Token Selection

Posted by AJ <aj...@dude.podzone.net>.
On 6/16/2011 9:45 PM, aaron morton wrote:
>> But, I'm thinking about using OldNetworkTopStrat.
> NetworkTopologyStrategy is where it's at.

Oh yeah?  It didn't look like it would serve my requirements.  I want 2 
full production geo-diverse data centers with each serving as a failover 
for the other.  Random Partitioner.  Each dc holds 2 replicas from the 
local clients and 1 replica goes to the other dc.  It doesn't look like 
I can do a ying-yang setup like that with NTS.  Am I wrong?

> A
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com


Re: Docs: Token Selection

Posted by aaron morton <aa...@thelastpickle.com>.
> But, I'm thinking about using OldNetworkTopStrat. 

NetworkTopologyStrategy is where it's at. 

A
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 17 Jun 2011, at 01:39, AJ wrote:

> Thanks Eric!  I've finally got it!  I feel like I've just been initiated or something by discovering this "secret".  I kid!
> 
> But, I'm thinking about using OldNetworkTopStrat.  Do you, or anyone else, know if the same rules for token assignment applies to ONTS?
> 
> 
> On 6/16/2011 7:21 AM, Eric tamme wrote:
>> AJ,
>> 
>> sorry I seemed to miss the original email on this thread.  As Aaron
>> said, when computing tokens for multiple data centers, you should
>> compute them independently for each data center - as if it were its
>> own Cassandra cluster.
>> 
>> You can have "overlapping" token ranges between multiple data centers,
>> but no two nodes can have the same token, so for subsequent data
>> centers I just increment the tokens.
>> 
>> For two data centers with two nodes each using RandomPartitioner
>> calculate the tokens for the first DC normally, but int he second data
>> center, increment the tokens by one.
>> 
>> In DC 1
>> node 1 = 0
>> node 2 = 85070591730234615865843651857942052864
>> 
>> In DC 2
>> node 1 = 1
>> node 2 =  85070591730234615865843651857942052865
>> 
>> For RowMutations this will give each data center a local set of nodes
>> that it can write to for complete coverage of the entire token space.
>> If you are using NetworkTopologyStrategy for replication, it will give
>> an offset mirror replication between the two data centers so that your
>> replicas will not get pinned to a node in the remote DC.  There are
>> other ways to select the tokens, but the increment method is the
>> simplest to manage and continue to grow with.
>> 
>> Hope that helps.
>> 
>> -Eric
>> 
> 


Re: Docs: Token Selection

Posted by Eric tamme <et...@gmail.com>.
On Thu, Jun 16, 2011 at 11:11 AM, Sasha Dolgy <sd...@gmail.com> wrote:
> So, with ec2 ... 3 regions (DC's), each one is +1 from another....?

I dont use ec2, so I am not familiar with the specifics of deployment
there.  That said, if you have 3 data centers with equal nodes in each
(so that you would calculate the same tokens for each DC) - the first
DC you would add 0, the second DC you would add 1, the third DC you
would add 2.

so it could look like the following

In DC 1
node 1 = 0
node 2 = 85070591730234615865843651857942052864

In DC 2
node 1 = 1
node 2 =  85070591730234615865843651857942052865

In DC 3
node 1 = 2
node 2 =  85070591730234615865843651857942052866

Keep in mind, the only reason you need to offset tokens is if there is
another node that would have the exact same token.  So if you have
different numbers of nodes in different data centers, it is possible
you wont need any token offsets.  Just calculate tokens normally, as
if the DC was the only one, then check for any node in another DC with
the same token and add +1 to offset the token.

-Eric

Re: Docs: Token Selection

Posted by Sasha Dolgy <sd...@gmail.com>.
So, with ec2 ... 3 regions (DC's), each one is +1 from another....?
On Jun 16, 2011 3:40 PM, "AJ" <aj...@dude.podzone.net> wrote:
> Thanks Eric! I've finally got it! I feel like I've just been initiated
> or something by discovering this "secret". I kid!
>
> But, I'm thinking about using OldNetworkTopStrat. Do you, or anyone
> else, know if the same rules for token assignment applies to ONTS?
>
>
> On 6/16/2011 7:21 AM, Eric tamme wrote:
>> AJ,
>>
>> sorry I seemed to miss the original email on this thread. As Aaron
>> said, when computing tokens for multiple data centers, you should
>> compute them independently for each data center - as if it were its
>> own Cassandra cluster.
>>
>> You can have "overlapping" token ranges between multiple data centers,
>> but no two nodes can have the same token, so for subsequent data
>> centers I just increment the tokens.
>>
>> For two data centers with two nodes each using RandomPartitioner
>> calculate the tokens for the first DC normally, but int he second data
>> center, increment the tokens by one.
>>
>> In DC 1
>> node 1 = 0
>> node 2 = 85070591730234615865843651857942052864
>>
>> In DC 2
>> node 1 = 1
>> node 2 = 85070591730234615865843651857942052865
>>
>> For RowMutations this will give each data center a local set of nodes
>> that it can write to for complete coverage of the entire token space.
>> If you are using NetworkTopologyStrategy for replication, it will give
>> an offset mirror replication between the two data centers so that your
>> replicas will not get pinned to a node in the remote DC. There are
>> other ways to select the tokens, but the increment method is the
>> simplest to manage and continue to grow with.
>>
>> Hope that helps.
>>
>> -Eric
>>
>

Re: Docs: Token Selection

Posted by AJ <aj...@dude.podzone.net>.
Thanks Eric!  I've finally got it!  I feel like I've just been initiated 
or something by discovering this "secret".  I kid!

But, I'm thinking about using OldNetworkTopStrat.  Do you, or anyone 
else, know if the same rules for token assignment applies to ONTS?


On 6/16/2011 7:21 AM, Eric tamme wrote:
> AJ,
>
> sorry I seemed to miss the original email on this thread.  As Aaron
> said, when computing tokens for multiple data centers, you should
> compute them independently for each data center - as if it were its
> own Cassandra cluster.
>
> You can have "overlapping" token ranges between multiple data centers,
> but no two nodes can have the same token, so for subsequent data
> centers I just increment the tokens.
>
> For two data centers with two nodes each using RandomPartitioner
> calculate the tokens for the first DC normally, but int he second data
> center, increment the tokens by one.
>
> In DC 1
> node 1 = 0
> node 2 = 85070591730234615865843651857942052864
>
> In DC 2
> node 1 = 1
> node 2 =  85070591730234615865843651857942052865
>
> For RowMutations this will give each data center a local set of nodes
> that it can write to for complete coverage of the entire token space.
> If you are using NetworkTopologyStrategy for replication, it will give
> an offset mirror replication between the two data centers so that your
> replicas will not get pinned to a node in the remote DC.  There are
> other ways to select the tokens, but the increment method is the
> simplest to manage and continue to grow with.
>
> Hope that helps.
>
> -Eric
>


Re: Docs: Token Selection

Posted by Eric tamme <et...@gmail.com>.
AJ,

sorry I seemed to miss the original email on this thread.  As Aaron
said, when computing tokens for multiple data centers, you should
compute them independently for each data center - as if it were its
own Cassandra cluster.

You can have "overlapping" token ranges between multiple data centers,
but no two nodes can have the same token, so for subsequent data
centers I just increment the tokens.

For two data centers with two nodes each using RandomPartitioner
calculate the tokens for the first DC normally, but int he second data
center, increment the tokens by one.

In DC 1
node 1 = 0
node 2 = 85070591730234615865843651857942052864

In DC 2
node 1 = 1
node 2 =  85070591730234615865843651857942052865

For RowMutations this will give each data center a local set of nodes
that it can write to for complete coverage of the entire token space.
If you are using NetworkTopologyStrategy for replication, it will give
an offset mirror replication between the two data centers so that your
replicas will not get pinned to a node in the remote DC.  There are
other ways to select the tokens, but the increment method is the
simplest to manage and continue to grow with.

Hope that helps.

-Eric

Re: Docs: Token Selection

Posted by aaron morton <aa...@thelastpickle.com>.
See this thread for background http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Replica-data-distributing-between-racks-td6324819.html

In a multi DC environment, if you calculate the initial tokens for the entire cluster data will not be evenly distributed. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 16 Jun 2011, at 15:51, Vijay wrote:

> +1 for more documentation (I guess contributions are always welcomed).... I will try to write it down sometime when we have a bit more time...
> 
> 0.8 nodetool ring command adds the DC and RAC information....
> 
> http://www.datastax.com/dev/blog/deploying-cassandra-across-multiple-data-centers
> http://www.datastax.com/products/opscenter
> 
> Hope this helps...
> 
> Regards,
> </VJ>
> 
> 
> 
> On Wed, Jun 15, 2011 at 7:24 PM, AJ <aj...@dude.podzone.net> wrote:
> Ok.  I understand the reasoning you laid out.  But, I think it should be documented more thoroughly.  I was trying to get an idea as to how flexible Cass lets you be with the various combinations of strategies, snitches, token ranges, etc..
> 
> It would be instructional to see what a graphical representation of a cluster ring with multiple data centers looks like.  Google turned-up nothing.  I imagine it's a multilayer ring; one layer per data center with the nodes of one layer slightly offset from the ones in the other (based on the example in the wiki).  I would also like to know which node is next in the ring such so as to understand replica placement in, for example, the OldNetworkTopologyStrategy when it's doc states,
> 
> "...It places one replica in a different data center from the first (if there is any such data center), the third replica in a different rack in the first datacenter, and any remaining replicas on the first unused nodes on the ring."
> 
> I can only assume for now that "the ring" referred to is the "local" ring of the first data center.
> 
> 
> 
> On 6/15/2011 5:51 PM, Vijay wrote:
>> 
>> No it wont.... it will assume you are doing the right thing... 
>> 
>> Regards,
>> </VJ>
>> 
>> 
>> 
>> On Wed, Jun 15, 2011 at 2:34 PM, AJ <aj...@dude.podzone.net> wrote:
>> Vijay, thank you for your thoughtful reply.  Will Cass complain if I don't setup my tokens like in the examples?
>> 
>> 
>> On 6/15/2011 2:41 PM, Vijay wrote:
>>> 
>>> All you heard is right... 
>>> You are not overriding Cassandra's token assignment by saying here is your token...
>>> 
>>> Logic is:
>>> Calculate a token for the given key...
>>> find the node in each region independently (If you use NTS and if you set the strategy options which says you want to replicate to the other region)... 
>>> Search for the ranges in each region independntly
>>> Replicate the data to that node.
>>> 
>>> For multi DC cassandra needs nodes to be equally partitioned within each dc (If you care that the load equally distributed).... as well                         as there shouldn't be any collusion of tokens within a cluster....
>>> 
>>> The documentation tried to explain the same and the example in the documentation. 
>>> Hope this clarifies...
>>> 
>>> More examples if it helps....
>>> 
>>> DC1 Node 1 : token 0
>>> DC1 Node 2 : token 8..
>>> 
>>> DC2 Node 1 : token 4..
>>> DC2 Node 1 : token 12..
>>> 
>>> or
>>> 
>>> DC1 Node 1 : token 0
>>> DC1 Node 2 : token 1..
>>> 
>>> DC2 Node 1 : token 8..
>>> DC2 Node 1 : token  7..
>>> 
>>> Regards,
>>> </VJ>
>>> 
>>> 
>>> 
>>> On Wed, Jun 15, 2011 at 12:28 PM, AJ <aj...@dude.podzone.net> wrote:
>>> On 6/15/2011 12:14 PM, Vijay wrote:
>>>> 
>>>> Correction.... 
>>>> 
>>>> "The problem in the above approach is you have 2 nodes between 12 to 4 in DC1 but from 4 to 12  you just have 1"
>>>> 
>>>> should be
>>>> 
>>>> "The problem in the above approach is you have 1 node between 0-4 (25%) and and one node covering the rest which is 4-16, 0-0 (75%)"
>>>> 
>>>> Regards,
>>>> </VJ>
>>>> 
>>> 
>>> Ok, I think you are saying that the computed token range intervals are incorrect and that they would be:
>>> 
>>> DC1
>>> *node 1 = 0      Range: (4, 16], (0, 0]
>>> 
>>> node 2 = 4      Range: (0, 4]
>>> 
>>> DC2
>>> *node 3 = 8      Range: (12, 16], (0, 8]
>>> 
>>> node 4 = 12   Range: (8, 12]
>>> 
>>> If so, then yes, this is what I am seeking to confirm since I haven't found any documentation stating this directly and that reference that I gave only implies this; that is, that the token ranges are calculated per data center rather than per cluster.  I just need someone to confirm that 100% because it doesn't sound right to me based on everything else I've read.  
>>> 
>>> SO, the question is:  Does Cass calculate the consecutive node token ranges A.) per cluster, or B.) for the whole data center?
>>> 
>>> From all I understand, the answer is B.  But, that documentation (reprinted below) implies A... or something that doesn't make sense to me because of the token placement in the example:
>>> 
>>> "With NetworkTopologyStrategy, you should calculate the tokens the nodes in each DC independantly...
>>> 
>>> DC1
>>> node 1 = 0
>>> node 2 = 85070591730234615865843651857942052864
>>> 
>>> DC2
>>> node 3 = 1
>>> node 4 = 850705917302346158658436518579
>>> 42052865"
>>> 
>>> 
>>> However, I do see why this would be helpful, but first I'm just asking if this token assignment is absolutely mandatory 
>>> or if it's just a technique to achieve some end.
>>> 
>>> 
>>> 
>> 
>> 
> 
> 


Re: Docs: Token Selection

Posted by Vijay <vi...@gmail.com>.
+1 for more documentation (I guess contributions are always welcomed).... I
will try to write it down sometime when we have a bit more time...

0.8 nodetool ring command adds the DC and RAC information....

http://www.datastax.com/dev/blog/deploying-cassandra-across-multiple-data-centers
http://www.datastax.com/products/opscenter

Hope this helps...

Regards,
</VJ>



On Wed, Jun 15, 2011 at 7:24 PM, AJ <aj...@dude.podzone.net> wrote:

>  Ok.  I understand the reasoning you laid out.  But, I think it should be
> documented more thoroughly.  I was trying to get an idea as to how flexible
> Cass lets you be with the various combinations of strategies, snitches,
> token ranges, etc..
>
> It would be instructional to see what a graphical representation of a
> cluster ring with multiple data centers looks like.  Google turned-up
> nothing.  I imagine it's a multilayer ring; one layer per data center with
> the nodes of one layer slightly offset from the ones in the other (based on
> the example in the wiki).  I would also like to know which node is next in
> the ring such so as to understand replica placement in, for example, the
> OldNetworkTopologyStrategy when it's doc states,
>
> "...It places one replica in a different data center from the first (if
> there is any such data center), the third replica in a different rack in the
> first datacenter, and any remaining replicas on the first unused nodes on
> the ring."
>
> I can only assume for now that "the ring" referred to is the "local" ring
> of the first data center.
>
>
>
> On 6/15/2011 5:51 PM, Vijay wrote:
>
> No it wont.... it will assume you are doing the right thing...
>
> Regards,
> </VJ>
>
>
>
> On Wed, Jun 15, 2011 at 2:34 PM, AJ <aj...@dude.podzone.net> wrote:
>
>>  Vijay, thank you for your thoughtful reply.  Will Cass complain if I
>> don't setup my tokens like in the examples?
>>
>>
>> On 6/15/2011 2:41 PM, Vijay wrote:
>>
>> All you heard is right...
>> You are not overriding Cassandra's token assignment by saying here is your
>> token...
>>
>>  Logic is:
>> Calculate a token for the given key...
>> find the node in each region independently (If you use NTS and if you set
>> the strategy options which says you want to replicate to the other
>> region)...
>> Search for the ranges in each region independntly
>> Replicate the data to that node.
>>
>> For multi DC cassandra needs nodes to be equally partitioned within each
>> dc (If you care that the load equally distributed).... as well as
>> there shouldn't be any collusion of tokens within a cluster....
>>
>>  The documentation tried to explain the same and the example in the
>> documentation.
>> Hope this clarifies...
>>
>>  More examples if it helps....
>>
>>   DC1 Node 1 : token 0
>> DC1 Node 2 : token 8..
>>
>>  DC2 Node 1 : token 4..
>> DC2 Node 1 : token 12..
>>
>>  or
>>
>>  DC1 Node 1 : token 0
>> DC1 Node 2 : token 1..
>>
>>  DC2 Node 1 : token 8..
>> DC2 Node 1 : token  7..
>>
>>  Regards,
>> </VJ>
>>
>>
>>
>> On Wed, Jun 15, 2011 at 12:28 PM, AJ <aj...@dude.podzone.net> wrote:
>>
>>>  On 6/15/2011 12:14 PM, Vijay wrote:
>>>
>>> Correction....
>>>
>>>  "The problem in the above approach is you have 2 nodes between 12 to 4
>>> in DC1 but from 4 to 12  you just have 1"
>>>
>>>  should be
>>>
>>>  "The problem in the above approach is you have 1 node between 0-4 (25%)
>>> and and one node covering the rest which is 4-16, 0-0 (75%)"
>>>
>>> Regards,
>>> </VJ>
>>>
>>>
>>>  Ok, I think you are saying that the computed token range intervals are
>>> incorrect and that they would be:
>>>
>>> DC1
>>> *node 1 = 0      Range: (4, 16], (0, 0]
>>>
>>> node 2 = 4      Range: (0, 4]
>>>
>>> DC2
>>>  *node 3 = 8      Range: (12, 16], (0, 8]
>>>
>>> node 4 = 12   Range: (8, 12]
>>>
>>>  If so, then yes, this is what I am seeking to confirm since I haven't
>>> found any documentation stating this directly and that reference that I gave
>>> only implies this; that is, that the token ranges are calculated per data
>>> center rather than per cluster.  I just need someone to confirm that 100%
>>> because it doesn't sound right to me based on everything else I've read.
>>>
>>> SO, the question is:  Does Cass calculate the consecutive node token
>>> ranges A.) per cluster, or B.) for the whole data center?
>>>
>>> From all I understand, the answer is B.  But, that documentation
>>> (reprinted below) implies A... or something that doesn't make sense to me
>>> because of the token placement in the example:
>>>
>>> "With NetworkTopologyStrategy, you should calculate the tokens the nodes
>>> in each DC independantly...
>>>
>>> DC1
>>> node 1 = 0
>>> node 2 = 85070591730234615865843651857942052864
>>>
>>> DC2
>>> node 3 = 1
>>> node 4 = 850705917302346158658436518579
>>> 42052865"
>>>
>>>
>>> However, I do see why this would be helpful, but first I'm just asking if this token assignment is absolutely mandatory
>>> or if it's just a technique to achieve some end.
>>>
>>>
>>>
>>>
>>
>>
>
>

Re: Docs: Token Selection

Posted by AJ <aj...@dude.podzone.net>.
Ok.  I understand the reasoning you laid out.  But, I think it should be 
documented more thoroughly.  I was trying to get an idea as to how 
flexible Cass lets you be with the various combinations of strategies, 
snitches, token ranges, etc..

It would be instructional to see what a graphical representation of a 
cluster ring with multiple data centers looks like.  Google turned-up 
nothing.  I imagine it's a multilayer ring; one layer per data center 
with the nodes of one layer slightly offset from the ones in the other 
(based on the example in the wiki).  I would also like to know which 
node is next in the ring such so as to understand replica placement in, 
for example, the OldNetworkTopologyStrategy when it's doc states,

"...It places one replica in a different data center from the first (if 
there is any such data center), the third replica in a different rack in 
the first datacenter, and any remaining replicas on the first unused 
nodes on the ring."

I can only assume for now that "the ring" referred to is the "local" 
ring of the first data center.


On 6/15/2011 5:51 PM, Vijay wrote:
> No it wont.... it will assume you are doing the right thing...
>
> Regards,
> </VJ>
>
>
>
> On Wed, Jun 15, 2011 at 2:34 PM, AJ <aj@dude.podzone.net 
> <ma...@dude.podzone.net>> wrote:
>
>     Vijay, thank you for your thoughtful reply.  Will Cass complain if
>     I don't setup my tokens like in the examples?
>
>
>     On 6/15/2011 2:41 PM, Vijay wrote:
>>     All you heard is right...
>>     You are not overriding Cassandra's token assignment by saying
>>     here is your token...
>>
>>     Logic is:
>>     Calculate a token for the given key...
>>     find the node in each region independently (If you use NTS and if
>>     you set the strategy options which says you want to replicate to
>>     the other region)...
>>     Search for the ranges in each region independntly
>>     Replicate the data to that node.
>>
>>     For multi DC cassandra needs nodes to be equally partitioned
>>     within each dc (If you care that the load equally
>>     distributed).... as well as there shouldn't be any collusion of
>>     tokens within a cluster....
>>
>>     The documentation tried to explain the same and the example in
>>     the documentation.
>>     Hope this clarifies...
>>
>>     More examples if it helps....
>>
>>     DC1 Node 1 : token 0
>>     DC1 Node 2 : token 8..
>>
>>     DC2 Node 1 : token 4..
>>     DC2 Node 1 : token 12..
>>
>>     or
>>
>>     DC1 Node 1 : token 0
>>     DC1 Node 2 : token 1..
>>
>>     DC2 Node 1 : token 8..
>>     DC2 Node 1 : token  7..
>>
>>     Regards,
>>     </VJ>
>>
>>
>>
>>     On Wed, Jun 15, 2011 at 12:28 PM, AJ <aj@dude.podzone.net
>>     <ma...@dude.podzone.net>> wrote:
>>
>>         On 6/15/2011 12:14 PM, Vijay wrote:
>>>         Correction....
>>>
>>>         "The problem in the above approach is you have 2 nodes
>>>         between 12 to 4 in DC1 but from 4 to 12  you just have 1"
>>>
>>>         should be
>>>
>>>         "The problem in the above approach is you have 1 node
>>>         between 0-4 (25%) and and one node covering the rest which
>>>         is 4-16, 0-0 (75%)"
>>>
>>>         Regards,
>>>         </VJ>
>>>
>>
>>         Ok, I think you are saying that the computed token range
>>         intervals are incorrect and that they would be:
>>
>>         DC1
>>         *node 1 = 0      Range: (4, 16], (0, 0]
>>
>>         node 2 = 4      Range: (0, 4]
>>
>>         DC2
>>         *node 3 = 8      Range: (12, 16], (0, 8]
>>
>>         node 4 = 12   Range: (8, 12]
>>
>>         If so, then yes, this is what I am seeking to confirm since I
>>         haven't found any documentation stating this directly and
>>         that reference that I gave only implies this; that is, that
>>         the token ranges are calculated per data center rather than
>>         per cluster.  I just need someone to confirm that 100%
>>         because it doesn't sound right to me based on everything else
>>         I've read.
>>
>>         SO, the question is:  Does Cass calculate the consecutive
>>         node token ranges A.) per cluster, or B.) for the whole data
>>         center?
>>
>>         From all I understand, the answer is B.  But, that
>>         documentation (reprinted below) implies A... or something
>>         that doesn't make sense to me because of the token placement
>>         in the example:
>>
>>         "With NetworkTopologyStrategy, you should calculate the
>>         tokens the nodes in each DC independantly...
>>
>>         DC1 node 1 = 0 node 2 =
>>         85070591730234615865843651857942052864 DC2 node 3 = 1 node 4
>>         = 850705917302346158658436518579
>>         42052865"
>>
>>
>>         However, I do see why this would be helpful, but first I'm just asking if this token assignment is absolutely mandatory
>>         or if it's just a technique to achieve some end.
>>
>>
>>
>>
>
>


Re: Docs: Token Selection

Posted by Vijay <vi...@gmail.com>.
No it wont.... it will assume you are doing the right thing...

Regards,
</VJ>



On Wed, Jun 15, 2011 at 2:34 PM, AJ <aj...@dude.podzone.net> wrote:

>  Vijay, thank you for your thoughtful reply.  Will Cass complain if I don't
> setup my tokens like in the examples?
>
>
> On 6/15/2011 2:41 PM, Vijay wrote:
>
> All you heard is right...
> You are not overriding Cassandra's token assignment by saying here is your
> token...
>
>  Logic is:
> Calculate a token for the given key...
> find the node in each region independently (If you use NTS and if you set
> the strategy options which says you want to replicate to the other
> region)...
> Search for the ranges in each region independntly
> Replicate the data to that node.
>
> For multi DC cassandra needs nodes to be equally partitioned within each
> dc (If you care that the load equally distributed).... as well as
> there shouldn't be any collusion of tokens within a cluster....
>
>  The documentation tried to explain the same and the example in the
> documentation.
> Hope this clarifies...
>
>  More examples if it helps....
>
>   DC1 Node 1 : token 0
> DC1 Node 2 : token 8..
>
>  DC2 Node 1 : token 4..
> DC2 Node 1 : token 12..
>
>  or
>
>  DC1 Node 1 : token 0
> DC1 Node 2 : token 1..
>
>  DC2 Node 1 : token 8..
> DC2 Node 1 : token  7..
>
>  Regards,
> </VJ>
>
>
>
> On Wed, Jun 15, 2011 at 12:28 PM, AJ <aj...@dude.podzone.net> wrote:
>
>>  On 6/15/2011 12:14 PM, Vijay wrote:
>>
>> Correction....
>>
>>  "The problem in the above approach is you have 2 nodes between 12 to 4
>> in DC1 but from 4 to 12  you just have 1"
>>
>>  should be
>>
>>  "The problem in the above approach is you have 1 node between 0-4 (25%)
>> and and one node covering the rest which is 4-16, 0-0 (75%)"
>>
>> Regards,
>> </VJ>
>>
>>
>>  Ok, I think you are saying that the computed token range intervals are
>> incorrect and that they would be:
>>
>> DC1
>> *node 1 = 0      Range: (4, 16], (0, 0]
>>
>> node 2 = 4      Range: (0, 4]
>>
>> DC2
>>  *node 3 = 8      Range: (12, 16], (0, 8]
>>
>> node 4 = 12   Range: (8, 12]
>>
>>  If so, then yes, this is what I am seeking to confirm since I haven't
>> found any documentation stating this directly and that reference that I gave
>> only implies this; that is, that the token ranges are calculated per data
>> center rather than per cluster.  I just need someone to confirm that 100%
>> because it doesn't sound right to me based on everything else I've read.
>>
>> SO, the question is:  Does Cass calculate the consecutive node token
>> ranges A.) per cluster, or B.) for the whole data center?
>>
>> From all I understand, the answer is B.  But, that documentation
>> (reprinted below) implies A... or something that doesn't make sense to me
>> because of the token placement in the example:
>>
>> "With NetworkTopologyStrategy, you should calculate the tokens the nodes
>> in each DC independantly...
>>
>> DC1
>> node 1 = 0
>> node 2 = 85070591730234615865843651857942052864
>>
>> DC2
>> node 3 = 1
>> node 4 = 850705917302346158658436518579
>> 42052865"
>>
>>
>> However, I do see why this would be helpful, but first I'm just asking if this token assignment is absolutely mandatory
>> or if it's just a technique to achieve some end.
>>
>>
>>
>>
>
>

Re: Docs: Token Selection

Posted by AJ <aj...@dude.podzone.net>.
Vijay, thank you for your thoughtful reply.  Will Cass complain if I 
don't setup my tokens like in the examples?

On 6/15/2011 2:41 PM, Vijay wrote:
> All you heard is right...
> You are not overriding Cassandra's token assignment by saying here is 
> your token...
>
> Logic is:
> Calculate a token for the given key...
> find the node in each region independently (If you use NTS and if you 
> set the strategy options which says you want to replicate to the other 
> region)...
> Search for the ranges in each region independntly
> Replicate the data to that node.
>
> For multi DC cassandra needs nodes to be equally partitioned 
> within each dc (If you care that the load equally distributed).... as 
> well as there shouldn't be any collusion of tokens within a cluster....
>
> The documentation tried to explain the same and the example in the 
> documentation.
> Hope this clarifies...
>
> More examples if it helps....
>
> DC1 Node 1 : token 0
> DC1 Node 2 : token 8..
>
> DC2 Node 1 : token 4..
> DC2 Node 1 : token 12..
>
> or
>
> DC1 Node 1 : token 0
> DC1 Node 2 : token 1..
>
> DC2 Node 1 : token 8..
> DC2 Node 1 : token  7..
>
> Regards,
> </VJ>
>
>
>
> On Wed, Jun 15, 2011 at 12:28 PM, AJ <aj@dude.podzone.net 
> <ma...@dude.podzone.net>> wrote:
>
>     On 6/15/2011 12:14 PM, Vijay wrote:
>>     Correction....
>>
>>     "The problem in the above approach is you have 2 nodes between 12
>>     to 4 in DC1 but from 4 to 12  you just have 1"
>>
>>     should be
>>
>>     "The problem in the above approach is you have 1 node between 0-4
>>     (25%) and and one node covering the rest which is 4-16, 0-0 (75%)"
>>
>>     Regards,
>>     </VJ>
>>
>
>     Ok, I think you are saying that the computed token range intervals
>     are incorrect and that they would be:
>
>     DC1
>     *node 1 = 0      Range: (4, 16], (0, 0]
>
>     node 2 = 4      Range: (0, 4]
>
>     DC2
>     *node 3 = 8      Range: (12, 16], (0, 8]
>
>     node 4 = 12   Range: (8, 12]
>
>     If so, then yes, this is what I am seeking to confirm since I
>     haven't found any documentation stating this directly and that
>     reference that I gave only implies this; that is, that the token
>     ranges are calculated per data center rather than per cluster.  I
>     just need someone to confirm that 100% because it doesn't sound
>     right to me based on everything else I've read.
>
>     SO, the question is:  Does Cass calculate the consecutive node
>     token ranges A.) per cluster, or B.) for the whole data center?
>
>     From all I understand, the answer is B.  But, that documentation
>     (reprinted below) implies A... or something that doesn't make
>     sense to me because of the token placement in the example:
>
>     "With NetworkTopologyStrategy, you should calculate the tokens the
>     nodes in each DC independantly...
>
>     DC1 node 1 = 0 node 2 = 85070591730234615865843651857942052864 DC2
>     node 3 = 1 node 4 = 850705917302346158658436518579
>     42052865"
>
>
>     However, I do see why this would be helpful, but first I'm just asking if this token assignment is absolutely mandatory
>     or if it's just a technique to achieve some end.
>
>
>
>


Re: Docs: Token Selection

Posted by Vijay <vi...@gmail.com>.
All you heard is right...
You are not overriding Cassandra's token assignment by saying here is your
token...

Logic is:
Calculate a token for the given key...
find the node in each region independently (If you use NTS and if you set
the strategy options which says you want to replicate to the other
region)...
Search for the ranges in each region independntly
Replicate the data to that node.

For multi DC cassandra needs nodes to be equally partitioned within each
dc (If you care that the load equally distributed).... as well as
there shouldn't be any collusion of tokens within a cluster....

The documentation tried to explain the same and the example in the
documentation.
Hope this clarifies...

More examples if it helps....

DC1 Node 1 : token 0
DC1 Node 2 : token 8..

DC2 Node 1 : token 4..
DC2 Node 1 : token 12..

or

DC1 Node 1 : token 0
DC1 Node 2 : token 1..

DC2 Node 1 : token 8..
DC2 Node 1 : token  7..

Regards,
</VJ>



On Wed, Jun 15, 2011 at 12:28 PM, AJ <aj...@dude.podzone.net> wrote:

>  On 6/15/2011 12:14 PM, Vijay wrote:
>
> Correction....
>
>  "The problem in the above approach is you have 2 nodes between 12 to 4 in
> DC1 but from 4 to 12  you just have 1"
>
>  should be
>
>  "The problem in the above approach is you have 1 node between 0-4 (25%)
> and and one node covering the rest which is 4-16, 0-0 (75%)"
>
> Regards,
> </VJ>
>
>
> Ok, I think you are saying that the computed token range intervals are
> incorrect and that they would be:
>
> DC1
> *node 1 = 0      Range: (4, 16], (0, 0]
>
> node 2 = 4      Range: (0, 4]
>
> DC2
> *node 3 = 8      Range: (12, 16], (0, 8]
>
> node 4 = 12   Range: (8, 12]
>
> If so, then yes, this is what I am seeking to confirm since I haven't found
> any documentation stating this directly and that reference that I gave only
> implies this; that is, that the token ranges are calculated per data center
> rather than per cluster.  I just need someone to confirm that 100% because
> it doesn't sound right to me based on everything else I've read.
>
> SO, the question is:  Does Cass calculate the consecutive node token ranges
> A.) per cluster, or B.) for the whole data center?
>
> From all I understand, the answer is B.  But, that documentation (reprinted
> below) implies A... or something that doesn't make sense to me because of
> the token placement in the example:
>
> "With NetworkTopologyStrategy, you should calculate the tokens the nodes in
> each DC independantly...
>
> DC1
> node 1 = 0
> node 2 = 85070591730234615865843651857942052864
>
> DC2
> node 3 = 1
> node 4 = 850705917302346158658436518579
> 42052865"
>
>
> However, I do see why this would be helpful, but first I'm just asking if this token assignment is absolutely mandatory
> or if it's just a technique to achieve some end.
>
>
>
>

Re: Docs: Token Selection

Posted by AJ <aj...@dude.podzone.net>.
On 6/15/2011 12:14 PM, Vijay wrote:
> Correction....
>
> "The problem in the above approach is you have 2 nodes between 12 to 4 
> in DC1 but from 4 to 12  you just have 1"
>
> should be
>
> "The problem in the above approach is you have 1 node between 0-4 
> (25%) and and one node covering the rest which is 4-16, 0-0 (75%)"
>
> Regards,
> </VJ>
>

Ok, I think you are saying that the computed token range intervals are 
incorrect and that they would be:

DC1
*node 1 = 0      Range: (4, 16], (0, 0]
node 2 = 4      Range: (0, 4]

DC2
*node 3 = 8      Range: (12, 16], (0, 8]
node 4 = 12   Range: (8, 12]

If so, then yes, this is what I am seeking to confirm since I haven't 
found any documentation stating this directly and that reference that I 
gave only implies this; that is, that the token ranges are calculated 
per data center rather than per cluster.  I just need someone to confirm 
that 100% because it doesn't sound right to me based on everything else 
I've read.

SO, the question is:  Does Cass calculate the consecutive node token 
ranges A.) per cluster, or B.) for the whole data center?

 From all I understand, the answer is B.  But, that documentation 
(reprinted below) implies A... or something that doesn't make sense to 
me because of the token placement in the example:

"With NetworkTopologyStrategy, you should calculate the tokens the nodes 
in each DC independantly...

DC1
node 1 = 0
node 2 = 85070591730234615865843651857942052864

DC2
node 3 = 1
node 4 = 85070591730234615865843651857942052865"


However, I do see why this would be helpful, but first I'm just asking if this token assignment is absolutely mandatory
or if it's just a technique to achieve some end.




Re: Docs: Token Selection

Posted by Vijay <vi...@gmail.com>.
Correction....

"The problem in the above approach is you have 2 nodes between 12 to 4 in
DC1 but from 4 to 12  you just have 1"

should be

"The problem in the above approach is you have 1 node between 0-4 (25%) and
and one node covering the rest which is 4-16, 0-0 (75%)"

Regards,
</VJ>



On Wed, Jun 15, 2011 at 11:10 AM, Vijay <vi...@gmail.com> wrote:

> The problem in the above approach is you have 2 nodes between 12 to 4 in
> DC1 but from 4 to 12  you just have 1.... (Which will cause uneven
> distribution of data the node)
> It is easier to think of the DCs as ring and split equally and interleave
> them together....
>
> DC1 Node 1 : token 0
> DC1 Node 2 : token 8..
>
> DC2 Node 1 : token 4..
> DC2 Node 1 : token 12..
>
> Regards,
> </VJ>
>
>
>
>
> On Tue, Jun 14, 2011 at 7:31 PM, AJ <aj...@dude.podzone.net> wrote:
>
>>  Yes, which means that the ranges overlap each other.
>>
>> Is this just a convention, or is it technically required when using
>> NetworkTopologyStrategy?  Would it be acceptable to split the ranges into
>> quarters by ignoring the data centers, such as:
>>
>> DC1
>> node 1 = 0      Range: (12, 16], (0, 0]
>> node 2 = 4      Range: (0, 4]
>>
>> DC2
>> node 3 = 8      Range: (4, 8]
>> node 4 = 12   Range: (8, 12]
>>
>> If this is OK, are there any drawbacks to this?
>>
>>
>>
>> On 6/14/2011 6:10 PM, Vijay wrote:
>>
>> Yes... Thats right...  If you are trying to say the below...
>>
>>  DC1
>>  Node1 Owns 50%
>>
>>  (Ranges 8..4 -> 8..5 & 8..5 -> 0)
>>
>> Node2 Owns 50%
>>
>> (Ranges 0 -> 1 & 1 -> 8..4)
>>
>>
>>  DC2
>>  Node1 Owns 50%
>>
>>  (Ranges 8..5 -> 0 & 0 -> 1)
>>
>>   Node2 Owns 50%
>>
>>   (Ranges 1 -> 8..4 & 8..4 -> 8..5)
>>
>>
>>  Regards,
>> </VJ>
>>
>>
>>
>> On Tue, Jun 14, 2011 at 3:47 PM, AJ <aj...@dude.podzone.net> wrote:
>>
>>> This http://wiki.apache.org/cassandra/Operations#Token_selection  says:
>>>
>>> "With NetworkTopologyStrategy, you should calculate the tokens the nodes
>>> in each DC independantly."
>>>
>>> and gives the example:
>>>
>>> DC1
>>> node 1 = 0
>>> node 2 = 85070591730234615865843651857942052864
>>>
>>> DC2
>>> node 3 = 1
>>> node 4 = 85070591730234615865843651857942052865
>>>
>>>
>>> So, according to the above, the token ranges would be (abbreviated nums):
>>>
>>> DC1
>>> node 1 = 0      Range: (8..4, 16], (0, 0]
>>> node 2 = 8..4   Range: (0, 8..4]
>>>
>>> DC2
>>> node 3 = 1      Range: (8..5, 16], (0, 1]
>>> node 4 = 8..5   Range: (1, 8..5]
>>>
>>>
>>> If the above is correct, then I would be surprised as this paragraph is
>>> the only place were one would discover this and may be easy to miss...
>>> unless there's a doc buried somewhere in plain view that I missed.
>>>
>>> So, have I interpreted this paragraph correctly?  Was this design to help
>>> keep data somewhat localized if that was important, such as a geographically
>>> dispersed DC?
>>>
>>> Thanks!
>>>
>>
>>
>>
>

Re: Docs: Token Selection

Posted by Vijay <vi...@gmail.com>.
The problem in the above approach is you have 2 nodes between 12 to 4 in DC1
but from 4 to 12  you just have 1.... (Which will cause uneven distribution
of data the node)
It is easier to think of the DCs as ring and split equally and interleave
them together....

DC1 Node 1 : token 0
DC1 Node 2 : token 8..

DC2 Node 1 : token 4..
DC2 Node 1 : token 12..

Regards,
</VJ>



On Tue, Jun 14, 2011 at 7:31 PM, AJ <aj...@dude.podzone.net> wrote:

>  Yes, which means that the ranges overlap each other.
>
> Is this just a convention, or is it technically required when using
> NetworkTopologyStrategy?  Would it be acceptable to split the ranges into
> quarters by ignoring the data centers, such as:
>
> DC1
> node 1 = 0      Range: (12, 16], (0, 0]
> node 2 = 4      Range: (0, 4]
>
> DC2
> node 3 = 8      Range: (4, 8]
> node 4 = 12   Range: (8, 12]
>
> If this is OK, are there any drawbacks to this?
>
>
>
> On 6/14/2011 6:10 PM, Vijay wrote:
>
> Yes... Thats right...  If you are trying to say the below...
>
>  DC1
>  Node1 Owns 50%
>
>  (Ranges 8..4 -> 8..5 & 8..5 -> 0)
>
> Node2 Owns 50%
>
> (Ranges 0 -> 1 & 1 -> 8..4)
>
>
>  DC2
>  Node1 Owns 50%
>
>  (Ranges 8..5 -> 0 & 0 -> 1)
>
>   Node2 Owns 50%
>
>   (Ranges 1 -> 8..4 & 8..4 -> 8..5)
>
>
>  Regards,
> </VJ>
>
>
>
> On Tue, Jun 14, 2011 at 3:47 PM, AJ <aj...@dude.podzone.net> wrote:
>
>> This http://wiki.apache.org/cassandra/Operations#Token_selection  says:
>>
>> "With NetworkTopologyStrategy, you should calculate the tokens the nodes
>> in each DC independantly."
>>
>> and gives the example:
>>
>> DC1
>> node 1 = 0
>> node 2 = 85070591730234615865843651857942052864
>>
>> DC2
>> node 3 = 1
>> node 4 = 85070591730234615865843651857942052865
>>
>>
>> So, according to the above, the token ranges would be (abbreviated nums):
>>
>> DC1
>> node 1 = 0      Range: (8..4, 16], (0, 0]
>> node 2 = 8..4   Range: (0, 8..4]
>>
>> DC2
>> node 3 = 1      Range: (8..5, 16], (0, 1]
>> node 4 = 8..5   Range: (1, 8..5]
>>
>>
>> If the above is correct, then I would be surprised as this paragraph is
>> the only place were one would discover this and may be easy to miss...
>> unless there's a doc buried somewhere in plain view that I missed.
>>
>> So, have I interpreted this paragraph correctly?  Was this design to help
>> keep data somewhat localized if that was important, such as a geographically
>> dispersed DC?
>>
>> Thanks!
>>
>
>
>

Re: Docs: Token Selection

Posted by AJ <aj...@dude.podzone.net>.
Yes, which means that the ranges overlap each other.

Is this just a convention, or is it technically required when using 
NetworkTopologyStrategy?  Would it be acceptable to split the ranges 
into quarters by ignoring the data centers, such as:

DC1
node 1 = 0      Range: (12, 16], (0, 0]
node 2 = 4      Range: (0, 4]

DC2
node 3 = 8      Range: (4, 8]
node 4 = 12   Range: (8, 12]

If this is OK, are there any drawbacks to this?



On 6/14/2011 6:10 PM, Vijay wrote:
> Yes... Thats right...  If you are trying to say the below...
>
> DC1
> Node1 Owns 50%
>
>     (Ranges 8..4 -> 8..5 & 8..5 -> 0)
>
> Node2 Owns 50%
>
>     (Ranges 0 -> 1 & 1 -> 8..4)
>
>
> DC2
> Node1 Owns 50%
>
>     (Ranges 8..5 -> 0 & 0 -> 1)
>
> Node2 Owns 50%
>
>     (Ranges 1 -> 8..4 & 8..4 -> 8..5)
>
>
> Regards,
> </VJ>
>
>
>
> On Tue, Jun 14, 2011 at 3:47 PM, AJ <aj@dude.podzone.net 
> <ma...@dude.podzone.net>> wrote:
>
>     This http://wiki.apache.org/cassandra/Operations#Token_selection
>      says:
>
>     "With NetworkTopologyStrategy, you should calculate the tokens the
>     nodes in each DC independantly."
>
>     and gives the example:
>
>     DC1
>     node 1 = 0
>     node 2 = 85070591730234615865843651857942052864
>
>     DC2
>     node 3 = 1
>     node 4 = 85070591730234615865843651857942052865
>
>
>     So, according to the above, the token ranges would be (abbreviated
>     nums):
>
>     DC1
>     node 1 = 0      Range: (8..4, 16], (0, 0]
>     node 2 = 8..4   Range: (0, 8..4]
>
>     DC2
>     node 3 = 1      Range: (8..5, 16], (0, 1]
>     node 4 = 8..5   Range: (1, 8..5]
>
>
>     If the above is correct, then I would be surprised as this
>     paragraph is the only place were one would discover this and may
>     be easy to miss... unless there's a doc buried somewhere in plain
>     view that I missed.
>
>     So, have I interpreted this paragraph correctly?  Was this design
>     to help keep data somewhat localized if that was important, such
>     as a geographically dispersed DC?
>
>     Thanks!
>
>


Re: Docs: Token Selection

Posted by Vijay <vi...@gmail.com>.
Yes... Thats right...  If you are trying to say the below...

DC1
Node1 Owns 50%

(Ranges 8..4 -> 8..5 & 8..5 -> 0)

Node2 Owns 50%

(Ranges 0 -> 1 & 1 -> 8..4)


DC2
Node1 Owns 50%

(Ranges 8..5 -> 0 & 0 -> 1)

Node2 Owns 50%

(Ranges 1 -> 8..4 & 8..4 -> 8..5)


Regards,
</VJ>



On Tue, Jun 14, 2011 at 3:47 PM, AJ <aj...@dude.podzone.net> wrote:

> This http://wiki.apache.org/cassandra/Operations#Token_selection  says:
>
> "With NetworkTopologyStrategy, you should calculate the tokens the nodes in
> each DC independantly."
>
> and gives the example:
>
> DC1
> node 1 = 0
> node 2 = 85070591730234615865843651857942052864
>
> DC2
> node 3 = 1
> node 4 = 85070591730234615865843651857942052865
>
>
> So, according to the above, the token ranges would be (abbreviated nums):
>
> DC1
> node 1 = 0      Range: (8..4, 16], (0, 0]
> node 2 = 8..4   Range: (0, 8..4]
>
> DC2
> node 3 = 1      Range: (8..5, 16], (0, 1]
> node 4 = 8..5   Range: (1, 8..5]
>
>
> If the above is correct, then I would be surprised as this paragraph is the
> only place were one would discover this and may be easy to miss... unless
> there's a doc buried somewhere in plain view that I missed.
>
> So, have I interpreted this paragraph correctly?  Was this design to help
> keep data somewhat localized if that was important, such as a geographically
> dispersed DC?
>
> Thanks!
>