Re: Deployment on AWS and replication strategies

[Mike Gallamore <mi...@googlemail.com>]

Hi everyone,

At my work we are in the early stages of moving our data which lives on EC2 machines from a Flare/memcache system to Cassandra so your chat has been interesting to me.

I realize that this might complicate things and make things less "simple" but would it be useful for the nodes themselves to advertise some of their info? So for example a node starts to bootstrap, it pushes its specs over to the seed node, the seed node uses that to figure out what configuration to push back.

Useful things that nodes could advertise:

data-centre they are in,
performance info: mem, CPU etc (these could be used to more intelligently decide how to partition the data that the new node gets for example)
geographical info
perhaps a preferred hash range not just a token (and presumably everything else would automatically rebalance itself to make that happen)

P.S.The last two could be useful for someone if they had their data in Cassandra but it was more relevant more local to the geography. Think of something like Craigslist. Having the data corresponding to San Fransisco lists just happen to bootstrap over to a datacenter on the east coast wouldn't be very efficient. But having two completely separate datastores might not be the simplest design either. It would be nice to just tell the datastore where the info is most relevant and have it make intelligent choices of where to store things for you.

 In my case we are making a reputation system. It would be nice if we had a way to make sure that at least one replica of the data stayed on the customers machine and one or more copies over on our servers. I don't know how to do that and the reverse would be important too make sure other customers data doesn't get replicated to another customers node. I guess rather than a ring topology I'd like to try to get a star "everything in the center + location specific info at the points". An option would be to use different datastores at both ends and push updates over to the central store which would be Cassandra but that isn't as transparent as just having Cassandra nodes everywhere and just have the replication happen in a smart way.

On 2010-04-03, at 3:04 PM, Joe Stump wrote:

> 
> On Apr 3, 2010, at 2:54 PM, Benjamin Black wrote:
> 
>> I'm pretty familiar with EC2, hence the question.  I don't believe any
>> patches are required to do these things.  Regardless, as I noted in
>> that ticket, you definitely do NOT need AWS credentials to determine
>> your availability zone.  It is available through the metadata web
>> server for each instance as 'placement_availability_zone', avoiding
>> the need to speak the EC2 API or store credentials in the configs.
> 
> Good point on the metadata web server. Though I'm unsure how Cassandra would know anything about those AZ's without using code that's aware of such things, such as the rack-aware strategy we made.
> 
> Am I missing something further? I asked a friend on the EC2 networking team if you could determine AZ by IP address and he said, "No." 
> 
> --Joe
> 

Re: Deployment on AWS and replication strategies

[Mike Gallamore <mi...@googlemail.com>]

Pluggable placement: that is cool. It wasn't something that was obvious to me that was available from the documentation I read.  I thought maybe the the rackaware and rackunaware were hard coded in somewhere. I'm not a java developer so I haven't looked at the code much. That said I'll take a look and see if I can figure out how it works. I have coded in C/C++ so I probably can handle the logic part of Java code okay.
On 2010-04-04, at 1:18 AM, Benjamin Black wrote:

> On Sat, Apr 3, 2010 at 8:23 PM, Mike Gallamore
> <mi...@googlemail.com> wrote:
>>> 
>> I didn't mean a real time determination, more of if the nodes aren't identical. For example if you have a cluster made up of a bunch of EC2 light instances and decide to add a large instance, it would be nice if the new node would get a proportional amount of work based on what its system specs are.
> 
> Sure, set the token(s) appropriately.
> 
>>> 
>>>> perhaps a preferred hash range not just a token (and presumably everything else would automatically rebalance itself to make that happen)
>>>> 
>>> 
>>> Unclear what this would do.
>> Well rather than getting half of the most busy nodes work (which is how I understand it works now) you'd get an amount of work that is proportional to the power of the node.
> 
> Assuming you allow it to automatically assign its own token, the new
> node will get have the range of the node with the most data, not the
> most 'busy'.  The amount of work being done by the nodes is not a
> consideration, nor would you want automatic selection of that within
> cassandra except with significant support for long term trend
> collection and analysis, pluggable policies for calculating 'load',
> etc.
> 
>>> 
>>> Or just set the token specifically for each node you bootstrap.
>>> Starting a node and crossing your fingers on its token selection is a
>>> recipe for interesting times :)
>> Can you specify a token based on a real key value? How do you know what token to use to make sure that locally relevant data gets at least one copy stored locally?
> 
> Again, placement strategy is what you want to investigate.
> 
>> My understanding is rackawarestrategy puts the data on the next node in the token ring that is in a different datacenter. The problem is if you want a specific "other datacenter" not just the next one in the list.
> 
> Right, I suggested looking at the source as an example.  If you want a
> more sophisticated placement policy, write one.  They are not
> complicated and you will have a much deeper understanding of the
> mechanism.  IMO, pluggable placement is a remarkable feature.
> 
> 
> b

Re: Deployment on AWS and replication strategies

[Benjamin Black <b...@b3k.us>]

On Sat, Apr 3, 2010 at 8:23 PM, Mike Gallamore
<mi...@googlemail.com> wrote:
>>
> I didn't mean a real time determination, more of if the nodes aren't identical. For example if you have a cluster made up of a bunch of EC2 light instances and decide to add a large instance, it would be nice if the new node would get a proportional amount of work based on what its system specs are.

Sure, set the token(s) appropriately.

>>
>>> perhaps a preferred hash range not just a token (and presumably everything else would automatically rebalance itself to make that happen)
>>>
>>
>> Unclear what this would do.
> Well rather than getting half of the most busy nodes work (which is how I understand it works now) you'd get an amount of work that is proportional to the power of the node.

Assuming you allow it to automatically assign its own token, the new
node will get have the range of the node with the most data, not the
most 'busy'.  The amount of work being done by the nodes is not a
consideration, nor would you want automatic selection of that within
cassandra except with significant support for long term trend
collection and analysis, pluggable policies for calculating 'load',
etc.

>>
>> Or just set the token specifically for each node you bootstrap.
>> Starting a node and crossing your fingers on its token selection is a
>> recipe for interesting times :)
> Can you specify a token based on a real key value? How do you know what token to use to make sure that locally relevant data gets at least one copy stored locally?

Again, placement strategy is what you want to investigate.

> My understanding is rackawarestrategy puts the data on the next node in the token ring that is in a different datacenter. The problem is if you want a specific "other datacenter" not just the next one in the list.

Right, I suggested looking at the source as an example.  If you want a
more sophisticated placement policy, write one.  They are not
complicated and you will have a much deeper understanding of the
mechanism.  IMO, pluggable placement is a remarkable feature.


b

Re: Deployment on AWS and replication strategies

[Mike Gallamore <mi...@googlemail.com>]

Hi Benjamin,

Thanks for the reply.
On 2010-04-03, at 8:12 PM, Benjamin Black wrote:

> On Sat, Apr 3, 2010 at 3:41 PM, Mike Gallamore
> <mi...@googlemail.com> wrote:
>> 
>> Useful things that nodes could advertise:
>> 
>> data-centre they are in,
> 
> This is what the snitches do.
Cool.
> 
>> performance info: mem, CPU etc (these could be used to more intelligently decide how to partition the data that the new node gets for example)
> 
> Not convinced this is useful as it changes rapidly, so either causes
> lots of gossip or is always out of date.  Better to use a real
> monitoring system.
> 
I didn't mean a real time determination, more of if the nodes aren't identical. For example if you have a cluster made up of a bunch of EC2 light instances and decide to add a large instance, it would be nice if the new node would get a proportional amount of work based on what its system specs are.
>> geographical info
> 
> Snitches.
> 
>> perhaps a preferred hash range not just a token (and presumably everything else would automatically rebalance itself to make that happen)
>> 
> 
> Unclear what this would do.
Well rather than getting half of the most busy nodes work (which is how I understand it works now) you'd get an amount of work that is proportional to the power of the node.
> 
>> P.S.The last two could be useful for someone if they had their data in Cassandra but it was more relevant more local to the geography. Think of something like Craigslist. Having the data corresponding to San Fransisco lists just happen to bootstrap over to a datacenter on the east coast wouldn't be very efficient. But having two completely separate datastores might not be the simplest design either. It would be nice to just tell the datastore where the info is most relevant and have it make intelligent choices of where to store things for you.
>> 
> 
> Or just set the token specifically for each node you bootstrap.
> Starting a node and crossing your fingers on its token selection is a
> recipe for interesting times :)
Can you specify a token based on a real key value? How do you know what token to use to make sure that locally relevant data gets at least one copy stored locally?
> 
>>  In my case we are making a reputation system. It would be nice if we had a way to make sure that at least one replica of the data stayed on the customers machine and one or more copies over on our servers. I don't know how to do that and the reverse would be important too make sure other customers data doesn't get replicated to another customers node. I guess rather than a ring topology I'd like to try to get a star "everything in the center + location specific info at the points". An option would be to use different datastores at both ends and push updates over to the central store which would be Cassandra but that isn't as transparent as just having Cassandra nodes everywhere and just have the replication happen in a smart way.
> 
> This is what placement strategies do.  Have a look at the
> RackAwareStrategy, for example.
My understanding is rackawarestrategy puts the data on the next node in the token ring that is in a different datacenter. The problem is if you want a specific "other datacenter" not just the next one in the list.
> 
> 
> b

Re: Deployment on AWS and replication strategies

[Benjamin Black <b...@b3k.us>]

On Sat, Apr 3, 2010 at 3:41 PM, Mike Gallamore
<mi...@googlemail.com> wrote:
>
> Useful things that nodes could advertise:
>
> data-centre they are in,

This is what the snitches do.

> performance info: mem, CPU etc (these could be used to more intelligently decide how to partition the data that the new node gets for example)

Not convinced this is useful as it changes rapidly, so either causes
lots of gossip or is always out of date.  Better to use a real
monitoring system.

> geographical info

Snitches.

> perhaps a preferred hash range not just a token (and presumably everything else would automatically rebalance itself to make that happen)
>

Unclear what this would do.

> P.S.The last two could be useful for someone if they had their data in Cassandra but it was more relevant more local to the geography. Think of something like Craigslist. Having the data corresponding to San Fransisco lists just happen to bootstrap over to a datacenter on the east coast wouldn't be very efficient. But having two completely separate datastores might not be the simplest design either. It would be nice to just tell the datastore where the info is most relevant and have it make intelligent choices of where to store things for you.
>

Or just set the token specifically for each node you bootstrap.
Starting a node and crossing your fingers on its token selection is a
recipe for interesting times :)

>  In my case we are making a reputation system. It would be nice if we had a way to make sure that at least one replica of the data stayed on the customers machine and one or more copies over on our servers. I don't know how to do that and the reverse would be important too make sure other customers data doesn't get replicated to another customers node. I guess rather than a ring topology I'd like to try to get a star "everything in the center + location specific info at the points". An option would be to use different datastores at both ends and push updates over to the central store which would be Cassandra but that isn't as transparent as just having Cassandra nodes everywhere and just have the replication happen in a smart way.

This is what placement strategies do.  Have a look at the
RackAwareStrategy, for example.


b