You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@helix.apache.org by Ivan Kelly <iv...@apache.org> on 2013/04/16 15:36:16 UTC
Using partitions in Helix
Hi,
I'm sketching out the design of a prototype key value store which I'd
like to use helix in. However, I'm unclear of how to use partitions in
helix.
So, the number of partitions for a particular resource is defined when
I call #addResource from helix admin. Lets say I want to split my key
space into 5 partitions. How do I define what the boundaries for these
partitions are? Do I have to create my own mapping of key range to
partition? Or do I have to provide my own implementation of
IdealState?
Thanks
-Ivan
Re: Using partitions in Helix
Posted by Ivan Kelly <iv...@apache.org>.
Great! Thanks
-Ivan
On Tue, Apr 16, 2013 at 10:51:40AM -0700, kishore g wrote:
> This was asked earlier on the mailing list.
>
> http://markmail.org/thread/k4asnfgu7pcmrose#query:+page:1+mid:dkjaxewbrrf64wq7+state:results
>
> Here is the paste bin http://pastebin.com/e2LXtq46
>
> We also added idealstatebuilders that might make it easier.
>
> https://git-wip-us.apache.org/repos/asf?p=incubator-helix.git;a=tree;f=helix-core/src/main/java/org/apache/helix/model/builder;hb=6119a4467b0fadbab09267a68d15aa470410a182
>
> Adding it to example/recipe
>
> thanks,
> Kishore G
>
>
>
>
> On Tue, Apr 16, 2013 at 10:09 AM, Ivan Kelly <iv...@apache.org> wrote:
>
> > Hey Kishore,
> >
> > Thanks for your response.
> >
> > > 2. Set your own idealstate
> > >
> > > In Helix partition is just a logical name. So you can always generate an
> > > idealstate with partition names as follows.
> > >
> > > {resource_k1start, resource_k2start, resource_k3start} or even
> > > {resource_k1start-k1end}.. to make the mapping explicit.
> > Is there an example of this in the code/tests? I couldn't see anything
> > in the recipes.
> >
> > > Currently Helix does not understand the concept of a partition being
> > split,
> > > you will have to program it yourself by changing the idealstate during
> > > splits. We have a jira for this and we are working on the initial design.
> > > Your inputs will be valuable. I will try to add more details to the JIRA.
> > > This is an important feature and we plan to support it as a first class
> > > citizen in Helix so that one can benefit from other features like
> > > throttling ( for example, one could say that dont do more than X splits
> > > across cluster, node etc)
> > >
> > > https://issues.apache.org/jira/browse/HELIX-69.
> > I'll need to get some hands on experience before being able to give
> > any valuable feedback. I'll definitely keep an eye on it
> > though. Splitting is something we don't plan for our prototype, but
> > will be something we'll want in the future if we proceed. We'd be
> > happen to contribute our findings to helix if there isn't already a
> > solution by then.
> >
> > -Ivan
> >
Re: Using partitions in Helix
Posted by kishore g <g....@gmail.com>.
This was asked earlier on the mailing list.
http://markmail.org/thread/k4asnfgu7pcmrose#query:+page:1+mid:dkjaxewbrrf64wq7+state:results
Here is the paste bin http://pastebin.com/e2LXtq46
We also added idealstatebuilders that might make it easier.
https://git-wip-us.apache.org/repos/asf?p=incubator-helix.git;a=tree;f=helix-core/src/main/java/org/apache/helix/model/builder;hb=6119a4467b0fadbab09267a68d15aa470410a182
Adding it to example/recipe
thanks,
Kishore G
On Tue, Apr 16, 2013 at 10:09 AM, Ivan Kelly <iv...@apache.org> wrote:
> Hey Kishore,
>
> Thanks for your response.
>
> > 2. Set your own idealstate
> >
> > In Helix partition is just a logical name. So you can always generate an
> > idealstate with partition names as follows.
> >
> > {resource_k1start, resource_k2start, resource_k3start} or even
> > {resource_k1start-k1end}.. to make the mapping explicit.
> Is there an example of this in the code/tests? I couldn't see anything
> in the recipes.
>
> > Currently Helix does not understand the concept of a partition being
> split,
> > you will have to program it yourself by changing the idealstate during
> > splits. We have a jira for this and we are working on the initial design.
> > Your inputs will be valuable. I will try to add more details to the JIRA.
> > This is an important feature and we plan to support it as a first class
> > citizen in Helix so that one can benefit from other features like
> > throttling ( for example, one could say that dont do more than X splits
> > across cluster, node etc)
> >
> > https://issues.apache.org/jira/browse/HELIX-69.
> I'll need to get some hands on experience before being able to give
> any valuable feedback. I'll definitely keep an eye on it
> though. Splitting is something we don't plan for our prototype, but
> will be something we'll want in the future if we proceed. We'd be
> happen to contribute our findings to helix if there isn't already a
> solution by then.
>
> -Ivan
>
Re: Using partitions in Helix
Posted by Ivan Kelly <iv...@apache.org>.
Hey Kishore,
Thanks for your response.
> 2. Set your own idealstate
>
> In Helix partition is just a logical name. So you can always generate an
> idealstate with partition names as follows.
>
> {resource_k1start, resource_k2start, resource_k3start} or even
> {resource_k1start-k1end}.. to make the mapping explicit.
Is there an example of this in the code/tests? I couldn't see anything
in the recipes.
> Currently Helix does not understand the concept of a partition being split,
> you will have to program it yourself by changing the idealstate during
> splits. We have a jira for this and we are working on the initial design.
> Your inputs will be valuable. I will try to add more details to the JIRA.
> This is an important feature and we plan to support it as a first class
> citizen in Helix so that one can benefit from other features like
> throttling ( for example, one could say that dont do more than X splits
> across cluster, node etc)
>
> https://issues.apache.org/jira/browse/HELIX-69.
I'll need to get some hands on experience before being able to give
any valuable feedback. I'll definitely keep an eye on it
though. Splitting is something we don't plan for our prototype, but
will be something we'll want in the future if we proceed. We'd be
happen to contribute our findings to helix if there isn't already a
solution by then.
-Ivan
Re: Using partitions in Helix
Posted by kishore g <g....@gmail.com>.
Hi Ivan,
Thanks for the question. You can chose one of the two.
1. Use HelixAdmin and create your own mapping of partition to key range
Basically you will have 5 partitions { resource_0, resource_1, resource_2,
resource_3, resource_4}, then you can have config for each partition which
tells the key range it is associated with.
{resource_0=k1start} {resource_1=k2start}.......
This works fine initially but when you further split your key ranges the
partition numbering becomes confusing and difficult to map.
2. Set your own idealstate
In Helix partition is just a logical name. So you can always generate an
idealstate with partition names as follows.
{resource_k1start, resource_k2start, resource_k3start} or even
{resource_k1start-k1end}.. to make the mapping explicit.
This is easier to understand and reason about. External view then becomes
the routing table.
Currently Helix does not understand the concept of a partition being split,
you will have to program it yourself by changing the idealstate during
splits. We have a jira for this and we are working on the initial design.
Your inputs will be valuable. I will try to add more details to the JIRA.
This is an important feature and we plan to support it as a first class
citizen in Helix so that one can benefit from other features like
throttling ( for example, one could say that dont do more than X splits
across cluster, node etc)
https://issues.apache.org/jira/browse/HELIX-69.
thanks,
Kishore G
On Tue, Apr 16, 2013 at 6:36 AM, Ivan Kelly <iv...@apache.org> wrote:
> Hi,
>
> I'm sketching out the design of a prototype key value store which I'd
> like to use helix in. However, I'm unclear of how to use partitions in
> helix.
>
> So, the number of partitions for a particular resource is defined when
> I call #addResource from helix admin. Lets say I want to split my key
> space into 5 partitions. How do I define what the boundaries for these
> partitions are? Do I have to create my own mapping of key range to
> partition? Or do I have to provide my own implementation of
> IdealState?
>
> Thanks
>
> -Ivan
>