state model with variable number of partitions

[Massimo Calderoni <mc...@vmware.com>]

Hi, 
was not able to find an answer to this question: 
with Helix, is it possible to have a state model where partitions are variable in number? 

If it is, are there any examples? 


Read the previous post talking about AUTO and examples like BaseStageTest, but they don't seem to address a system with varying partitions. 
In playing with code, seems like partitions must be established upfront: once StateModelDef is defined partitions are frozen. 


What i am trying to get to is a DDS that adds a partition as a node gets added, much like in system using consistent hashing: the node that gets added also adds a new master partition, that is, every node has one master partition. 


Max 

Re: state model with variable number of partitions

[kishore g <g....@gmail.com>]

Hi Max,


First of all note that state model and number of partitions are orthogonal.
The state model def is defined at a partition granularity. It is common to
all partitions of a given resource irrespective of number of partitions.

The idealstate of the resource defines the number of partitions. This can
be changed dynamically by changing the idealstate. You can do this by using
CustomCodeInvoker and changing the idealstate(change the partition number)
when a new node is added.

However, it is quite easy to add this support to AUTO_REBALANCE mode, where
we can say the number of partitions is 'N' where N denotes the number of
nodes. This will avoid the need to change idealstate every time a new node
is added.

Filed https://issues.apache.org/jira/browse/HELIX-52.

Let us know if this is something that will work for your use case and
thanks for bringing up this up.

Thanks,
Kishore G



On Wed, Feb 20, 2013 at 9:03 PM, Massimo Calderoni <mc...@vmware.com>wrote:

> Hi,
> was not able to find an answer to this question:
> with Helix, is it possible to have a state model where partitions are
> variable in number?
> If it is, are there any examples?
>
> Read the previous post talking about AUTO and examples like BaseStageTest,
> but they don't seem to address a system with varying partitions.
> In playing with code, seems like partitions must be established upfront:
> once StateModelDef is defined partitions are frozen.
>
> What i am trying to get to is a DDS that adds a partition as a node gets
> added, much like in system using consistent hashing: the node that gets
> added also adds a new master partition, that is, every node has one master
> partition.
>
> Max
>
>
>