You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@helix.apache.org by kishore g <g....@gmail.com> on 2013/09/26 19:53:44 UTC
New rebalancers: deltarebalancer & rebalancer
Hi,
I am thinking of adding a delta rebalancer that simple compares the delta
between idealstate and currentstate and generate transitions for the delta.
The main difference from existing implementation is that this does not need
a state machine or any constraints on the state.
Where can this be useful
One scenario where this can be useful is managing the config versions. For
example we can say
Node1:1.3
Node2:1.3
Each version can correspond to a set a config properties. If a new config
is uploaded with new version, we send the transition for each node to
refresh the config. The benefit of this is clients will know what is the
config version the participant is running. And leader can ensure that all
participants are upgraded to latest config.
We can take this a step further and allow the idealstate to be more
expressive
for example the state can be composed of multiple attributes
N1: {a: aval, b:bval, c:cval}
N1: {a: aval, b:bval, c:cval}
And we can change each value independently.
N1:{a:avalnew, b:bval, c:cval}
we could then invoke the appropriate call back on the participant that a
changed and once the callback is processed the current state changes.
This kind of becomes a general way to manage configuration and keep track
of what config the node is using. Note this is not supposed to say every
single config property but it could be used to represent high level config
properties.
Thoughts/feedback? At LinkedIn, i think this will help us manage schema
versions.
Thanks,
Kishore G
Re: New rebalancers: deltarebalancer & rebalancer
Posted by kishore g <g....@gmail.com>.
I think the change that we need will be in find next state machine. We need
an implementation that will take current state and target state and returns
the target state.
On Thu, Sep 26, 2013 at 4:54 PM, Kanak Biscuitwala <ka...@hotmail.com>wrote:
> Well actually I guess I can answer my own question: the current rebalancer
> interface seems to be general enough to support adding a new delta
> implementation that would fit in the current pipeline.
>
> ----------------------------------------
> > From: kanak.b@hotmail.com
> > To: user@helix.incubator.apache.org; dev@helix.incubator.apache.org
> > Subject: RE: New rebalancers: deltarebalancer & rebalancer
> > Date: Thu, 26 Sep 2013 16:51:42 -0700
> >
> > Seems reasonable. I guess this doesn't fit into a traditional state
> model because the rebalancer can generate a new "state" each time. Do you
> envision this being a generalization of the current rebalancer, or as part
> of a new pipeline?
> >
> > Kanak
> > ________________________________
> >> Date: Thu, 26 Sep 2013 10:53:44 -0700
> >> Subject: New rebalancers: deltarebalancer & rebalancer
> >> From: g.kishore@gmail.com
> >> To: dev@helix.incubator.apache.org; user@helix.incubator.apache.org
> >>
> >> Hi,
> >>
> >> I am thinking of adding a delta rebalancer that simple compares the
> >> delta between idealstate and currentstate and generate transitions for
> >> the delta. The main difference from existing implementation is that
> >> this does not need a state machine or any constraints on the state.
> >>
> >> Where can this be useful
> >> One scenario where this can be useful is managing the config versions.
> >> For example we can say
> >>
> >> Node1:1.3
> >> Node2:1.3
> >>
> >> Each version can correspond to a set a config properties. If a new
> >> config is uploaded with new version, we send the transition for each
> >> node to refresh the config. The benefit of this is clients will know
> >> what is the config version the participant is running. And leader can
> >> ensure that all participants are upgraded to latest config.
> >>
> >> We can take this a step further and allow the idealstate to be more
> >> expressive
> >>
> >> for example the state can be composed of multiple attributes
> >>
> >> N1: {a: aval, b:bval, c:cval}
> >> N1: {a: aval, b:bval, c:cval}
> >>
> >>
> >> And we can change each value independently.
> >>
> >> N1:{a:avalnew, b:bval, c:cval}
> >>
> >> we could then invoke the appropriate call back on the participant that
> >> a changed and once the callback is processed the current state changes.
> >>
> >> This kind of becomes a general way to manage configuration and keep
> >> track of what config the node is using. Note this is not supposed to
> >> say every single config property but it could be used to represent high
> >> level config properties.
> >>
> >>
> >> Thoughts/feedback? At LinkedIn, i think this will help us manage schema
> >> versions.
> >>
> >> Thanks,
> >> Kishore G
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
>
Re: New rebalancers: deltarebalancer & rebalancer
Posted by kishore g <g....@gmail.com>.
I think the change that we need will be in find next state machine. We need
an implementation that will take current state and target state and returns
the target state.
On Thu, Sep 26, 2013 at 4:54 PM, Kanak Biscuitwala <ka...@hotmail.com>wrote:
> Well actually I guess I can answer my own question: the current rebalancer
> interface seems to be general enough to support adding a new delta
> implementation that would fit in the current pipeline.
>
> ----------------------------------------
> > From: kanak.b@hotmail.com
> > To: user@helix.incubator.apache.org; dev@helix.incubator.apache.org
> > Subject: RE: New rebalancers: deltarebalancer & rebalancer
> > Date: Thu, 26 Sep 2013 16:51:42 -0700
> >
> > Seems reasonable. I guess this doesn't fit into a traditional state
> model because the rebalancer can generate a new "state" each time. Do you
> envision this being a generalization of the current rebalancer, or as part
> of a new pipeline?
> >
> > Kanak
> > ________________________________
> >> Date: Thu, 26 Sep 2013 10:53:44 -0700
> >> Subject: New rebalancers: deltarebalancer & rebalancer
> >> From: g.kishore@gmail.com
> >> To: dev@helix.incubator.apache.org; user@helix.incubator.apache.org
> >>
> >> Hi,
> >>
> >> I am thinking of adding a delta rebalancer that simple compares the
> >> delta between idealstate and currentstate and generate transitions for
> >> the delta. The main difference from existing implementation is that
> >> this does not need a state machine or any constraints on the state.
> >>
> >> Where can this be useful
> >> One scenario where this can be useful is managing the config versions.
> >> For example we can say
> >>
> >> Node1:1.3
> >> Node2:1.3
> >>
> >> Each version can correspond to a set a config properties. If a new
> >> config is uploaded with new version, we send the transition for each
> >> node to refresh the config. The benefit of this is clients will know
> >> what is the config version the participant is running. And leader can
> >> ensure that all participants are upgraded to latest config.
> >>
> >> We can take this a step further and allow the idealstate to be more
> >> expressive
> >>
> >> for example the state can be composed of multiple attributes
> >>
> >> N1: {a: aval, b:bval, c:cval}
> >> N1: {a: aval, b:bval, c:cval}
> >>
> >>
> >> And we can change each value independently.
> >>
> >> N1:{a:avalnew, b:bval, c:cval}
> >>
> >> we could then invoke the appropriate call back on the participant that
> >> a changed and once the callback is processed the current state changes.
> >>
> >> This kind of becomes a general way to manage configuration and keep
> >> track of what config the node is using. Note this is not supposed to
> >> say every single config property but it could be used to represent high
> >> level config properties.
> >>
> >>
> >> Thoughts/feedback? At LinkedIn, i think this will help us manage schema
> >> versions.
> >>
> >> Thanks,
> >> Kishore G
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
>
RE: New rebalancers: deltarebalancer & rebalancer
Posted by Kanak Biscuitwala <ka...@hotmail.com>.
Well actually I guess I can answer my own question: the current rebalancer interface seems to be general enough to support adding a new delta implementation that would fit in the current pipeline.
----------------------------------------
> From: kanak.b@hotmail.com
> To: user@helix.incubator.apache.org; dev@helix.incubator.apache.org
> Subject: RE: New rebalancers: deltarebalancer & rebalancer
> Date: Thu, 26 Sep 2013 16:51:42 -0700
>
> Seems reasonable. I guess this doesn't fit into a traditional state model because the rebalancer can generate a new "state" each time. Do you envision this being a generalization of the current rebalancer, or as part of a new pipeline?
>
> Kanak
> ________________________________
>> Date: Thu, 26 Sep 2013 10:53:44 -0700
>> Subject: New rebalancers: deltarebalancer & rebalancer
>> From: g.kishore@gmail.com
>> To: dev@helix.incubator.apache.org; user@helix.incubator.apache.org
>>
>> Hi,
>>
>> I am thinking of adding a delta rebalancer that simple compares the
>> delta between idealstate and currentstate and generate transitions for
>> the delta. The main difference from existing implementation is that
>> this does not need a state machine or any constraints on the state.
>>
>> Where can this be useful
>> One scenario where this can be useful is managing the config versions.
>> For example we can say
>>
>> Node1:1.3
>> Node2:1.3
>>
>> Each version can correspond to a set a config properties. If a new
>> config is uploaded with new version, we send the transition for each
>> node to refresh the config. The benefit of this is clients will know
>> what is the config version the participant is running. And leader can
>> ensure that all participants are upgraded to latest config.
>>
>> We can take this a step further and allow the idealstate to be more
>> expressive
>>
>> for example the state can be composed of multiple attributes
>>
>> N1: {a: aval, b:bval, c:cval}
>> N1: {a: aval, b:bval, c:cval}
>>
>>
>> And we can change each value independently.
>>
>> N1:{a:avalnew, b:bval, c:cval}
>>
>> we could then invoke the appropriate call back on the participant that
>> a changed and once the callback is processed the current state changes.
>>
>> This kind of becomes a general way to manage configuration and keep
>> track of what config the node is using. Note this is not supposed to
>> say every single config property but it could be used to represent high
>> level config properties.
>>
>>
>> Thoughts/feedback? At LinkedIn, i think this will help us manage schema
>> versions.
>>
>> Thanks,
>> Kishore G
>>
>>
>>
>>
>>
>>
>>
>>
>>
RE: New rebalancers: deltarebalancer & rebalancer
Posted by Kanak Biscuitwala <ka...@hotmail.com>.
Well actually I guess I can answer my own question: the current rebalancer interface seems to be general enough to support adding a new delta implementation that would fit in the current pipeline.
----------------------------------------
> From: kanak.b@hotmail.com
> To: user@helix.incubator.apache.org; dev@helix.incubator.apache.org
> Subject: RE: New rebalancers: deltarebalancer & rebalancer
> Date: Thu, 26 Sep 2013 16:51:42 -0700
>
> Seems reasonable. I guess this doesn't fit into a traditional state model because the rebalancer can generate a new "state" each time. Do you envision this being a generalization of the current rebalancer, or as part of a new pipeline?
>
> Kanak
> ________________________________
>> Date: Thu, 26 Sep 2013 10:53:44 -0700
>> Subject: New rebalancers: deltarebalancer & rebalancer
>> From: g.kishore@gmail.com
>> To: dev@helix.incubator.apache.org; user@helix.incubator.apache.org
>>
>> Hi,
>>
>> I am thinking of adding a delta rebalancer that simple compares the
>> delta between idealstate and currentstate and generate transitions for
>> the delta. The main difference from existing implementation is that
>> this does not need a state machine or any constraints on the state.
>>
>> Where can this be useful
>> One scenario where this can be useful is managing the config versions.
>> For example we can say
>>
>> Node1:1.3
>> Node2:1.3
>>
>> Each version can correspond to a set a config properties. If a new
>> config is uploaded with new version, we send the transition for each
>> node to refresh the config. The benefit of this is clients will know
>> what is the config version the participant is running. And leader can
>> ensure that all participants are upgraded to latest config.
>>
>> We can take this a step further and allow the idealstate to be more
>> expressive
>>
>> for example the state can be composed of multiple attributes
>>
>> N1: {a: aval, b:bval, c:cval}
>> N1: {a: aval, b:bval, c:cval}
>>
>>
>> And we can change each value independently.
>>
>> N1:{a:avalnew, b:bval, c:cval}
>>
>> we could then invoke the appropriate call back on the participant that
>> a changed and once the callback is processed the current state changes.
>>
>> This kind of becomes a general way to manage configuration and keep
>> track of what config the node is using. Note this is not supposed to
>> say every single config property but it could be used to represent high
>> level config properties.
>>
>>
>> Thoughts/feedback? At LinkedIn, i think this will help us manage schema
>> versions.
>>
>> Thanks,
>> Kishore G
>>
>>
>>
>>
>>
>>
>>
>>
>>
RE: New rebalancers: deltarebalancer & rebalancer
Posted by Kanak Biscuitwala <ka...@hotmail.com>.
Seems reasonable. I guess this doesn't fit into a traditional state model because the rebalancer can generate a new "state" each time. Do you envision this being a generalization of the current rebalancer, or as part of a new pipeline?
Kanak
________________________________
> Date: Thu, 26 Sep 2013 10:53:44 -0700
> Subject: New rebalancers: deltarebalancer & rebalancer
> From: g.kishore@gmail.com
> To: dev@helix.incubator.apache.org; user@helix.incubator.apache.org
>
> Hi,
>
> I am thinking of adding a delta rebalancer that simple compares the
> delta between idealstate and currentstate and generate transitions for
> the delta. The main difference from existing implementation is that
> this does not need a state machine or any constraints on the state.
>
> Where can this be useful
> One scenario where this can be useful is managing the config versions.
> For example we can say
>
> Node1:1.3
> Node2:1.3
>
> Each version can correspond to a set a config properties. If a new
> config is uploaded with new version, we send the transition for each
> node to refresh the config. The benefit of this is clients will know
> what is the config version the participant is running. And leader can
> ensure that all participants are upgraded to latest config.
>
> We can take this a step further and allow the idealstate to be more
> expressive
>
> for example the state can be composed of multiple attributes
>
> N1: {a: aval, b:bval, c:cval}
> N1: {a: aval, b:bval, c:cval}
>
>
> And we can change each value independently.
>
> N1:{a:avalnew, b:bval, c:cval}
>
> we could then invoke the appropriate call back on the participant that
> a changed and once the callback is processed the current state changes.
>
> This kind of becomes a general way to manage configuration and keep
> track of what config the node is using. Note this is not supposed to
> say every single config property but it could be used to represent high
> level config properties.
>
>
> Thoughts/feedback? At LinkedIn, i think this will help us manage schema
> versions.
>
> Thanks,
> Kishore G
>
>
>
>
>
>
>
>
>
RE: New rebalancers: deltarebalancer & rebalancer
Posted by Kanak Biscuitwala <ka...@hotmail.com>.
Seems reasonable. I guess this doesn't fit into a traditional state model because the rebalancer can generate a new "state" each time. Do you envision this being a generalization of the current rebalancer, or as part of a new pipeline?
Kanak
________________________________
> Date: Thu, 26 Sep 2013 10:53:44 -0700
> Subject: New rebalancers: deltarebalancer & rebalancer
> From: g.kishore@gmail.com
> To: dev@helix.incubator.apache.org; user@helix.incubator.apache.org
>
> Hi,
>
> I am thinking of adding a delta rebalancer that simple compares the
> delta between idealstate and currentstate and generate transitions for
> the delta. The main difference from existing implementation is that
> this does not need a state machine or any constraints on the state.
>
> Where can this be useful
> One scenario where this can be useful is managing the config versions.
> For example we can say
>
> Node1:1.3
> Node2:1.3
>
> Each version can correspond to a set a config properties. If a new
> config is uploaded with new version, we send the transition for each
> node to refresh the config. The benefit of this is clients will know
> what is the config version the participant is running. And leader can
> ensure that all participants are upgraded to latest config.
>
> We can take this a step further and allow the idealstate to be more
> expressive
>
> for example the state can be composed of multiple attributes
>
> N1: {a: aval, b:bval, c:cval}
> N1: {a: aval, b:bval, c:cval}
>
>
> And we can change each value independently.
>
> N1:{a:avalnew, b:bval, c:cval}
>
> we could then invoke the appropriate call back on the participant that
> a changed and once the callback is processed the current state changes.
>
> This kind of becomes a general way to manage configuration and keep
> track of what config the node is using. Note this is not supposed to
> say every single config property but it could be used to represent high
> level config properties.
>
>
> Thoughts/feedback? At LinkedIn, i think this will help us manage schema
> versions.
>
> Thanks,
> Kishore G
>
>
>
>
>
>
>
>
>