You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@helix.apache.org by Changgeng Li <li...@gmail.com> on 2015/06/25 00:42:55 UTC
Customized rebalancer
Hello,
We have a use case that when adding or removing a new node, we hope to
minimize the shuffle of partitions between nodes. I'm trying to implement a
customized rebalancer calculating the new resource assignment based on the
previous resource assignment. When a new node is added, just move some
partitions from existing node to the new node, and when a node is down,
move the partitions on this node to the other nodes. Partitions would not
move between two nodes if the both status are not changed.
The documentation says:
In rebalance(), ... the third is the output of the previous invocation of
this method (if supported) ...
During my testing I found the previous resource assignment passed in is
always null. my question is what "if supported" means here? Does the
rebalancer need to support something?
Any other suggestions are also welcome.
Thanks,
Changgeng
Re: Customized rebalancer
Posted by Changgeng Li <li...@gmail.com>.
It seems we don't have access to the event that triggers rebalance in
rebalancer.
Maybe we can add another method in HelixRebalancer similar to something:
shouldRebalance(ClusterEvent: event): boolean, and if this returns false
just don't rebalance.
On Thu, Jun 25, 2015 at 11:07 AM, kishore g <g....@gmail.com> wrote:
> Hi Changgeng,
>
> Yes, prevAssignment will be preserved even if controller moves to another
> node. Agree, we can either have flag in resourceAssignment/marker
> interface/annotation on the computeAssignment method.
>
> Yeah using current state of each partition should work. One thing to note
> that is pending state is not guaranteed to reach (the transition might
> fail). In fact, all the in built rebalancers in Helix work off of current
> state.
>
> Yes, rebalancer is invoked on every change in the cluster. You can check
> for the event type and return the same assignment for some event types.
> Just basing on LiveNodes may not work if the participants fails to handle
> the transition.
>
> thanks,
> Kishore G
>
> On Thu, Jun 25, 2015 at 9:55 AM, Changgeng Li <li...@gmail.com>
> wrote:
>
>> I'm also trying to work around this for now by using currentState to
>> figure out the previous resource assignment, using pending state override
>> the current state of each partition. Do you think there is any potential
>> issue with this approach before prevResourceAssignment is persisted?
>>
>> I also have another question about when the rebalancer is invoked. My
>> test shows it's called a lot of times when add just one new node. Looks
>> like it's also called when the current state is changed. If so, we probably
>> need implement a shortcut that if livenodes are constraints are not
>> changed, just return the previous resource assignment.
>>
>>
>> On Thu, Jun 25, 2015 at 9:50 AM, Changgeng Li <li...@gmail.com>
>> wrote:
>>
>>> Actually I was going to ask whether prevAssignment is preserved if the
>>> controller moves to another node. A marker either for the rebalancer or for
>>> the returned resource assignment would work. It would be more flexible if
>>> there's a flag in the returned resource assignment.
>>>
>>>
>>>
>>> On Wed, Jun 24, 2015 at 11:44 PM, kishore g <g....@gmail.com> wrote:
>>>
>>>> You are right. I think this is a bug. We need to preserve the previous
>>>> assignment and pass it on. Having said that we should not do this for all
>>>> types of rebalancers. Doing so will introduce additional latency(additional
>>>> writes to Zookeeper) and will impact fail over time. Most rebalancers also
>>>> tend to be idempotent, i.e they compute the same output given the same set
>>>> of inputs. This property in general is not error prone and makes debugging
>>>> easier.
>>>>
>>>> One idea is to have a marker interface Persistable or something and
>>>> only save it when the implementation implements this interface. The fix is
>>>> easy but love to hear your feedback.
>>>>
>>>> Also, the AutoRebalancer that comes with Helix, already provides the
>>>> features that you are planning to implement. May be you can clone it and
>>>> modify it to suite your needs.
>>>>
>>>>
>>>> https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/rebalancer/FullAutoRebalancer.java
>>>>
>>>> https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/strategy/AutoRebalanceStrategy.java
>>>>
>>>>
>>>> thanks,
>>>> Kishore G
>>>>
>>>> On Wed, Jun 24, 2015 at 11:13 PM, Changgeng Li <li...@gmail.com>
>>>> wrote:
>>>>
>>>>> yes, it's always null.
>>>>>
>>>>> actually I checked the code that parameter was never set.
>>>>>
>>>>> https://github.com/apache/helix/blob/acd902e2433f65e9864ccf49fcd1a04c36b1f206/helix-core/src/main/java/org/apache/helix/controller/stages/BestPossibleStateCalcStage.java#L222
>>>>>
>>>>> On Wed, Jun 24, 2015 at 10:41 PM, kishore g <g....@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I see that you haven't subscribed to mailing list yet. Please do that
>>>>>> my sending an email to user-subscribe@helix.apache.org.
>>>>>>
>>>>>> I am also available on #apachehelix irc if you have additional
>>>>>> questions. http://helix.apache.org/IRC.html
>>>>>>
>>>>>>
>>>>>> ---------- Forwarded message ----------
>>>>>> From: kishore g <g....@gmail.com>
>>>>>> Date: Wed, Jun 24, 2015 at 10:37 PM
>>>>>> Subject: Re: Customized rebalancer
>>>>>> To: "user@helix.apache.org" <us...@helix.apache.org>
>>>>>>
>>>>>>
>>>>>> Hi Changgeng,
>>>>>>
>>>>>> I think the first invocation will always be null, subsequent
>>>>>> invocations should provide you the previous resource assignment. Are you
>>>>>> saying its always null?.
>>>>>>
>>>>>> thanks,
>>>>>> Kishore G
>>>>>>
>>>>>> On Wed, Jun 24, 2015 at 3:42 PM, Changgeng Li <li.changgeng@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> We have a use case that when adding or removing a new node, we hope
>>>>>>> to minimize the shuffle of partitions between nodes. I'm trying to
>>>>>>> implement a customized rebalancer calculating the new resource assignment
>>>>>>> based on the previous resource assignment. When a new node is added, just
>>>>>>> move some partitions from existing node to the new node, and when a node is
>>>>>>> down, move the partitions on this node to the other nodes. Partitions would
>>>>>>> not move between two nodes if the both status are not changed.
>>>>>>>
>>>>>>> The documentation says:
>>>>>>> In rebalance(), ... the third is the output of the previous
>>>>>>> invocation of this method (if supported) ...
>>>>>>>
>>>>>>> During my testing I found the previous resource assignment passed in
>>>>>>> is always null. my question is what "if supported" means here? Does the
>>>>>>> rebalancer need to support something?
>>>>>>>
>>>>>>>
>>>>>>> Any other suggestions are also welcome.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Changgeng
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Customized rebalancer
Posted by kishore g <g....@gmail.com>.
Hi Changgeng,
Yes, prevAssignment will be preserved even if controller moves to another
node. Agree, we can either have flag in resourceAssignment/marker
interface/annotation on the computeAssignment method.
Yeah using current state of each partition should work. One thing to note
that is pending state is not guaranteed to reach (the transition might
fail). In fact, all the in built rebalancers in Helix work off of current
state.
Yes, rebalancer is invoked on every change in the cluster. You can check
for the event type and return the same assignment for some event types.
Just basing on LiveNodes may not work if the participants fails to handle
the transition.
thanks,
Kishore G
On Thu, Jun 25, 2015 at 9:55 AM, Changgeng Li <li...@gmail.com>
wrote:
> I'm also trying to work around this for now by using currentState to
> figure out the previous resource assignment, using pending state override
> the current state of each partition. Do you think there is any potential
> issue with this approach before prevResourceAssignment is persisted?
>
> I also have another question about when the rebalancer is invoked. My test
> shows it's called a lot of times when add just one new node. Looks like
> it's also called when the current state is changed. If so, we probably need
> implement a shortcut that if livenodes are constraints are not changed,
> just return the previous resource assignment.
>
>
> On Thu, Jun 25, 2015 at 9:50 AM, Changgeng Li <li...@gmail.com>
> wrote:
>
>> Actually I was going to ask whether prevAssignment is preserved if the
>> controller moves to another node. A marker either for the rebalancer or for
>> the returned resource assignment would work. It would be more flexible if
>> there's a flag in the returned resource assignment.
>>
>>
>>
>> On Wed, Jun 24, 2015 at 11:44 PM, kishore g <g....@gmail.com> wrote:
>>
>>> You are right. I think this is a bug. We need to preserve the previous
>>> assignment and pass it on. Having said that we should not do this for all
>>> types of rebalancers. Doing so will introduce additional latency(additional
>>> writes to Zookeeper) and will impact fail over time. Most rebalancers also
>>> tend to be idempotent, i.e they compute the same output given the same set
>>> of inputs. This property in general is not error prone and makes debugging
>>> easier.
>>>
>>> One idea is to have a marker interface Persistable or something and only
>>> save it when the implementation implements this interface. The fix is easy
>>> but love to hear your feedback.
>>>
>>> Also, the AutoRebalancer that comes with Helix, already provides the
>>> features that you are planning to implement. May be you can clone it and
>>> modify it to suite your needs.
>>>
>>>
>>> https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/rebalancer/FullAutoRebalancer.java
>>>
>>> https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/strategy/AutoRebalanceStrategy.java
>>>
>>>
>>> thanks,
>>> Kishore G
>>>
>>> On Wed, Jun 24, 2015 at 11:13 PM, Changgeng Li <li...@gmail.com>
>>> wrote:
>>>
>>>> yes, it's always null.
>>>>
>>>> actually I checked the code that parameter was never set.
>>>>
>>>> https://github.com/apache/helix/blob/acd902e2433f65e9864ccf49fcd1a04c36b1f206/helix-core/src/main/java/org/apache/helix/controller/stages/BestPossibleStateCalcStage.java#L222
>>>>
>>>> On Wed, Jun 24, 2015 at 10:41 PM, kishore g <g....@gmail.com>
>>>> wrote:
>>>>
>>>>> I see that you haven't subscribed to mailing list yet. Please do that
>>>>> my sending an email to user-subscribe@helix.apache.org.
>>>>>
>>>>> I am also available on #apachehelix irc if you have additional
>>>>> questions. http://helix.apache.org/IRC.html
>>>>>
>>>>>
>>>>> ---------- Forwarded message ----------
>>>>> From: kishore g <g....@gmail.com>
>>>>> Date: Wed, Jun 24, 2015 at 10:37 PM
>>>>> Subject: Re: Customized rebalancer
>>>>> To: "user@helix.apache.org" <us...@helix.apache.org>
>>>>>
>>>>>
>>>>> Hi Changgeng,
>>>>>
>>>>> I think the first invocation will always be null, subsequent
>>>>> invocations should provide you the previous resource assignment. Are you
>>>>> saying its always null?.
>>>>>
>>>>> thanks,
>>>>> Kishore G
>>>>>
>>>>> On Wed, Jun 24, 2015 at 3:42 PM, Changgeng Li <li...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> We have a use case that when adding or removing a new node, we hope
>>>>>> to minimize the shuffle of partitions between nodes. I'm trying to
>>>>>> implement a customized rebalancer calculating the new resource assignment
>>>>>> based on the previous resource assignment. When a new node is added, just
>>>>>> move some partitions from existing node to the new node, and when a node is
>>>>>> down, move the partitions on this node to the other nodes. Partitions would
>>>>>> not move between two nodes if the both status are not changed.
>>>>>>
>>>>>> The documentation says:
>>>>>> In rebalance(), ... the third is the output of the previous
>>>>>> invocation of this method (if supported) ...
>>>>>>
>>>>>> During my testing I found the previous resource assignment passed in
>>>>>> is always null. my question is what "if supported" means here? Does the
>>>>>> rebalancer need to support something?
>>>>>>
>>>>>>
>>>>>> Any other suggestions are also welcome.
>>>>>>
>>>>>> Thanks,
>>>>>> Changgeng
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Customized rebalancer
Posted by Changgeng Li <li...@gmail.com>.
I'm also trying to work around this for now by using currentState to figure
out the previous resource assignment, using pending state override the
current state of each partition. Do you think there is any potential issue
with this approach before prevResourceAssignment is persisted?
I also have another question about when the rebalancer is invoked. My test
shows it's called a lot of times when add just one new node. Looks like
it's also called when the current state is changed. If so, we probably need
implement a shortcut that if livenodes are constraints are not changed,
just return the previous resource assignment.
On Thu, Jun 25, 2015 at 9:50 AM, Changgeng Li <li...@gmail.com>
wrote:
> Actually I was going to ask whether prevAssignment is preserved if the
> controller moves to another node. A marker either for the rebalancer or for
> the returned resource assignment would work. It would be more flexible if
> there's a flag in the returned resource assignment.
>
>
>
> On Wed, Jun 24, 2015 at 11:44 PM, kishore g <g....@gmail.com> wrote:
>
>> You are right. I think this is a bug. We need to preserve the previous
>> assignment and pass it on. Having said that we should not do this for all
>> types of rebalancers. Doing so will introduce additional latency(additional
>> writes to Zookeeper) and will impact fail over time. Most rebalancers also
>> tend to be idempotent, i.e they compute the same output given the same set
>> of inputs. This property in general is not error prone and makes debugging
>> easier.
>>
>> One idea is to have a marker interface Persistable or something and only
>> save it when the implementation implements this interface. The fix is easy
>> but love to hear your feedback.
>>
>> Also, the AutoRebalancer that comes with Helix, already provides the
>> features that you are planning to implement. May be you can clone it and
>> modify it to suite your needs.
>>
>>
>> https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/rebalancer/FullAutoRebalancer.java
>>
>> https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/strategy/AutoRebalanceStrategy.java
>>
>>
>> thanks,
>> Kishore G
>>
>> On Wed, Jun 24, 2015 at 11:13 PM, Changgeng Li <li...@gmail.com>
>> wrote:
>>
>>> yes, it's always null.
>>>
>>> actually I checked the code that parameter was never set.
>>>
>>> https://github.com/apache/helix/blob/acd902e2433f65e9864ccf49fcd1a04c36b1f206/helix-core/src/main/java/org/apache/helix/controller/stages/BestPossibleStateCalcStage.java#L222
>>>
>>> On Wed, Jun 24, 2015 at 10:41 PM, kishore g <g....@gmail.com> wrote:
>>>
>>>> I see that you haven't subscribed to mailing list yet. Please do that
>>>> my sending an email to user-subscribe@helix.apache.org.
>>>>
>>>> I am also available on #apachehelix irc if you have additional
>>>> questions. http://helix.apache.org/IRC.html
>>>>
>>>>
>>>> ---------- Forwarded message ----------
>>>> From: kishore g <g....@gmail.com>
>>>> Date: Wed, Jun 24, 2015 at 10:37 PM
>>>> Subject: Re: Customized rebalancer
>>>> To: "user@helix.apache.org" <us...@helix.apache.org>
>>>>
>>>>
>>>> Hi Changgeng,
>>>>
>>>> I think the first invocation will always be null, subsequent
>>>> invocations should provide you the previous resource assignment. Are you
>>>> saying its always null?.
>>>>
>>>> thanks,
>>>> Kishore G
>>>>
>>>> On Wed, Jun 24, 2015 at 3:42 PM, Changgeng Li <li...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> We have a use case that when adding or removing a new node, we hope to
>>>>> minimize the shuffle of partitions between nodes. I'm trying to implement a
>>>>> customized rebalancer calculating the new resource assignment based on the
>>>>> previous resource assignment. When a new node is added, just move some
>>>>> partitions from existing node to the new node, and when a node is down,
>>>>> move the partitions on this node to the other nodes. Partitions would not
>>>>> move between two nodes if the both status are not changed.
>>>>>
>>>>> The documentation says:
>>>>> In rebalance(), ... the third is the output of the previous
>>>>> invocation of this method (if supported) ...
>>>>>
>>>>> During my testing I found the previous resource assignment passed in
>>>>> is always null. my question is what "if supported" means here? Does the
>>>>> rebalancer need to support something?
>>>>>
>>>>>
>>>>> Any other suggestions are also welcome.
>>>>>
>>>>> Thanks,
>>>>> Changgeng
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: Customized rebalancer
Posted by Changgeng Li <li...@gmail.com>.
Actually I was going to ask whether prevAssignment is preserved if the
controller moves to another node. A marker either for the rebalancer or for
the returned resource assignment would work. It would be more flexible if
there's a flag in the returned resource assignment.
On Wed, Jun 24, 2015 at 11:44 PM, kishore g <g....@gmail.com> wrote:
> You are right. I think this is a bug. We need to preserve the previous
> assignment and pass it on. Having said that we should not do this for all
> types of rebalancers. Doing so will introduce additional latency(additional
> writes to Zookeeper) and will impact fail over time. Most rebalancers also
> tend to be idempotent, i.e they compute the same output given the same set
> of inputs. This property in general is not error prone and makes debugging
> easier.
>
> One idea is to have a marker interface Persistable or something and only
> save it when the implementation implements this interface. The fix is easy
> but love to hear your feedback.
>
> Also, the AutoRebalancer that comes with Helix, already provides the
> features that you are planning to implement. May be you can clone it and
> modify it to suite your needs.
>
>
> https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/rebalancer/FullAutoRebalancer.java
>
> https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/strategy/AutoRebalanceStrategy.java
>
>
> thanks,
> Kishore G
>
> On Wed, Jun 24, 2015 at 11:13 PM, Changgeng Li <li...@gmail.com>
> wrote:
>
>> yes, it's always null.
>>
>> actually I checked the code that parameter was never set.
>>
>> https://github.com/apache/helix/blob/acd902e2433f65e9864ccf49fcd1a04c36b1f206/helix-core/src/main/java/org/apache/helix/controller/stages/BestPossibleStateCalcStage.java#L222
>>
>> On Wed, Jun 24, 2015 at 10:41 PM, kishore g <g....@gmail.com> wrote:
>>
>>> I see that you haven't subscribed to mailing list yet. Please do that my
>>> sending an email to user-subscribe@helix.apache.org.
>>>
>>> I am also available on #apachehelix irc if you have additional
>>> questions. http://helix.apache.org/IRC.html
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: kishore g <g....@gmail.com>
>>> Date: Wed, Jun 24, 2015 at 10:37 PM
>>> Subject: Re: Customized rebalancer
>>> To: "user@helix.apache.org" <us...@helix.apache.org>
>>>
>>>
>>> Hi Changgeng,
>>>
>>> I think the first invocation will always be null, subsequent invocations
>>> should provide you the previous resource assignment. Are you saying its
>>> always null?.
>>>
>>> thanks,
>>> Kishore G
>>>
>>> On Wed, Jun 24, 2015 at 3:42 PM, Changgeng Li <li...@gmail.com>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> We have a use case that when adding or removing a new node, we hope to
>>>> minimize the shuffle of partitions between nodes. I'm trying to implement a
>>>> customized rebalancer calculating the new resource assignment based on the
>>>> previous resource assignment. When a new node is added, just move some
>>>> partitions from existing node to the new node, and when a node is down,
>>>> move the partitions on this node to the other nodes. Partitions would not
>>>> move between two nodes if the both status are not changed.
>>>>
>>>> The documentation says:
>>>> In rebalance(), ... the third is the output of the previous invocation
>>>> of this method (if supported) ...
>>>>
>>>> During my testing I found the previous resource assignment passed in is
>>>> always null. my question is what "if supported" means here? Does the
>>>> rebalancer need to support something?
>>>>
>>>>
>>>> Any other suggestions are also welcome.
>>>>
>>>> Thanks,
>>>> Changgeng
>>>>
>>>
>>>
>>>
>>
>
Re: Customized rebalancer
Posted by kishore g <g....@gmail.com>.
You are right. I think this is a bug. We need to preserve the previous
assignment and pass it on. Having said that we should not do this for all
types of rebalancers. Doing so will introduce additional latency(additional
writes to Zookeeper) and will impact fail over time. Most rebalancers also
tend to be idempotent, i.e they compute the same output given the same set
of inputs. This property in general is not error prone and makes debugging
easier.
One idea is to have a marker interface Persistable or something and only
save it when the implementation implements this interface. The fix is easy
but love to hear your feedback.
Also, the AutoRebalancer that comes with Helix, already provides the
features that you are planning to implement. May be you can clone it and
modify it to suite your needs.
https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/rebalancer/FullAutoRebalancer.java
https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/strategy/AutoRebalanceStrategy.java
thanks,
Kishore G
On Wed, Jun 24, 2015 at 11:13 PM, Changgeng Li <li...@gmail.com>
wrote:
> yes, it's always null.
>
> actually I checked the code that parameter was never set.
>
> https://github.com/apache/helix/blob/acd902e2433f65e9864ccf49fcd1a04c36b1f206/helix-core/src/main/java/org/apache/helix/controller/stages/BestPossibleStateCalcStage.java#L222
>
> On Wed, Jun 24, 2015 at 10:41 PM, kishore g <g....@gmail.com> wrote:
>
>> I see that you haven't subscribed to mailing list yet. Please do that my
>> sending an email to user-subscribe@helix.apache.org.
>>
>> I am also available on #apachehelix irc if you have additional questions.
>> http://helix.apache.org/IRC.html
>>
>>
>> ---------- Forwarded message ----------
>> From: kishore g <g....@gmail.com>
>> Date: Wed, Jun 24, 2015 at 10:37 PM
>> Subject: Re: Customized rebalancer
>> To: "user@helix.apache.org" <us...@helix.apache.org>
>>
>>
>> Hi Changgeng,
>>
>> I think the first invocation will always be null, subsequent invocations
>> should provide you the previous resource assignment. Are you saying its
>> always null?.
>>
>> thanks,
>> Kishore G
>>
>> On Wed, Jun 24, 2015 at 3:42 PM, Changgeng Li <li...@gmail.com>
>> wrote:
>>
>>> Hello,
>>>
>>> We have a use case that when adding or removing a new node, we hope to
>>> minimize the shuffle of partitions between nodes. I'm trying to implement a
>>> customized rebalancer calculating the new resource assignment based on the
>>> previous resource assignment. When a new node is added, just move some
>>> partitions from existing node to the new node, and when a node is down,
>>> move the partitions on this node to the other nodes. Partitions would not
>>> move between two nodes if the both status are not changed.
>>>
>>> The documentation says:
>>> In rebalance(), ... the third is the output of the previous invocation
>>> of this method (if supported) ...
>>>
>>> During my testing I found the previous resource assignment passed in is
>>> always null. my question is what "if supported" means here? Does the
>>> rebalancer need to support something?
>>>
>>>
>>> Any other suggestions are also welcome.
>>>
>>> Thanks,
>>> Changgeng
>>>
>>
>>
>>
>
Re: Customized rebalancer
Posted by Changgeng Li <li...@gmail.com>.
yes, it's always null.
actually I checked the code that parameter was never set.
https://github.com/apache/helix/blob/acd902e2433f65e9864ccf49fcd1a04c36b1f206/helix-core/src/main/java/org/apache/helix/controller/stages/BestPossibleStateCalcStage.java#L222
On Wed, Jun 24, 2015 at 10:41 PM, kishore g <g....@gmail.com> wrote:
> I see that you haven't subscribed to mailing list yet. Please do that my
> sending an email to user-subscribe@helix.apache.org.
>
> I am also available on #apachehelix irc if you have additional questions.
> http://helix.apache.org/IRC.html
>
>
> ---------- Forwarded message ----------
> From: kishore g <g....@gmail.com>
> Date: Wed, Jun 24, 2015 at 10:37 PM
> Subject: Re: Customized rebalancer
> To: "user@helix.apache.org" <us...@helix.apache.org>
>
>
> Hi Changgeng,
>
> I think the first invocation will always be null, subsequent invocations
> should provide you the previous resource assignment. Are you saying its
> always null?.
>
> thanks,
> Kishore G
>
> On Wed, Jun 24, 2015 at 3:42 PM, Changgeng Li <li...@gmail.com>
> wrote:
>
>> Hello,
>>
>> We have a use case that when adding or removing a new node, we hope to
>> minimize the shuffle of partitions between nodes. I'm trying to implement a
>> customized rebalancer calculating the new resource assignment based on the
>> previous resource assignment. When a new node is added, just move some
>> partitions from existing node to the new node, and when a node is down,
>> move the partitions on this node to the other nodes. Partitions would not
>> move between two nodes if the both status are not changed.
>>
>> The documentation says:
>> In rebalance(), ... the third is the output of the previous invocation
>> of this method (if supported) ...
>>
>> During my testing I found the previous resource assignment passed in is
>> always null. my question is what "if supported" means here? Does the
>> rebalancer need to support something?
>>
>>
>> Any other suggestions are also welcome.
>>
>> Thanks,
>> Changgeng
>>
>
>
>
Re: Customized rebalancer
Posted by kishore g <g....@gmail.com>.
Hi Changgeng,
I think the first invocation will always be null, subsequent invocations
should provide you the previous resource assignment. Are you saying its
always null?.
thanks,
Kishore G
On Wed, Jun 24, 2015 at 3:42 PM, Changgeng Li <li...@gmail.com>
wrote:
> Hello,
>
> We have a use case that when adding or removing a new node, we hope to
> minimize the shuffle of partitions between nodes. I'm trying to implement a
> customized rebalancer calculating the new resource assignment based on the
> previous resource assignment. When a new node is added, just move some
> partitions from existing node to the new node, and when a node is down,
> move the partitions on this node to the other nodes. Partitions would not
> move between two nodes if the both status are not changed.
>
> The documentation says:
> In rebalance(), ... the third is the output of the previous invocation of
> this method (if supported) ...
>
> During my testing I found the previous resource assignment passed in is
> always null. my question is what "if supported" means here? Does the
> rebalancer need to support something?
>
>
> Any other suggestions are also welcome.
>
> Thanks,
> Changgeng
>