You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@helix.apache.org by Hang Qi <ha...@gmail.com> on 2015/09/15 08:04:21 UTC
Question regarding to CUSTOMIZED balance mode
Hi,
We are evaluating CUSTOMIZED balance mode in helix 0.6.4 currently, and I
have one question regarding to it.
How do I handle cluster failure case, e.g. the initial ideal state as
following:
"mapFields" : {
"MyResource_0" : {
"N1" : "MASTER",
"N2" : "SLAVE",
}
}
If N1 goes down, N2 will not become MASTER automatically, right?
If so, how do I register the listener for cluster state change to update
ideal state? The doc
<http://helix.apache.org/0.6.4-docs/tutorial_rebalance.html> says "The
application needs to implement a callback interface that Helix invokes when
the cluster state changes. Within this callback, the application can
recompute the idealstate.", is that interface Rebalancer
<https://github.com/apache/helix/blob/helix-0.6.4/helix-core/src/main/java/org/apache/helix/controller/rebalancer/Rebalancer.java>
?
Do we have an example for CUSTOMIZED balance?
Besides, when looking at the code, CustomRebalancer
<https://github.com/apache/helix/blob/helix-0.6.4/helix-core/src/main/java/org/apache/helix/controller/rebalancer/CustomRebalancer.java>
implements
Rebalancer interface, and just returns currentIdealState when
computeNewIdealState is called.
Thanks
Hang Qi
Re: Question regarding to CUSTOMIZED balance mode
Posted by Hang Qi <ha...@gmail.com>.
Hi Krishore,
I'v opened a pull request https://github.com/apache/helix/pull/37 for the
rebalancer.
It is not a complete pull request, but gives some idea what we are trying
to do with the rebalancer.
Thanks
Hang Qi
On Mon, Sep 21, 2015 at 12:56 PM, Hang Qi <ha...@gmail.com> wrote:
> Sure, I will first test the new rebalancer, and after that I can send out
> the pull request, and Iet's see whether it is generalized or not.
>
> Thanks
> Hang Qi
>
>
> On Monday, September 21, 2015, kishore g <g....@gmail.com> wrote:
>
>> Please consider contributing the rebalancer to Helix if it can be
>> generalized.
>> On Sep 21, 2015 11:57 AM, "Hang Qi" <ha...@gmail.com> wrote:
>>
>>> Hi Kanak & Kishore,
>>>
>>> Thanks very much for your reply.
>>>
>>> Yes, I have looked into the HelixCustomCodeRunner by myself, however I
>>> ended up to go with option 2, write our own rebalancer.
>>>
>>> Our use case is that we want to extend the Master/Slave state model by
>>> adding one special state for special purpose, like partition splitting.
>>> Normally, partitions still follow Master/Slave model to perform the state
>>> transition. And we have a admin node, which can mark some partitions on one
>>> instance to this special state in IdealState before the operation, and
>>> after the operation is done, the admin node can remove the special state
>>> for those partitions on the instance in IdealState.
>>>
>>> We are using SEMI_AUTO mode right now, so we are looking into the CUSTOM
>>> mode, later on, we think the our new rebalancer is kind of mixing of
>>> SEMI_AUTO and CUSTOM, thus we go with option 2.
>>>
>>> Thanks
>>> Hang Qi
>>>
>>> On Sat, Sep 19, 2015 at 2:04 PM, kishore g <g....@gmail.com> wrote:
>>>
>>>> There are two ways to write your own rebalancer
>>>>
>>>> #1. Using CustomCode Runner. This is regsistered in any of the
>>>> participants and can be invoked on any node in the cluster starts/stops.
>>>> Kanak's email points to this option.
>>>>
>>>> #2. Implement the rebalancer interface. This is one you have specified
>>>> in your email. This is run in controller and is invoked on each and every
>>>> change in the cluster such as node added/removed, start/stop, resource
>>>> add/remove etc.
>>>>
>>>> Let me know which mode you are planning to use, we can provide
>>>> additional details.
>>>>
>>>>
>>>>
>>>> On Sat, Sep 19, 2015 at 11:19 AM, Kanak Biscuitwala <
>>>> kanak.b@hotmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>> Yeah, you need to register a custom code runner. Here is an example:
>>>>> https://github.com/apache/helix/blob/master/helix-core/src/test/java/org/apache/helix/integration/TestHelixCustomCodeRunner.java
>>>>>
>>>>>
>>>>>
>>>>> Kanak
>>>>>
>>>>> From: hangq.1985@gmail.com
>>>>> Date: Mon, 14 Sep 2015 23:04:21 -0700
>>>>> Subject: Question regarding to CUSTOMIZED balance mode
>>>>> To: user@helix.apache.org
>>>>>
>>>>> Hi,
>>>>>
>>>>> We are evaluating CUSTOMIZED balance mode in helix 0.6.4 currently,
>>>>> and I have one question regarding to it.
>>>>>
>>>>> How do I handle cluster failure case, e.g. the initial ideal state as
>>>>> following:
>>>>> "mapFields" : {
>>>>> "MyResource_0" : {
>>>>> "N1" : "MASTER",
>>>>> "N2" : "SLAVE",
>>>>> }
>>>>> }
>>>>> If N1 goes down, N2 will not become MASTER automatically, right?
>>>>>
>>>>> If so, how do I register the listener for cluster state change to
>>>>> update ideal state? The doc says "The application needs to implement a
>>>>> callback interface that Helix invokes when the cluster state changes.
>>>>> Within this callback, the application can recompute the idealstate.", is
>>>>> that interface Rebalancer?
>>>>>
>>>>> Do we have an example for CUSTOMIZED balance?
>>>>>
>>>>> Besides, when looking at the code, CustomRebalancer implements
>>>>> Rebalancer interface, and just returns currentIdealState
>>>>> when computeNewIdealState is called.
>>>>>
>>>>> Thanks
>>>>> Hang Qi
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>
> --
> Qi hang
>
--
Qi hang
Re: Question regarding to CUSTOMIZED balance mode
Posted by Hang Qi <ha...@gmail.com>.
Sure, I will first test the new rebalancer, and after that I can send out
the pull request, and Iet's see whether it is generalized or not.
Thanks
Hang Qi
On Monday, September 21, 2015, kishore g <g....@gmail.com> wrote:
> Please consider contributing the rebalancer to Helix if it can be
> generalized.
> On Sep 21, 2015 11:57 AM, "Hang Qi" <hangq.1985@gmail.com
> <javascript:_e(%7B%7D,'cvml','hangq.1985@gmail.com');>> wrote:
>
>> Hi Kanak & Kishore,
>>
>> Thanks very much for your reply.
>>
>> Yes, I have looked into the HelixCustomCodeRunner by myself, however I
>> ended up to go with option 2, write our own rebalancer.
>>
>> Our use case is that we want to extend the Master/Slave state model by
>> adding one special state for special purpose, like partition splitting.
>> Normally, partitions still follow Master/Slave model to perform the state
>> transition. And we have a admin node, which can mark some partitions on one
>> instance to this special state in IdealState before the operation, and
>> after the operation is done, the admin node can remove the special state
>> for those partitions on the instance in IdealState.
>>
>> We are using SEMI_AUTO mode right now, so we are looking into the CUSTOM
>> mode, later on, we think the our new rebalancer is kind of mixing of
>> SEMI_AUTO and CUSTOM, thus we go with option 2.
>>
>> Thanks
>> Hang Qi
>>
>> On Sat, Sep 19, 2015 at 2:04 PM, kishore g <g.kishore@gmail.com
>> <javascript:_e(%7B%7D,'cvml','g.kishore@gmail.com');>> wrote:
>>
>>> There are two ways to write your own rebalancer
>>>
>>> #1. Using CustomCode Runner. This is regsistered in any of the
>>> participants and can be invoked on any node in the cluster starts/stops.
>>> Kanak's email points to this option.
>>>
>>> #2. Implement the rebalancer interface. This is one you have specified
>>> in your email. This is run in controller and is invoked on each and every
>>> change in the cluster such as node added/removed, start/stop, resource
>>> add/remove etc.
>>>
>>> Let me know which mode you are planning to use, we can provide
>>> additional details.
>>>
>>>
>>>
>>> On Sat, Sep 19, 2015 at 11:19 AM, Kanak Biscuitwala <kanak.b@hotmail.com
>>> <javascript:_e(%7B%7D,'cvml','kanak.b@hotmail.com');>> wrote:
>>>
>>>> Hi,
>>>>
>>>>
>>>> Yeah, you need to register a custom code runner. Here is an example:
>>>> https://github.com/apache/helix/blob/master/helix-core/src/test/java/org/apache/helix/integration/TestHelixCustomCodeRunner.java
>>>>
>>>>
>>>>
>>>> Kanak
>>>>
>>>> From: hangq.1985@gmail.com
>>>> <javascript:_e(%7B%7D,'cvml','hangq.1985@gmail.com');>
>>>> Date: Mon, 14 Sep 2015 23:04:21 -0700
>>>> Subject: Question regarding to CUSTOMIZED balance mode
>>>> To: user@helix.apache.org
>>>> <javascript:_e(%7B%7D,'cvml','user@helix.apache.org');>
>>>>
>>>> Hi,
>>>>
>>>> We are evaluating CUSTOMIZED balance mode in helix 0.6.4 currently, and
>>>> I have one question regarding to it.
>>>>
>>>> How do I handle cluster failure case, e.g. the initial ideal state as
>>>> following:
>>>> "mapFields" : {
>>>> "MyResource_0" : {
>>>> "N1" : "MASTER",
>>>> "N2" : "SLAVE",
>>>> }
>>>> }
>>>> If N1 goes down, N2 will not become MASTER automatically, right?
>>>>
>>>> If so, how do I register the listener for cluster state change to
>>>> update ideal state? The doc says "The application needs to implement a
>>>> callback interface that Helix invokes when the cluster state changes.
>>>> Within this callback, the application can recompute the idealstate.", is
>>>> that interface Rebalancer?
>>>>
>>>> Do we have an example for CUSTOMIZED balance?
>>>>
>>>> Besides, when looking at the code, CustomRebalancer implements
>>>> Rebalancer interface, and just returns currentIdealState
>>>> when computeNewIdealState is called.
>>>>
>>>> Thanks
>>>> Hang Qi
>>>>
>>>
>>>
>>
>>
>>
>>
--
Qi hang
Re: Question regarding to CUSTOMIZED balance mode
Posted by kishore g <g....@gmail.com>.
Please consider contributing the rebalancer to Helix if it can be
generalized.
On Sep 21, 2015 11:57 AM, "Hang Qi" <ha...@gmail.com> wrote:
> Hi Kanak & Kishore,
>
> Thanks very much for your reply.
>
> Yes, I have looked into the HelixCustomCodeRunner by myself, however I
> ended up to go with option 2, write our own rebalancer.
>
> Our use case is that we want to extend the Master/Slave state model by
> adding one special state for special purpose, like partition splitting.
> Normally, partitions still follow Master/Slave model to perform the state
> transition. And we have a admin node, which can mark some partitions on one
> instance to this special state in IdealState before the operation, and
> after the operation is done, the admin node can remove the special state
> for those partitions on the instance in IdealState.
>
> We are using SEMI_AUTO mode right now, so we are looking into the CUSTOM
> mode, later on, we think the our new rebalancer is kind of mixing of
> SEMI_AUTO and CUSTOM, thus we go with option 2.
>
> Thanks
> Hang Qi
>
> On Sat, Sep 19, 2015 at 2:04 PM, kishore g <g....@gmail.com> wrote:
>
>> There are two ways to write your own rebalancer
>>
>> #1. Using CustomCode Runner. This is regsistered in any of the
>> participants and can be invoked on any node in the cluster starts/stops.
>> Kanak's email points to this option.
>>
>> #2. Implement the rebalancer interface. This is one you have specified in
>> your email. This is run in controller and is invoked on each and every
>> change in the cluster such as node added/removed, start/stop, resource
>> add/remove etc.
>>
>> Let me know which mode you are planning to use, we can provide additional
>> details.
>>
>>
>>
>> On Sat, Sep 19, 2015 at 11:19 AM, Kanak Biscuitwala <ka...@hotmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>>
>>> Yeah, you need to register a custom code runner. Here is an example:
>>> https://github.com/apache/helix/blob/master/helix-core/src/test/java/org/apache/helix/integration/TestHelixCustomCodeRunner.java
>>>
>>>
>>>
>>> Kanak
>>>
>>> From: hangq.1985@gmail.com
>>> Date: Mon, 14 Sep 2015 23:04:21 -0700
>>> Subject: Question regarding to CUSTOMIZED balance mode
>>> To: user@helix.apache.org
>>>
>>> Hi,
>>>
>>> We are evaluating CUSTOMIZED balance mode in helix 0.6.4 currently, and
>>> I have one question regarding to it.
>>>
>>> How do I handle cluster failure case, e.g. the initial ideal state as
>>> following:
>>> "mapFields" : {
>>> "MyResource_0" : {
>>> "N1" : "MASTER",
>>> "N2" : "SLAVE",
>>> }
>>> }
>>> If N1 goes down, N2 will not become MASTER automatically, right?
>>>
>>> If so, how do I register the listener for cluster state change to update
>>> ideal state? The doc says "The application needs to implement a callback
>>> interface that Helix invokes when the cluster state changes. Within this
>>> callback, the application can recompute the idealstate.", is that interface
>>> Rebalancer?
>>>
>>> Do we have an example for CUSTOMIZED balance?
>>>
>>> Besides, when looking at the code, CustomRebalancer implements
>>> Rebalancer interface, and just returns currentIdealState
>>> when computeNewIdealState is called.
>>>
>>> Thanks
>>> Hang Qi
>>>
>>
>>
>
>
>
>
Re: Question regarding to CUSTOMIZED balance mode
Posted by Hang Qi <ha...@gmail.com>.
Hi Kanak & Kishore,
Thanks very much for your reply.
Yes, I have looked into the HelixCustomCodeRunner by myself, however I
ended up to go with option 2, write our own rebalancer.
Our use case is that we want to extend the Master/Slave state model by
adding one special state for special purpose, like partition splitting.
Normally, partitions still follow Master/Slave model to perform the state
transition. And we have a admin node, which can mark some partitions on one
instance to this special state in IdealState before the operation, and
after the operation is done, the admin node can remove the special state
for those partitions on the instance in IdealState.
We are using SEMI_AUTO mode right now, so we are looking into the CUSTOM
mode, later on, we think the our new rebalancer is kind of mixing of
SEMI_AUTO and CUSTOM, thus we go with option 2.
Thanks
Hang Qi
On Sat, Sep 19, 2015 at 2:04 PM, kishore g <g....@gmail.com> wrote:
> There are two ways to write your own rebalancer
>
> #1. Using CustomCode Runner. This is regsistered in any of the
> participants and can be invoked on any node in the cluster starts/stops.
> Kanak's email points to this option.
>
> #2. Implement the rebalancer interface. This is one you have specified in
> your email. This is run in controller and is invoked on each and every
> change in the cluster such as node added/removed, start/stop, resource
> add/remove etc.
>
> Let me know which mode you are planning to use, we can provide additional
> details.
>
>
>
> On Sat, Sep 19, 2015 at 11:19 AM, Kanak Biscuitwala <ka...@hotmail.com>
> wrote:
>
>> Hi,
>>
>>
>> Yeah, you need to register a custom code runner. Here is an example:
>> https://github.com/apache/helix/blob/master/helix-core/src/test/java/org/apache/helix/integration/TestHelixCustomCodeRunner.java
>>
>>
>>
>> Kanak
>>
>> From: hangq.1985@gmail.com
>> Date: Mon, 14 Sep 2015 23:04:21 -0700
>> Subject: Question regarding to CUSTOMIZED balance mode
>> To: user@helix.apache.org
>>
>> Hi,
>>
>> We are evaluating CUSTOMIZED balance mode in helix 0.6.4 currently, and I
>> have one question regarding to it.
>>
>> How do I handle cluster failure case, e.g. the initial ideal state as
>> following:
>> "mapFields" : {
>> "MyResource_0" : {
>> "N1" : "MASTER",
>> "N2" : "SLAVE",
>> }
>> }
>> If N1 goes down, N2 will not become MASTER automatically, right?
>>
>> If so, how do I register the listener for cluster state change to update
>> ideal state? The doc says "The application needs to implement a callback
>> interface that Helix invokes when the cluster state changes. Within this
>> callback, the application can recompute the idealstate.", is that interface
>> Rebalancer?
>>
>> Do we have an example for CUSTOMIZED balance?
>>
>> Besides, when looking at the code, CustomRebalancer implements Rebalancer
>> interface, and just returns currentIdealState when computeNewIdealState is
>> called.
>>
>> Thanks
>> Hang Qi
>>
>
>
Re: Question regarding to CUSTOMIZED balance mode
Posted by kishore g <g....@gmail.com>.
There are two ways to write your own rebalancer
#1. Using CustomCode Runner. This is regsistered in any of the participants
and can be invoked on any node in the cluster starts/stops. Kanak's email
points to this option.
#2. Implement the rebalancer interface. This is one you have specified in
your email. This is run in controller and is invoked on each and every
change in the cluster such as node added/removed, start/stop, resource
add/remove etc.
Let me know which mode you are planning to use, we can provide additional
details.
On Sat, Sep 19, 2015 at 11:19 AM, Kanak Biscuitwala <ka...@hotmail.com>
wrote:
> Hi,
>
>
> Yeah, you need to register a custom code runner. Here is an example:
> https://github.com/apache/helix/blob/master/helix-core/src/test/java/org/apache/helix/integration/TestHelixCustomCodeRunner.java
>
>
>
> Kanak
>
> From: hangq.1985@gmail.com
> Date: Mon, 14 Sep 2015 23:04:21 -0700
> Subject: Question regarding to CUSTOMIZED balance mode
> To: user@helix.apache.org
>
> Hi,
>
> We are evaluating CUSTOMIZED balance mode in helix 0.6.4 currently, and I
> have one question regarding to it.
>
> How do I handle cluster failure case, e.g. the initial ideal state as
> following:
> "mapFields" : {
> "MyResource_0" : {
> "N1" : "MASTER",
> "N2" : "SLAVE",
> }
> }
> If N1 goes down, N2 will not become MASTER automatically, right?
>
> If so, how do I register the listener for cluster state change to update
> ideal state? The doc says "The application needs to implement a callback
> interface that Helix invokes when the cluster state changes. Within this
> callback, the application can recompute the idealstate.", is that interface
> Rebalancer?
>
> Do we have an example for CUSTOMIZED balance?
>
> Besides, when looking at the code, CustomRebalancer implements Rebalancer
> interface, and just returns currentIdealState when computeNewIdealState is
> called.
>
> Thanks
> Hang Qi
>
RE: Question regarding to CUSTOMIZED balance mode
Posted by Kanak Biscuitwala <ka...@hotmail.com>.
Hi,
Yeah, you need to register a custom code runner. Here is an example: https://github.com/apache/helix/blob/master/helix-core/src/test/java/org/apache/helix/integration/TestHelixCustomCodeRunner.java
Kanak
From: hangq.1985@gmail.com
Date: Mon, 14 Sep 2015 23:04:21 -0700
Subject: Question regarding to CUSTOMIZED balance mode
To: user@helix.apache.org
Hi,
We are evaluating CUSTOMIZED balance mode in helix 0.6.4 currently, and I have one question regarding to it.
How do I handle cluster failure case, e.g. the initial ideal state as following:
"mapFields" : {
"MyResource_0" : {
"N1" : "MASTER",
"N2" : "SLAVE",
}
}
If N1 goes down, N2 will not become MASTER automatically, right?
If so, how do I register the listener for cluster state change to update ideal state? The doc says "The application needs to implement a callback interface that Helix invokes when the cluster state changes. Within this callback, the application can recompute the idealstate.", is that interface Rebalancer?
Do we have an example for CUSTOMIZED balance?
Besides, when looking at the code, CustomRebalancer implements Rebalancer interface, and just returns currentIdealState when computeNewIdealState is called.
Thanks
Hang Qi