You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@helix.apache.org by Rob McKinnon <ro...@gmail.com> on 2018/10/08 12:36:42 UTC
User-Define Rebalancing Issue
I am working through the tutorial on the User-Defined Rebalancing and I am
not able to get my Rebalancer to be invoked.
Sample Code
======
public static void main(String[] args) {
ZKHelixAdmin admin = new ZKHelixAdmin(Conf.ZK_ADDRESS);
IdealState idealState =
admin.getResourceIdealState(Conf.CLUSTER_NAME, Conf.RESOURCE_NAME);
idealState.setRebalanceMode(IdealState.RebalanceMode.USER_DEFINED);
idealState.setRebalancerClassName(MyRebalancer.class.getName());
admin.setResourceIdealState(Conf.CLUSTER_NAME, Conf.RESOURCE_NAME,
idealState);
admin.rebalance(Conf.CLUSTER_NAME, RESOURCE_NAME, NUM_REPLICAS);
}
======
I was expecting that when calling the "admin.rebalance" method, it would
invoke "MyRebalance" code but when I run it "MyRebalance" code was not
invoked.
Thanks,
Rob
Re: User-Define Rebalancing Issue
Posted by Rob McKinnon <ro...@gmail.com>.
Kishore,
I pushed the code to git@github.com:rmckinnon/helix-poc.git, the rebalance
code is copied from the tutorial.
1. The user defined configuration for rebalancing can be found in
STEP5_Rebalance.java
2. The participants should be running before the rebalance happens
After the rebalancing is invoked, I see the following information about the
cluster:
IdealState for data:
{
"id" : "data",
"mapFields" : {
"data_0" : {
},
"data_1" : {
},
"data_2" : {
},
"data_3" : {
},
"data_4" : {
},
"data_5" : {
}
},
"listFields" : {
"data_0" : [ ],
"data_1" : [ ],
"data_2" : [ ],
"data_3" : [ ],
"data_4" : [ ],
"data_5" : [ ]
},
"simpleFields" : {
"IDEAL_STATE_MODE" : "AUTO",
"NUM_PARTITIONS" : "6",
"REBALANCER_CLASS_NAME" : "rob.local.MyRebalancer",
"REBALANCE_MODE" : "USER_DEFINED",
"REBALANCE_STRATEGY" : "DEFAULT",
"REPLICAS" : "1",
"STATE_MODEL_DEF_REF" : "OnlineOffline",
"STATE_MODEL_FACTORY_NAME" : "DEFAULT"
}
}
ExternalView for data:
{
"id" : "data",
"mapFields" : {
},
"listFields" : {
},
"simpleFields" : {
"BUCKET_SIZE" : "0",
"IDEAL_STATE_MODE" : "AUTO",
"NUM_PARTITIONS" : "6",
"REBALANCER_CLASS_NAME" : "rob.local.MyRebalancer",
"REBALANCE_MODE" : "USER_DEFINED",
"REBALANCE_STRATEGY" : "DEFAULT",
"REPLICAS" : "1",
"STATE_MODEL_DEF_REF" : "OnlineOffline",
"STATE_MODEL_FACTORY_NAME" : "DEFAULT"
}
}
I was expecting that the IdealState or the External View mapfields would
have changed after the rebalancing. I did have my own implementation but
defaulted back to the Tutorial example after I saw my code had no affect on
the cluster.
Thanks,
Rob
On Tue, Oct 16, 2018 at 10:17 AM kishore g <g....@gmail.com> wrote:
> Can you share the code? Feel free to IM me on gtalk.
>
> On Tue, Oct 16, 2018 at 4:54 AM Rob McKinnon <ro...@gmail.com>
> wrote:
>
>> Kishore or Lei,
>>
>> I am wondering if you can comment on my question? I am trying to figure
>> out if I am understanding the User-Define Rebalancing correctly or is there
>> an issue with the software.
>>
>>
>> Thanks,
>>
>> Rob
>>
>> On Wed, Oct 10, 2018 at 2:48 PM Rob McKinnon <ro...@gmail.com>
>> wrote:
>>
>>> I was using the "run-helix-controll.sh" script to handle the controller
>>> and after implementing a custom controller, I am now able to get the
>>> Rebalancer to be invoked.
>>>
>>> I am running into a related issue as the result of my rebalancing
>>> doesn't seem to have any affect on the cluster. I am assuming that the
>>> Rebalance functionality affects the external view? In the tutorial example,
>>> it looks like the currentIdealState preference list is being updated and a
>>> variable called "assignment" is returned. I did similar things in my
>>> implementations but I am not seeing any changes to the cluster. 1. Am I
>>> understanding this correctly? 2. Should I see the changes in the external
>>> view?
>>>
>>> On Tue, Oct 9, 2018 at 1:02 PM kishore g <g....@gmail.com> wrote:
>>>
>>>> Admin.rebalance method won’t invoke user rebalancer
>>>>
>>>> The rebalancer is invoked when ever there is a change in the cluster
>>>> such as nodes added/started/stopped etc.
>>>>
>>>> Also look for error message in the log as Lei suggested. Note the
>>>> controller should have access to the user defined rebalancer class
>>>>
>>>> On Tue, Oct 9, 2018 at 9:46 AM Lei Xia <xi...@gmail.com> wrote:
>>>>
>>>>> Hi, Rob
>>>>>
>>>>> It seems your rebalancer class is not correctly linked into Helix
>>>>> lib, i.e, Helix can not find your rebalancer class at runtime. Do you see
>>>>> some error log from Helix like: ""Exception while invoking custom
>>>>> rebalancer class:"?
>>>>>
>>>>> On Mon, Oct 8, 2018 at 5:37 AM Rob McKinnon <ro...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I am working through the tutorial on the User-Defined Rebalancing and
>>>>>> I am not able to get my Rebalancer to be invoked.
>>>>>>
>>>>>> Sample Code
>>>>>> ======
>>>>>>
>>>>>> public static void main(String[] args) {
>>>>>> ZKHelixAdmin admin = new ZKHelixAdmin(Conf.ZK_ADDRESS);
>>>>>>
>>>>>> IdealState idealState = admin.getResourceIdealState(Conf.CLUSTER_NAME, Conf.RESOURCE_NAME);
>>>>>> idealState.setRebalanceMode(IdealState.RebalanceMode.USER_DEFINED);
>>>>>> idealState.setRebalancerClassName(MyRebalancer.class.getName());
>>>>>> admin.setResourceIdealState(Conf.CLUSTER_NAME, Conf.RESOURCE_NAME, idealState);
>>>>>>
>>>>>>
>>>>>> admin.rebalance(Conf.CLUSTER_NAME, RESOURCE_NAME, NUM_REPLICAS);
>>>>>> }
>>>>>>
>>>>>> ======
>>>>>> I was expecting that when calling the "admin.rebalance" method, it
>>>>>> would invoke "MyRebalance" code but when I run it "MyRebalance" code was
>>>>>> not invoked.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Rob
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Lei Xia
>>>>>
>>>>
Re: User-Define Rebalancing Issue
Posted by kishore g <g....@gmail.com>.
Can you share the code? Feel free to IM me on gtalk.
On Tue, Oct 16, 2018 at 4:54 AM Rob McKinnon <ro...@gmail.com> wrote:
> Kishore or Lei,
>
> I am wondering if you can comment on my question? I am trying to figure
> out if I am understanding the User-Define Rebalancing correctly or is there
> an issue with the software.
>
>
> Thanks,
>
> Rob
>
> On Wed, Oct 10, 2018 at 2:48 PM Rob McKinnon <ro...@gmail.com>
> wrote:
>
>> I was using the "run-helix-controll.sh" script to handle the controller
>> and after implementing a custom controller, I am now able to get the
>> Rebalancer to be invoked.
>>
>> I am running into a related issue as the result of my rebalancing doesn't
>> seem to have any affect on the cluster. I am assuming that the Rebalance
>> functionality affects the external view? In the tutorial example, it looks
>> like the currentIdealState preference list is being updated and a variable
>> called "assignment" is returned. I did similar things in my implementations
>> but I am not seeing any changes to the cluster. 1. Am I understanding this
>> correctly? 2. Should I see the changes in the external view?
>>
>> On Tue, Oct 9, 2018 at 1:02 PM kishore g <g....@gmail.com> wrote:
>>
>>> Admin.rebalance method won’t invoke user rebalancer
>>>
>>> The rebalancer is invoked when ever there is a change in the cluster
>>> such as nodes added/started/stopped etc.
>>>
>>> Also look for error message in the log as Lei suggested. Note the
>>> controller should have access to the user defined rebalancer class
>>>
>>> On Tue, Oct 9, 2018 at 9:46 AM Lei Xia <xi...@gmail.com> wrote:
>>>
>>>> Hi, Rob
>>>>
>>>> It seems your rebalancer class is not correctly linked into Helix
>>>> lib, i.e, Helix can not find your rebalancer class at runtime. Do you see
>>>> some error log from Helix like: ""Exception while invoking custom
>>>> rebalancer class:"?
>>>>
>>>> On Mon, Oct 8, 2018 at 5:37 AM Rob McKinnon <ro...@gmail.com>
>>>> wrote:
>>>>
>>>>> I am working through the tutorial on the User-Defined Rebalancing and
>>>>> I am not able to get my Rebalancer to be invoked.
>>>>>
>>>>> Sample Code
>>>>> ======
>>>>>
>>>>> public static void main(String[] args) {
>>>>> ZKHelixAdmin admin = new ZKHelixAdmin(Conf.ZK_ADDRESS);
>>>>>
>>>>> IdealState idealState = admin.getResourceIdealState(Conf.CLUSTER_NAME, Conf.RESOURCE_NAME);
>>>>> idealState.setRebalanceMode(IdealState.RebalanceMode.USER_DEFINED);
>>>>> idealState.setRebalancerClassName(MyRebalancer.class.getName());
>>>>> admin.setResourceIdealState(Conf.CLUSTER_NAME, Conf.RESOURCE_NAME, idealState);
>>>>>
>>>>>
>>>>> admin.rebalance(Conf.CLUSTER_NAME, RESOURCE_NAME, NUM_REPLICAS);
>>>>> }
>>>>>
>>>>> ======
>>>>> I was expecting that when calling the "admin.rebalance" method, it
>>>>> would invoke "MyRebalance" code but when I run it "MyRebalance" code was
>>>>> not invoked.
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Rob
>>>>>
>>>>
>>>>
>>>> --
>>>> Lei Xia
>>>>
>>>
Re: User-Define Rebalancing Issue
Posted by Rob McKinnon <ro...@gmail.com>.
Kishore or Lei,
I am wondering if you can comment on my question? I am trying to figure out
if I am understanding the User-Define Rebalancing correctly or is there an
issue with the software.
Thanks,
Rob
On Wed, Oct 10, 2018 at 2:48 PM Rob McKinnon <ro...@gmail.com> wrote:
> I was using the "run-helix-controll.sh" script to handle the controller
> and after implementing a custom controller, I am now able to get the
> Rebalancer to be invoked.
>
> I am running into a related issue as the result of my rebalancing doesn't
> seem to have any affect on the cluster. I am assuming that the Rebalance
> functionality affects the external view? In the tutorial example, it looks
> like the currentIdealState preference list is being updated and a variable
> called "assignment" is returned. I did similar things in my implementations
> but I am not seeing any changes to the cluster. 1. Am I understanding this
> correctly? 2. Should I see the changes in the external view?
>
> On Tue, Oct 9, 2018 at 1:02 PM kishore g <g....@gmail.com> wrote:
>
>> Admin.rebalance method won’t invoke user rebalancer
>>
>> The rebalancer is invoked when ever there is a change in the cluster such
>> as nodes added/started/stopped etc.
>>
>> Also look for error message in the log as Lei suggested. Note the
>> controller should have access to the user defined rebalancer class
>>
>> On Tue, Oct 9, 2018 at 9:46 AM Lei Xia <xi...@gmail.com> wrote:
>>
>>> Hi, Rob
>>>
>>> It seems your rebalancer class is not correctly linked into Helix lib,
>>> i.e, Helix can not find your rebalancer class at runtime. Do you see some
>>> error log from Helix like: ""Exception while invoking custom rebalancer
>>> class:"?
>>>
>>> On Mon, Oct 8, 2018 at 5:37 AM Rob McKinnon <ro...@gmail.com>
>>> wrote:
>>>
>>>> I am working through the tutorial on the User-Defined Rebalancing and I
>>>> am not able to get my Rebalancer to be invoked.
>>>>
>>>> Sample Code
>>>> ======
>>>>
>>>> public static void main(String[] args) {
>>>> ZKHelixAdmin admin = new ZKHelixAdmin(Conf.ZK_ADDRESS);
>>>>
>>>> IdealState idealState = admin.getResourceIdealState(Conf.CLUSTER_NAME, Conf.RESOURCE_NAME);
>>>> idealState.setRebalanceMode(IdealState.RebalanceMode.USER_DEFINED);
>>>> idealState.setRebalancerClassName(MyRebalancer.class.getName());
>>>> admin.setResourceIdealState(Conf.CLUSTER_NAME, Conf.RESOURCE_NAME, idealState);
>>>>
>>>>
>>>> admin.rebalance(Conf.CLUSTER_NAME, RESOURCE_NAME, NUM_REPLICAS);
>>>> }
>>>>
>>>> ======
>>>> I was expecting that when calling the "admin.rebalance" method, it
>>>> would invoke "MyRebalance" code but when I run it "MyRebalance" code was
>>>> not invoked.
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Rob
>>>>
>>>
>>>
>>> --
>>> Lei Xia
>>>
>>
Re: User-Define Rebalancing Issue
Posted by Rob McKinnon <ro...@gmail.com>.
I was using the "run-helix-controll.sh" script to handle the controller and
after implementing a custom controller, I am now able to get the Rebalancer
to be invoked.
I am running into a related issue as the result of my rebalancing doesn't
seem to have any affect on the cluster. I am assuming that the Rebalance
functionality affects the external view? In the tutorial example, it looks
like the currentIdealState preference list is being updated and a variable
called "assignment" is returned. I did similar things in my implementations
but I am not seeing any changes to the cluster. 1. Am I understanding this
correctly? 2. Should I see the changes in the external view?
On Tue, Oct 9, 2018 at 1:02 PM kishore g <g....@gmail.com> wrote:
> Admin.rebalance method won’t invoke user rebalancer
>
> The rebalancer is invoked when ever there is a change in the cluster such
> as nodes added/started/stopped etc.
>
> Also look for error message in the log as Lei suggested. Note the
> controller should have access to the user defined rebalancer class
>
> On Tue, Oct 9, 2018 at 9:46 AM Lei Xia <xi...@gmail.com> wrote:
>
>> Hi, Rob
>>
>> It seems your rebalancer class is not correctly linked into Helix lib,
>> i.e, Helix can not find your rebalancer class at runtime. Do you see some
>> error log from Helix like: ""Exception while invoking custom rebalancer
>> class:"?
>>
>> On Mon, Oct 8, 2018 at 5:37 AM Rob McKinnon <ro...@gmail.com>
>> wrote:
>>
>>> I am working through the tutorial on the User-Defined Rebalancing and I
>>> am not able to get my Rebalancer to be invoked.
>>>
>>> Sample Code
>>> ======
>>>
>>> public static void main(String[] args) {
>>> ZKHelixAdmin admin = new ZKHelixAdmin(Conf.ZK_ADDRESS);
>>>
>>> IdealState idealState = admin.getResourceIdealState(Conf.CLUSTER_NAME, Conf.RESOURCE_NAME);
>>> idealState.setRebalanceMode(IdealState.RebalanceMode.USER_DEFINED);
>>> idealState.setRebalancerClassName(MyRebalancer.class.getName());
>>> admin.setResourceIdealState(Conf.CLUSTER_NAME, Conf.RESOURCE_NAME, idealState);
>>>
>>>
>>> admin.rebalance(Conf.CLUSTER_NAME, RESOURCE_NAME, NUM_REPLICAS);
>>> }
>>>
>>> ======
>>> I was expecting that when calling the "admin.rebalance" method, it would
>>> invoke "MyRebalance" code but when I run it "MyRebalance" code was not
>>> invoked.
>>>
>>>
>>> Thanks,
>>>
>>> Rob
>>>
>>
>>
>> --
>> Lei Xia
>>
>
Re: User-Define Rebalancing Issue
Posted by kishore g <g....@gmail.com>.
Admin.rebalance method won’t invoke user rebalancer
The rebalancer is invoked when ever there is a change in the cluster such
as nodes added/started/stopped etc.
Also look for error message in the log as Lei suggested. Note the
controller should have access to the user defined rebalancer class
On Tue, Oct 9, 2018 at 9:46 AM Lei Xia <xi...@gmail.com> wrote:
> Hi, Rob
>
> It seems your rebalancer class is not correctly linked into Helix lib,
> i.e, Helix can not find your rebalancer class at runtime. Do you see some
> error log from Helix like: ""Exception while invoking custom rebalancer
> class:"?
>
> On Mon, Oct 8, 2018 at 5:37 AM Rob McKinnon <ro...@gmail.com>
> wrote:
>
>> I am working through the tutorial on the User-Defined Rebalancing and I
>> am not able to get my Rebalancer to be invoked.
>>
>> Sample Code
>> ======
>>
>> public static void main(String[] args) {
>> ZKHelixAdmin admin = new ZKHelixAdmin(Conf.ZK_ADDRESS);
>>
>> IdealState idealState = admin.getResourceIdealState(Conf.CLUSTER_NAME, Conf.RESOURCE_NAME);
>> idealState.setRebalanceMode(IdealState.RebalanceMode.USER_DEFINED);
>> idealState.setRebalancerClassName(MyRebalancer.class.getName());
>> admin.setResourceIdealState(Conf.CLUSTER_NAME, Conf.RESOURCE_NAME, idealState);
>>
>>
>> admin.rebalance(Conf.CLUSTER_NAME, RESOURCE_NAME, NUM_REPLICAS);
>> }
>>
>> ======
>> I was expecting that when calling the "admin.rebalance" method, it would
>> invoke "MyRebalance" code but when I run it "MyRebalance" code was not
>> invoked.
>>
>>
>> Thanks,
>>
>> Rob
>>
>
>
> --
> Lei Xia
>
Re: User-Define Rebalancing Issue
Posted by Lei Xia <xi...@gmail.com>.
Hi, Rob
It seems your rebalancer class is not correctly linked into Helix lib,
i.e, Helix can not find your rebalancer class at runtime. Do you see some
error log from Helix like: ""Exception while invoking custom rebalancer
class:"?
On Mon, Oct 8, 2018 at 5:37 AM Rob McKinnon <ro...@gmail.com> wrote:
> I am working through the tutorial on the User-Defined Rebalancing and I am
> not able to get my Rebalancer to be invoked.
>
> Sample Code
> ======
>
> public static void main(String[] args) {
> ZKHelixAdmin admin = new ZKHelixAdmin(Conf.ZK_ADDRESS);
>
> IdealState idealState = admin.getResourceIdealState(Conf.CLUSTER_NAME, Conf.RESOURCE_NAME);
> idealState.setRebalanceMode(IdealState.RebalanceMode.USER_DEFINED);
> idealState.setRebalancerClassName(MyRebalancer.class.getName());
> admin.setResourceIdealState(Conf.CLUSTER_NAME, Conf.RESOURCE_NAME, idealState);
>
>
> admin.rebalance(Conf.CLUSTER_NAME, RESOURCE_NAME, NUM_REPLICAS);
> }
>
> ======
> I was expecting that when calling the "admin.rebalance" method, it would
> invoke "MyRebalance" code but when I run it "MyRebalance" code was not
> invoked.
>
>
> Thanks,
>
> Rob
>
--
Lei Xia