You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@helix.apache.org by Utsav Kanani <ut...@gmail.com> on 2018/02/23 22:33:24 UTC

Unable to get Rebalance Delay to work using the distributed lock manager recipe

I am trying to expand the Lockmanager example
http://helix.apache.org/0.6.2-incubating-docs/recipes/lock_manager.html to
introduce delay

tried doing something like this
IdealState state = admin.getResourceIdealState(clusterName, lockGroupName);
     state.setRebalanceDelay(100000);
     state.setDelayRebalanceEnabled(true);
     state.setRebalancerClassName(DelayedAutoRebalancer.class.getName());
     admin.setResourceIdealState(clusterName, lockGroupName, state);
     admin.rebalance(clusterName, lockGroupName, 1);

On killing a node there is immediate rebalancing which takes place. I was
hoping for a delay of 100 seconds before rebalancing but i am not seeing
that behavior


On Stopping localhost_12000 the locks are acquired immediately by
localhost_12001 and localhost_12002

on STARTING localhost_12000 the rebalance is again immediate.

localhost_12000 acquired lock:lock-group_11
localhost_12000 acquired lock:lock-group_7
localhost_12000 acquired lock:lock-group_0
localhost_12000 acquired lock:lock-group_4
STARTED localhost_12000
localhost_12001 releasing lock:lock-group_0
localhost_12001 releasing lock:lock-group_11
localhost_12002 releasing lock:lock-group_4
localhost_12002 releasing lock:lock-group_7


Here is the output
=========================================

STARTING localhost_12000
STARTING localhost_12001
STARTING localhost_12002
STARTED localhost_12001
STARTED localhost_12002
STARTED localhost_12000
localhost_12000 acquired lock:lock-group_11
localhost_12002 acquired lock:lock-group_10
localhost_12002 acquired lock:lock-group_9
localhost_12002 acquired lock:lock-group_3
localhost_12001 acquired lock:lock-group_2
localhost_12001 acquired lock:lock-group_1
localhost_12001 acquired lock:lock-group_8
localhost_12002 acquired lock:lock-group_6
localhost_12000 acquired lock:lock-group_4
localhost_12000 acquired lock:lock-group_0
localhost_12000 acquired lock:lock-group_7
localhost_12001 acquired lock:lock-group_5
lockName acquired By
======================================
lock-group_0 localhost_12000
lock-group_1 localhost_12001
lock-group_10 localhost_12002
lock-group_11 localhost_12000
lock-group_2 localhost_12001
lock-group_3 localhost_12002
lock-group_4 localhost_12000
lock-group_5 localhost_12001
lock-group_6 localhost_12002
lock-group_7 localhost_12000
lock-group_8 localhost_12001
lock-group_9 localhost_12002
Stopping localhost_12000
localhost_12000Interrupted
localhost_12002 acquired lock:lock-group_4
localhost_12001 acquired lock:lock-group_11
localhost_12002 acquired lock:lock-group_7
localhost_12001 acquired lock:lock-group_0
lockName acquired By
======================================
lock-group_0 localhost_12001
lock-group_1 localhost_12001
lock-group_10 localhost_12002
lock-group_11 localhost_12001
lock-group_2 localhost_12001
lock-group_3 localhost_12002
lock-group_4 localhost_12002
lock-group_5 localhost_12001
lock-group_6 localhost_12002
lock-group_7 localhost_12002
lock-group_8 localhost_12001
lock-group_9 localhost_12002
===Starting localhost_12000
STARTING localhost_12000
localhost_12000 acquired lock:lock-group_11
localhost_12000 acquired lock:lock-group_7
localhost_12000 acquired lock:lock-group_0
localhost_12000 acquired lock:lock-group_4
STARTED localhost_12000
localhost_12001 releasing lock:lock-group_0
localhost_12001 releasing lock:lock-group_11
localhost_12002 releasing lock:lock-group_4
localhost_12002 releasing lock:lock-group_7
lockName acquired By
======================================
lock-group_0 localhost_12000
lock-group_1 localhost_12001
lock-group_10 localhost_12002
lock-group_11 localhost_12000
lock-group_2 localhost_12001
lock-group_3 localhost_12002
lock-group_4 localhost_12000
lock-group_5 localhost_12001
lock-group_6 localhost_12002
lock-group_7 localhost_12000
lock-group_8 localhost_12001
lock-group_9 localhost_12002

Re: Unable to get Rebalance Delay to work using the distributed lock manager recipe

Posted by kishore g <g....@gmail.com>.

That's right. SEMI AUTO will ensure that the process always gets the lock.

https://helix.apache.org/0.6.4-docs/tutorial_rebalance.html

Do you mind describing the application?

On Thu, Mar 8, 2018 at 6:14 PM, Utsav Kanani <ut...@gmail.com> wrote:

> Thanks a ton for your help with this
>
> Had another quick question
>
> How can i configure such that there is no rebalancing and the same Process
> always gets the same lock.
> Is it with the SEMI_AUTO mode
>
> On Thu, Mar 8, 2018 at 6:11 PM, Utsav Kanani <ut...@gmail.com>
> wrote:
>
>> Ignore what is said it works with
>>
>> idealState.setRebalancerClassName(DelayedAutoRebalancer.class.getName());
>>
>>
>>
>> On Thu, Mar 8, 2018 at 6:08 PM, Utsav Kanani <ut...@gmail.com>
>> wrote:
>>
>>> sorry for the late response
>>> hey guys this does not help either
>>> do you guys want me to send you my code files?
>>>
>>> this is the code change i made in the ZKHelixAdmin class
>>> tried with both
>>>
>>> idealState.setRebalancerClassName(DelayedAutoRebalancer.class.getName());
>>>
>>> and withoug
>>>
>>> @Override
>>> public void addResource(String clusterName, String resourceName, int partitions,
>>>     String stateModelRef, String rebalancerMode, String rebalanceStrategy, int bucketSize,
>>>     int maxPartitionsPerInstance) {
>>>   if (!ZKUtil.isClusterSetup(clusterName, _zkClient)) {
>>>     throw new HelixException("cluster " + clusterName + " is not setup yet");
>>>   }
>>>
>>>   IdealState idealState = new IdealState(resourceName);
>>>   idealState.setNumPartitions(partitions);
>>>   idealState.setStateModelDefRef(stateModelRef);
>>>   RebalanceMode mode =
>>>       idealState.rebalanceModeFromString(rebalancerMode, RebalanceMode.SEMI_AUTO);
>>>   idealState.setRebalanceMode(mode);
>>>   idealState.setRebalanceStrategy(rebalanceStrategy);
>>>   -------->idealState.setMinActiveReplicas(0);
>>>   idealState.setStateModelFactoryName(HelixConstants.DEFAULT_STATE_MODEL_FACTORY);
>>>   idealState.setRebalanceDelay(100000);
>>>   idealState.setDelayRebalanceEnabled(true);
>>>   //idealState.setRebalancerClassName(DelayedAutoRebalancer.class.getName());
>>>
>>>   if (maxPartitionsPerInstance > 0 && maxPartitionsPerInstance < Integer.MAX_VALUE) {
>>>     idealState.setMaxPartitionsPerInstance(maxPartitionsPerInstance);
>>>   }
>>>   if (bucketSize > 0) {
>>>     idealState.setBucketSize(bucketSize);
>>>   }
>>>   addResource(clusterName, resourceName, idealState);
>>> }
>>>
>>>
>>>
>>> On Thu, Mar 1, 2018 at 11:20 AM, kishore g <g....@gmail.com> wrote:
>>>
>>>> We should have a recipe for delayed rebalancer
>>>>
>>>> On Thu, Mar 1, 2018 at 9:39 AM, Lei Xia <xi...@gmail.com> wrote:
>>>>
>>>>> Hi, Utsav
>>>>>
>>>>>   Sorry to get back to you late.  There is one more thing to config,
>>>>>
>>>>>     idealstate.setMinActiveReplicas(0);
>>>>>
>>>>>  This tell Helix the minimal replica it needs to maintain,  by default
>>>>> is set to 1, it means Helix needs to maintain at least 1 replica
>>>>> irregardless of delayed rebalancing.  For your case, you want to set it to
>>>>> 0.
>>>>>
>>>>>
>>>>> Lei
>>>>>
>>>>> On Mon, Feb 26, 2018 at 11:38 AM, Utsav Kanani <utsavjkanani@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Hi Lei,
>>>>>>
>>>>>> That did not work
>>>>>> Seeing the same behavior
>>>>>> Added the following method to ZKHelixAdmin Class
>>>>>>
>>>>>> public void enableClusterDelayMode(String clusterName) {
>>>>>>   ConfigAccessor configAccessor = new ConfigAccessor(_zkClient);
>>>>>>   ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
>>>>>>   clusterConfig.setDelayRebalaceEnabled(true);
>>>>>>   clusterConfig.setRebalanceDelayTime(100000);
>>>>>>   configAccessor.setClusterConfig(clusterName, clusterConfig);
>>>>>> }
>>>>>>
>>>>>> and calling it in the demo class
>>>>>>
>>>>>> HelixAdmin admin = new ZKHelixAdmin(zkAddress);
>>>>>> admin.addCluster(clusterName, true);
>>>>>> ---->((ZKHelixAdmin)admin).enableClusterDelayMode(clusterName);
>>>>>> StateModelConfigGenerator generator = new StateModelConfigGenerator();
>>>>>> admin.addStateModelDef(clusterName, "OnlineOffline",
>>>>>>     new StateModelDefinition(generator.generateConfigForOnlineOffline()));
>>>>>>
>>>>>> admin.addResource(clusterName, lockGroupName, numPartitions, "OnlineOffline",
>>>>>>     RebalanceMode.FULL_AUTO.toString());
>>>>>> admin.rebalance(clusterName, lockGroupName, 1);
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> STARTING localhost_12000
>>>>>> STARTING localhost_12001
>>>>>> STARTING localhost_12002
>>>>>> STARTED localhost_12000
>>>>>> STARTED localhost_12002
>>>>>> STARTED localhost_12001
>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>> localhost_12002 acquired lock:lock-group_3
>>>>>> localhost_12002 acquired lock:lock-group_9
>>>>>> localhost_12001 acquired lock:lock-group_2
>>>>>> localhost_12001 acquired lock:lock-group_5
>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>> localhost_12002 acquired lock:lock-group_6
>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>> localhost_12002 acquired lock:lock-group_10
>>>>>> localhost_12001 acquired lock:lock-group_8
>>>>>> localhost_12001 acquired lock:lock-group_1
>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>> lockName acquired By
>>>>>> ======================================
>>>>>> lock-group_0 localhost_12000
>>>>>> lock-group_1 localhost_12001
>>>>>> lock-group_10 localhost_12002
>>>>>> lock-group_11 localhost_12000
>>>>>> lock-group_2 localhost_12001
>>>>>> lock-group_3 localhost_12002
>>>>>> lock-group_4 localhost_12000
>>>>>> lock-group_5 localhost_12001
>>>>>> lock-group_6 localhost_12002
>>>>>> lock-group_7 localhost_12000
>>>>>> lock-group_8 localhost_12001
>>>>>> lock-group_9 localhost_12002
>>>>>> Stopping localhost_12000
>>>>>> localhost_12000Interrupted
>>>>>> localhost_12001 acquired lock:lock-group_11
>>>>>> localhost_12001 acquired lock:lock-group_0
>>>>>> localhost_12002 acquired lock:lock-group_7
>>>>>> localhost_12002 acquired lock:lock-group_4
>>>>>> lockName acquired By
>>>>>> ======================================
>>>>>> lock-group_0 localhost_12001
>>>>>> lock-group_1 localhost_12001
>>>>>> lock-group_10 localhost_12002
>>>>>> lock-group_11 localhost_12001
>>>>>> lock-group_2 localhost_12001
>>>>>> lock-group_3 localhost_12002
>>>>>> lock-group_4 localhost_12002
>>>>>> lock-group_5 localhost_12001
>>>>>> lock-group_6 localhost_12002
>>>>>> lock-group_7 localhost_12002
>>>>>> lock-group_8 localhost_12001
>>>>>> lock-group_9 localhost_12002
>>>>>> ===Starting localhost_12000
>>>>>> STARTING localhost_12000
>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>> STARTED localhost_12000
>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>> localhost_12001 releasing lock:lock-group_11
>>>>>> localhost_12001 releasing lock:lock-group_0
>>>>>> localhost_12002 releasing lock:lock-group_7
>>>>>> localhost_12002 releasing lock:lock-group_4
>>>>>> lockName acquired By
>>>>>> ======================================
>>>>>> lock-group_0 localhost_12000
>>>>>> lock-group_1 localhost_12001
>>>>>> lock-group_10 localhost_12002
>>>>>> lock-group_11 localhost_12000
>>>>>> lock-group_2 localhost_12001
>>>>>> lock-group_3 localhost_12002
>>>>>> lock-group_4 localhost_12000
>>>>>> lock-group_5 localhost_12001
>>>>>> lock-group_6 localhost_12002
>>>>>> lock-group_7 localhost_12000
>>>>>> lock-group_8 localhost_12001
>>>>>> lock-group_9 localhost_12002
>>>>>>
>>>>>>
>>>>>> On Sat, Feb 24, 2018 at 8:26 PM, Lei Xia <xi...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi, Utsav
>>>>>>>
>>>>>>>   Delay rebalancer by default is disabled in cluster level (this is
>>>>>>> to keep back-compatible somehow), you need to enable it in the
>>>>>>> clusterConfig, e.g
>>>>>>>
>>>>>>> ConfigAccessor configAccessor = new ConfigAccessor(zkClient);
>>>>>>> ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
>>>>>>> clusterConfig.setDelayRebalaceEnabled(enabled);
>>>>>>> configAccessor.setClusterConfig(clusterName, clusterConfig);
>>>>>>>
>>>>>>>
>>>>>>>   Could you please have a try and let me know whether it works or
>>>>>>> not?   Thanks
>>>>>>>
>>>>>>>
>>>>>>> Lei
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Feb 23, 2018 at 2:33 PM, Utsav Kanani <
>>>>>>> utsavjkanani@gmail.com> wrote:
>>>>>>>
>>>>>>>> I am trying to expand the Lockmanager example
>>>>>>>> http://helix.apache.org/0.6.2-incubating-docs/recipe
>>>>>>>> s/lock_manager.html to introduce delay
>>>>>>>>
>>>>>>>> tried doing something like this
>>>>>>>> IdealState state = admin.getResourceIdealState(clusterName,
>>>>>>>> lockGroupName);
>>>>>>>>      state.setRebalanceDelay(100000);
>>>>>>>>      state.setDelayRebalanceEnabled(true);
>>>>>>>>      state.setRebalancerClassName(DelayedAutoRebalancer.class.ge
>>>>>>>> tName());
>>>>>>>>      admin.setResourceIdealState(clusterName, lockGroupName,
>>>>>>>> state);
>>>>>>>>      admin.rebalance(clusterName, lockGroupName, 1);
>>>>>>>>
>>>>>>>> On killing a node there is immediate rebalancing which takes place.
>>>>>>>> I was hoping for a delay of 100 seconds before rebalancing but i am not
>>>>>>>> seeing that behavior
>>>>>>>>
>>>>>>>>
>>>>>>>> On Stopping localhost_12000 the locks are acquired immediately by
>>>>>>>> localhost_12001 and localhost_12002
>>>>>>>>
>>>>>>>> on STARTING localhost_12000 the rebalance is again immediate.
>>>>>>>>
>>>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>>>> STARTED localhost_12000
>>>>>>>> localhost_12001 releasing lock:lock-group_0
>>>>>>>> localhost_12001 releasing lock:lock-group_11
>>>>>>>> localhost_12002 releasing lock:lock-group_4
>>>>>>>> localhost_12002 releasing lock:lock-group_7
>>>>>>>>
>>>>>>>>
>>>>>>>> Here is the output
>>>>>>>> =========================================
>>>>>>>>
>>>>>>>> STARTING localhost_12000
>>>>>>>> STARTING localhost_12001
>>>>>>>> STARTING localhost_12002
>>>>>>>> STARTED localhost_12001
>>>>>>>> STARTED localhost_12002
>>>>>>>> STARTED localhost_12000
>>>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>>>> localhost_12002 acquired lock:lock-group_10
>>>>>>>> localhost_12002 acquired lock:lock-group_9
>>>>>>>> localhost_12002 acquired lock:lock-group_3
>>>>>>>> localhost_12001 acquired lock:lock-group_2
>>>>>>>> localhost_12001 acquired lock:lock-group_1
>>>>>>>> localhost_12001 acquired lock:lock-group_8
>>>>>>>> localhost_12002 acquired lock:lock-group_6
>>>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>>>> localhost_12001 acquired lock:lock-group_5
>>>>>>>> lockName acquired By
>>>>>>>> ======================================
>>>>>>>> lock-group_0 localhost_12000
>>>>>>>> lock-group_1 localhost_12001
>>>>>>>> lock-group_10 localhost_12002
>>>>>>>> lock-group_11 localhost_12000
>>>>>>>> lock-group_2 localhost_12001
>>>>>>>> lock-group_3 localhost_12002
>>>>>>>> lock-group_4 localhost_12000
>>>>>>>> lock-group_5 localhost_12001
>>>>>>>> lock-group_6 localhost_12002
>>>>>>>> lock-group_7 localhost_12000
>>>>>>>> lock-group_8 localhost_12001
>>>>>>>> lock-group_9 localhost_12002
>>>>>>>> Stopping localhost_12000
>>>>>>>> localhost_12000Interrupted
>>>>>>>> localhost_12002 acquired lock:lock-group_4
>>>>>>>> localhost_12001 acquired lock:lock-group_11
>>>>>>>> localhost_12002 acquired lock:lock-group_7
>>>>>>>> localhost_12001 acquired lock:lock-group_0
>>>>>>>> lockName acquired By
>>>>>>>> ======================================
>>>>>>>> lock-group_0 localhost_12001
>>>>>>>> lock-group_1 localhost_12001
>>>>>>>> lock-group_10 localhost_12002
>>>>>>>> lock-group_11 localhost_12001
>>>>>>>> lock-group_2 localhost_12001
>>>>>>>> lock-group_3 localhost_12002
>>>>>>>> lock-group_4 localhost_12002
>>>>>>>> lock-group_5 localhost_12001
>>>>>>>> lock-group_6 localhost_12002
>>>>>>>> lock-group_7 localhost_12002
>>>>>>>> lock-group_8 localhost_12001
>>>>>>>> lock-group_9 localhost_12002
>>>>>>>> ===Starting localhost_12000
>>>>>>>> STARTING localhost_12000
>>>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>>>> STARTED localhost_12000
>>>>>>>> localhost_12001 releasing lock:lock-group_0
>>>>>>>> localhost_12001 releasing lock:lock-group_11
>>>>>>>> localhost_12002 releasing lock:lock-group_4
>>>>>>>> localhost_12002 releasing lock:lock-group_7
>>>>>>>> lockName acquired By
>>>>>>>> ======================================
>>>>>>>> lock-group_0 localhost_12000
>>>>>>>> lock-group_1 localhost_12001
>>>>>>>> lock-group_10 localhost_12002
>>>>>>>> lock-group_11 localhost_12000
>>>>>>>> lock-group_2 localhost_12001
>>>>>>>> lock-group_3 localhost_12002
>>>>>>>> lock-group_4 localhost_12000
>>>>>>>> lock-group_5 localhost_12001
>>>>>>>> lock-group_6 localhost_12002
>>>>>>>> lock-group_7 localhost_12000
>>>>>>>> lock-group_8 localhost_12001
>>>>>>>> lock-group_9 localhost_12002
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Lei Xia
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Feb 24, 2018 at 8:26 PM, Lei Xia <xi...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi, Utsav
>>>>>>>
>>>>>>>   Delay rebalancer by default is disabled in cluster level (this is
>>>>>>> to keep back-compatible somehow), you need to enable it in the
>>>>>>> clusterConfig, e.g
>>>>>>>
>>>>>>> ConfigAccessor configAccessor = new ConfigAccessor(zkClient);
>>>>>>> ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
>>>>>>> clusterConfig.setDelayRebalaceEnabled(enabled);
>>>>>>> configAccessor.setClusterConfig(clusterName, clusterConfig);
>>>>>>>
>>>>>>>
>>>>>>>   Could you please have a try and let me know whether it works or
>>>>>>> not?   Thanks
>>>>>>>
>>>>>>>
>>>>>>> Lei
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Feb 23, 2018 at 2:33 PM, Utsav Kanani <
>>>>>>> utsavjkanani@gmail.com> wrote:
>>>>>>>
>>>>>>>> I am trying to expand the Lockmanager example
>>>>>>>> http://helix.apache.org/0.6.2-incubating-docs/recipes/lock_m
>>>>>>>> anager.html to introduce delay
>>>>>>>>
>>>>>>>> tried doing something like this
>>>>>>>> IdealState state = admin.getResourceIdealState(clusterName,
>>>>>>>> lockGroupName);
>>>>>>>>      state.setRebalanceDelay(100000);
>>>>>>>>      state.setDelayRebalanceEnabled(true);
>>>>>>>>      state.setRebalancerClassName(DelayedAutoRebalancer.class.ge
>>>>>>>> tName());
>>>>>>>>      admin.setResourceIdealState(clusterName, lockGroupName,
>>>>>>>> state);
>>>>>>>>      admin.rebalance(clusterName, lockGroupName, 1);
>>>>>>>>
>>>>>>>> On killing a node there is immediate rebalancing which takes place.
>>>>>>>> I was hoping for a delay of 100 seconds before rebalancing but i am not
>>>>>>>> seeing that behavior
>>>>>>>>
>>>>>>>>
>>>>>>>> On Stopping localhost_12000 the locks are acquired immediately by
>>>>>>>> localhost_12001 and localhost_12002
>>>>>>>>
>>>>>>>> on STARTING localhost_12000 the rebalance is again immediate.
>>>>>>>>
>>>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>>>> STARTED localhost_12000
>>>>>>>> localhost_12001 releasing lock:lock-group_0
>>>>>>>> localhost_12001 releasing lock:lock-group_11
>>>>>>>> localhost_12002 releasing lock:lock-group_4
>>>>>>>> localhost_12002 releasing lock:lock-group_7
>>>>>>>>
>>>>>>>>
>>>>>>>> Here is the output
>>>>>>>> =========================================
>>>>>>>>
>>>>>>>> STARTING localhost_12000
>>>>>>>> STARTING localhost_12001
>>>>>>>> STARTING localhost_12002
>>>>>>>> STARTED localhost_12001
>>>>>>>> STARTED localhost_12002
>>>>>>>> STARTED localhost_12000
>>>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>>>> localhost_12002 acquired lock:lock-group_10
>>>>>>>> localhost_12002 acquired lock:lock-group_9
>>>>>>>> localhost_12002 acquired lock:lock-group_3
>>>>>>>> localhost_12001 acquired lock:lock-group_2
>>>>>>>> localhost_12001 acquired lock:lock-group_1
>>>>>>>> localhost_12001 acquired lock:lock-group_8
>>>>>>>> localhost_12002 acquired lock:lock-group_6
>>>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>>>> localhost_12001 acquired lock:lock-group_5
>>>>>>>> lockName acquired By
>>>>>>>> ======================================
>>>>>>>> lock-group_0 localhost_12000
>>>>>>>> lock-group_1 localhost_12001
>>>>>>>> lock-group_10 localhost_12002
>>>>>>>> lock-group_11 localhost_12000
>>>>>>>> lock-group_2 localhost_12001
>>>>>>>> lock-group_3 localhost_12002
>>>>>>>> lock-group_4 localhost_12000
>>>>>>>> lock-group_5 localhost_12001
>>>>>>>> lock-group_6 localhost_12002
>>>>>>>> lock-group_7 localhost_12000
>>>>>>>> lock-group_8 localhost_12001
>>>>>>>> lock-group_9 localhost_12002
>>>>>>>> Stopping localhost_12000
>>>>>>>> localhost_12000Interrupted
>>>>>>>> localhost_12002 acquired lock:lock-group_4
>>>>>>>> localhost_12001 acquired lock:lock-group_11
>>>>>>>> localhost_12002 acquired lock:lock-group_7
>>>>>>>> localhost_12001 acquired lock:lock-group_0
>>>>>>>> lockName acquired By
>>>>>>>> ======================================
>>>>>>>> lock-group_0 localhost_12001
>>>>>>>> lock-group_1 localhost_12001
>>>>>>>> lock-group_10 localhost_12002
>>>>>>>> lock-group_11 localhost_12001
>>>>>>>> lock-group_2 localhost_12001
>>>>>>>> lock-group_3 localhost_12002
>>>>>>>> lock-group_4 localhost_12002
>>>>>>>> lock-group_5 localhost_12001
>>>>>>>> lock-group_6 localhost_12002
>>>>>>>> lock-group_7 localhost_12002
>>>>>>>> lock-group_8 localhost_12001
>>>>>>>> lock-group_9 localhost_12002
>>>>>>>> ===Starting localhost_12000
>>>>>>>> STARTING localhost_12000
>>>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>>>> STARTED localhost_12000
>>>>>>>> localhost_12001 releasing lock:lock-group_0
>>>>>>>> localhost_12001 releasing lock:lock-group_11
>>>>>>>> localhost_12002 releasing lock:lock-group_4
>>>>>>>> localhost_12002 releasing lock:lock-group_7
>>>>>>>> lockName acquired By
>>>>>>>> ======================================
>>>>>>>> lock-group_0 localhost_12000
>>>>>>>> lock-group_1 localhost_12001
>>>>>>>> lock-group_10 localhost_12002
>>>>>>>> lock-group_11 localhost_12000
>>>>>>>> lock-group_2 localhost_12001
>>>>>>>> lock-group_3 localhost_12002
>>>>>>>> lock-group_4 localhost_12000
>>>>>>>> lock-group_5 localhost_12001
>>>>>>>> lock-group_6 localhost_12002
>>>>>>>> lock-group_7 localhost_12000
>>>>>>>> lock-group_8 localhost_12001
>>>>>>>> lock-group_9 localhost_12002
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Lei Xia
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Lei Xia
>>>>>
>>>>
>>>>
>>>
>>
>

Re: Unable to get Rebalance Delay to work using the distributed lock manager recipe

Posted by Utsav Kanani <ut...@gmail.com>.

Thanks a ton for your help with this

Had another quick question

How can i configure such that there is no rebalancing and the same Process
always gets the same lock.
Is it with the SEMI_AUTO mode

On Thu, Mar 8, 2018 at 6:11 PM, Utsav Kanani <ut...@gmail.com> wrote:

> Ignore what is said it works with
>
> idealState.setRebalancerClassName(DelayedAutoRebalancer.class.getName());
>
>
>
> On Thu, Mar 8, 2018 at 6:08 PM, Utsav Kanani <ut...@gmail.com>
> wrote:
>
>> sorry for the late response
>> hey guys this does not help either
>> do you guys want me to send you my code files?
>>
>> this is the code change i made in the ZKHelixAdmin class
>> tried with both
>>
>> idealState.setRebalancerClassName(DelayedAutoRebalancer.class.getName());
>>
>> and withoug
>>
>> @Override
>> public void addResource(String clusterName, String resourceName, int partitions,
>>     String stateModelRef, String rebalancerMode, String rebalanceStrategy, int bucketSize,
>>     int maxPartitionsPerInstance) {
>>   if (!ZKUtil.isClusterSetup(clusterName, _zkClient)) {
>>     throw new HelixException("cluster " + clusterName + " is not setup yet");
>>   }
>>
>>   IdealState idealState = new IdealState(resourceName);
>>   idealState.setNumPartitions(partitions);
>>   idealState.setStateModelDefRef(stateModelRef);
>>   RebalanceMode mode =
>>       idealState.rebalanceModeFromString(rebalancerMode, RebalanceMode.SEMI_AUTO);
>>   idealState.setRebalanceMode(mode);
>>   idealState.setRebalanceStrategy(rebalanceStrategy);
>>   -------->idealState.setMinActiveReplicas(0);
>>   idealState.setStateModelFactoryName(HelixConstants.DEFAULT_STATE_MODEL_FACTORY);
>>   idealState.setRebalanceDelay(100000);
>>   idealState.setDelayRebalanceEnabled(true);
>>   //idealState.setRebalancerClassName(DelayedAutoRebalancer.class.getName());
>>
>>   if (maxPartitionsPerInstance > 0 && maxPartitionsPerInstance < Integer.MAX_VALUE) {
>>     idealState.setMaxPartitionsPerInstance(maxPartitionsPerInstance);
>>   }
>>   if (bucketSize > 0) {
>>     idealState.setBucketSize(bucketSize);
>>   }
>>   addResource(clusterName, resourceName, idealState);
>> }
>>
>>
>>
>> On Thu, Mar 1, 2018 at 11:20 AM, kishore g <g....@gmail.com> wrote:
>>
>>> We should have a recipe for delayed rebalancer
>>>
>>> On Thu, Mar 1, 2018 at 9:39 AM, Lei Xia <xi...@gmail.com> wrote:
>>>
>>>> Hi, Utsav
>>>>
>>>>   Sorry to get back to you late.  There is one more thing to config,
>>>>
>>>>     idealstate.setMinActiveReplicas(0);
>>>>
>>>>  This tell Helix the minimal replica it needs to maintain,  by default
>>>> is set to 1, it means Helix needs to maintain at least 1 replica
>>>> irregardless of delayed rebalancing.  For your case, you want to set it to
>>>> 0.
>>>>
>>>>
>>>> Lei
>>>>
>>>> On Mon, Feb 26, 2018 at 11:38 AM, Utsav Kanani <ut...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Lei,
>>>>>
>>>>> That did not work
>>>>> Seeing the same behavior
>>>>> Added the following method to ZKHelixAdmin Class
>>>>>
>>>>> public void enableClusterDelayMode(String clusterName) {
>>>>>   ConfigAccessor configAccessor = new ConfigAccessor(_zkClient);
>>>>>   ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
>>>>>   clusterConfig.setDelayRebalaceEnabled(true);
>>>>>   clusterConfig.setRebalanceDelayTime(100000);
>>>>>   configAccessor.setClusterConfig(clusterName, clusterConfig);
>>>>> }
>>>>>
>>>>> and calling it in the demo class
>>>>>
>>>>> HelixAdmin admin = new ZKHelixAdmin(zkAddress);
>>>>> admin.addCluster(clusterName, true);
>>>>> ---->((ZKHelixAdmin)admin).enableClusterDelayMode(clusterName);
>>>>> StateModelConfigGenerator generator = new StateModelConfigGenerator();
>>>>> admin.addStateModelDef(clusterName, "OnlineOffline",
>>>>>     new StateModelDefinition(generator.generateConfigForOnlineOffline()));
>>>>>
>>>>> admin.addResource(clusterName, lockGroupName, numPartitions, "OnlineOffline",
>>>>>     RebalanceMode.FULL_AUTO.toString());
>>>>> admin.rebalance(clusterName, lockGroupName, 1);
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> STARTING localhost_12000
>>>>> STARTING localhost_12001
>>>>> STARTING localhost_12002
>>>>> STARTED localhost_12000
>>>>> STARTED localhost_12002
>>>>> STARTED localhost_12001
>>>>> localhost_12000 acquired lock:lock-group_0
>>>>> localhost_12002 acquired lock:lock-group_3
>>>>> localhost_12002 acquired lock:lock-group_9
>>>>> localhost_12001 acquired lock:lock-group_2
>>>>> localhost_12001 acquired lock:lock-group_5
>>>>> localhost_12000 acquired lock:lock-group_11
>>>>> localhost_12002 acquired lock:lock-group_6
>>>>> localhost_12000 acquired lock:lock-group_7
>>>>> localhost_12002 acquired lock:lock-group_10
>>>>> localhost_12001 acquired lock:lock-group_8
>>>>> localhost_12001 acquired lock:lock-group_1
>>>>> localhost_12000 acquired lock:lock-group_4
>>>>> lockName acquired By
>>>>> ======================================
>>>>> lock-group_0 localhost_12000
>>>>> lock-group_1 localhost_12001
>>>>> lock-group_10 localhost_12002
>>>>> lock-group_11 localhost_12000
>>>>> lock-group_2 localhost_12001
>>>>> lock-group_3 localhost_12002
>>>>> lock-group_4 localhost_12000
>>>>> lock-group_5 localhost_12001
>>>>> lock-group_6 localhost_12002
>>>>> lock-group_7 localhost_12000
>>>>> lock-group_8 localhost_12001
>>>>> lock-group_9 localhost_12002
>>>>> Stopping localhost_12000
>>>>> localhost_12000Interrupted
>>>>> localhost_12001 acquired lock:lock-group_11
>>>>> localhost_12001 acquired lock:lock-group_0
>>>>> localhost_12002 acquired lock:lock-group_7
>>>>> localhost_12002 acquired lock:lock-group_4
>>>>> lockName acquired By
>>>>> ======================================
>>>>> lock-group_0 localhost_12001
>>>>> lock-group_1 localhost_12001
>>>>> lock-group_10 localhost_12002
>>>>> lock-group_11 localhost_12001
>>>>> lock-group_2 localhost_12001
>>>>> lock-group_3 localhost_12002
>>>>> lock-group_4 localhost_12002
>>>>> lock-group_5 localhost_12001
>>>>> lock-group_6 localhost_12002
>>>>> lock-group_7 localhost_12002
>>>>> lock-group_8 localhost_12001
>>>>> lock-group_9 localhost_12002
>>>>> ===Starting localhost_12000
>>>>> STARTING localhost_12000
>>>>> localhost_12000 acquired lock:lock-group_11
>>>>> localhost_12000 acquired lock:lock-group_0
>>>>> STARTED localhost_12000
>>>>> localhost_12000 acquired lock:lock-group_7
>>>>> localhost_12000 acquired lock:lock-group_4
>>>>> localhost_12001 releasing lock:lock-group_11
>>>>> localhost_12001 releasing lock:lock-group_0
>>>>> localhost_12002 releasing lock:lock-group_7
>>>>> localhost_12002 releasing lock:lock-group_4
>>>>> lockName acquired By
>>>>> ======================================
>>>>> lock-group_0 localhost_12000
>>>>> lock-group_1 localhost_12001
>>>>> lock-group_10 localhost_12002
>>>>> lock-group_11 localhost_12000
>>>>> lock-group_2 localhost_12001
>>>>> lock-group_3 localhost_12002
>>>>> lock-group_4 localhost_12000
>>>>> lock-group_5 localhost_12001
>>>>> lock-group_6 localhost_12002
>>>>> lock-group_7 localhost_12000
>>>>> lock-group_8 localhost_12001
>>>>> lock-group_9 localhost_12002
>>>>>
>>>>>
>>>>> On Sat, Feb 24, 2018 at 8:26 PM, Lei Xia <xi...@gmail.com> wrote:
>>>>>
>>>>>> Hi, Utsav
>>>>>>
>>>>>>   Delay rebalancer by default is disabled in cluster level (this is
>>>>>> to keep back-compatible somehow), you need to enable it in the
>>>>>> clusterConfig, e.g
>>>>>>
>>>>>> ConfigAccessor configAccessor = new ConfigAccessor(zkClient);
>>>>>> ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
>>>>>> clusterConfig.setDelayRebalaceEnabled(enabled);
>>>>>> configAccessor.setClusterConfig(clusterName, clusterConfig);
>>>>>>
>>>>>>
>>>>>>   Could you please have a try and let me know whether it works or
>>>>>> not?   Thanks
>>>>>>
>>>>>>
>>>>>> Lei
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 23, 2018 at 2:33 PM, Utsav Kanani <utsavjkanani@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> I am trying to expand the Lockmanager example http://helix.apache.or
>>>>>>> g/0.6.2-incubating-docs/recipes/lock_manager.html to introduce delay
>>>>>>>
>>>>>>> tried doing something like this
>>>>>>> IdealState state = admin.getResourceIdealState(clusterName,
>>>>>>> lockGroupName);
>>>>>>>      state.setRebalanceDelay(100000);
>>>>>>>      state.setDelayRebalanceEnabled(true);
>>>>>>>      state.setRebalancerClassName(DelayedAutoRebalancer.class.ge
>>>>>>> tName());
>>>>>>>      admin.setResourceIdealState(clusterName, lockGroupName, state);
>>>>>>>      admin.rebalance(clusterName, lockGroupName, 1);
>>>>>>>
>>>>>>> On killing a node there is immediate rebalancing which takes place.
>>>>>>> I was hoping for a delay of 100 seconds before rebalancing but i am not
>>>>>>> seeing that behavior
>>>>>>>
>>>>>>>
>>>>>>> On Stopping localhost_12000 the locks are acquired immediately by
>>>>>>> localhost_12001 and localhost_12002
>>>>>>>
>>>>>>> on STARTING localhost_12000 the rebalance is again immediate.
>>>>>>>
>>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>>> STARTED localhost_12000
>>>>>>> localhost_12001 releasing lock:lock-group_0
>>>>>>> localhost_12001 releasing lock:lock-group_11
>>>>>>> localhost_12002 releasing lock:lock-group_4
>>>>>>> localhost_12002 releasing lock:lock-group_7
>>>>>>>
>>>>>>>
>>>>>>> Here is the output
>>>>>>> =========================================
>>>>>>>
>>>>>>> STARTING localhost_12000
>>>>>>> STARTING localhost_12001
>>>>>>> STARTING localhost_12002
>>>>>>> STARTED localhost_12001
>>>>>>> STARTED localhost_12002
>>>>>>> STARTED localhost_12000
>>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>>> localhost_12002 acquired lock:lock-group_10
>>>>>>> localhost_12002 acquired lock:lock-group_9
>>>>>>> localhost_12002 acquired lock:lock-group_3
>>>>>>> localhost_12001 acquired lock:lock-group_2
>>>>>>> localhost_12001 acquired lock:lock-group_1
>>>>>>> localhost_12001 acquired lock:lock-group_8
>>>>>>> localhost_12002 acquired lock:lock-group_6
>>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>>> localhost_12001 acquired lock:lock-group_5
>>>>>>> lockName acquired By
>>>>>>> ======================================
>>>>>>> lock-group_0 localhost_12000
>>>>>>> lock-group_1 localhost_12001
>>>>>>> lock-group_10 localhost_12002
>>>>>>> lock-group_11 localhost_12000
>>>>>>> lock-group_2 localhost_12001
>>>>>>> lock-group_3 localhost_12002
>>>>>>> lock-group_4 localhost_12000
>>>>>>> lock-group_5 localhost_12001
>>>>>>> lock-group_6 localhost_12002
>>>>>>> lock-group_7 localhost_12000
>>>>>>> lock-group_8 localhost_12001
>>>>>>> lock-group_9 localhost_12002
>>>>>>> Stopping localhost_12000
>>>>>>> localhost_12000Interrupted
>>>>>>> localhost_12002 acquired lock:lock-group_4
>>>>>>> localhost_12001 acquired lock:lock-group_11
>>>>>>> localhost_12002 acquired lock:lock-group_7
>>>>>>> localhost_12001 acquired lock:lock-group_0
>>>>>>> lockName acquired By
>>>>>>> ======================================
>>>>>>> lock-group_0 localhost_12001
>>>>>>> lock-group_1 localhost_12001
>>>>>>> lock-group_10 localhost_12002
>>>>>>> lock-group_11 localhost_12001
>>>>>>> lock-group_2 localhost_12001
>>>>>>> lock-group_3 localhost_12002
>>>>>>> lock-group_4 localhost_12002
>>>>>>> lock-group_5 localhost_12001
>>>>>>> lock-group_6 localhost_12002
>>>>>>> lock-group_7 localhost_12002
>>>>>>> lock-group_8 localhost_12001
>>>>>>> lock-group_9 localhost_12002
>>>>>>> ===Starting localhost_12000
>>>>>>> STARTING localhost_12000
>>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>>> STARTED localhost_12000
>>>>>>> localhost_12001 releasing lock:lock-group_0
>>>>>>> localhost_12001 releasing lock:lock-group_11
>>>>>>> localhost_12002 releasing lock:lock-group_4
>>>>>>> localhost_12002 releasing lock:lock-group_7
>>>>>>> lockName acquired By
>>>>>>> ======================================
>>>>>>> lock-group_0 localhost_12000
>>>>>>> lock-group_1 localhost_12001
>>>>>>> lock-group_10 localhost_12002
>>>>>>> lock-group_11 localhost_12000
>>>>>>> lock-group_2 localhost_12001
>>>>>>> lock-group_3 localhost_12002
>>>>>>> lock-group_4 localhost_12000
>>>>>>> lock-group_5 localhost_12001
>>>>>>> lock-group_6 localhost_12002
>>>>>>> lock-group_7 localhost_12000
>>>>>>> lock-group_8 localhost_12001
>>>>>>> lock-group_9 localhost_12002
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Lei Xia
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Feb 24, 2018 at 8:26 PM, Lei Xia <xi...@gmail.com> wrote:
>>>>>
>>>>>> Hi, Utsav
>>>>>>
>>>>>>   Delay rebalancer by default is disabled in cluster level (this is
>>>>>> to keep back-compatible somehow), you need to enable it in the
>>>>>> clusterConfig, e.g
>>>>>>
>>>>>> ConfigAccessor configAccessor = new ConfigAccessor(zkClient);
>>>>>> ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
>>>>>> clusterConfig.setDelayRebalaceEnabled(enabled);
>>>>>> configAccessor.setClusterConfig(clusterName, clusterConfig);
>>>>>>
>>>>>>
>>>>>>   Could you please have a try and let me know whether it works or
>>>>>> not?   Thanks
>>>>>>
>>>>>>
>>>>>> Lei
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 23, 2018 at 2:33 PM, Utsav Kanani <utsavjkanani@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> I am trying to expand the Lockmanager example
>>>>>>> http://helix.apache.org/0.6.2-incubating-docs/recipes/lock_m
>>>>>>> anager.html to introduce delay
>>>>>>>
>>>>>>> tried doing something like this
>>>>>>> IdealState state = admin.getResourceIdealState(clusterName,
>>>>>>> lockGroupName);
>>>>>>>      state.setRebalanceDelay(100000);
>>>>>>>      state.setDelayRebalanceEnabled(true);
>>>>>>>      state.setRebalancerClassName(DelayedAutoRebalancer.class.ge
>>>>>>> tName());
>>>>>>>      admin.setResourceIdealState(clusterName, lockGroupName, state);
>>>>>>>      admin.rebalance(clusterName, lockGroupName, 1);
>>>>>>>
>>>>>>> On killing a node there is immediate rebalancing which takes place.
>>>>>>> I was hoping for a delay of 100 seconds before rebalancing but i am not
>>>>>>> seeing that behavior
>>>>>>>
>>>>>>>
>>>>>>> On Stopping localhost_12000 the locks are acquired immediately by
>>>>>>> localhost_12001 and localhost_12002
>>>>>>>
>>>>>>> on STARTING localhost_12000 the rebalance is again immediate.
>>>>>>>
>>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>>> STARTED localhost_12000
>>>>>>> localhost_12001 releasing lock:lock-group_0
>>>>>>> localhost_12001 releasing lock:lock-group_11
>>>>>>> localhost_12002 releasing lock:lock-group_4
>>>>>>> localhost_12002 releasing lock:lock-group_7
>>>>>>>
>>>>>>>
>>>>>>> Here is the output
>>>>>>> =========================================
>>>>>>>
>>>>>>> STARTING localhost_12000
>>>>>>> STARTING localhost_12001
>>>>>>> STARTING localhost_12002
>>>>>>> STARTED localhost_12001
>>>>>>> STARTED localhost_12002
>>>>>>> STARTED localhost_12000
>>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>>> localhost_12002 acquired lock:lock-group_10
>>>>>>> localhost_12002 acquired lock:lock-group_9
>>>>>>> localhost_12002 acquired lock:lock-group_3
>>>>>>> localhost_12001 acquired lock:lock-group_2
>>>>>>> localhost_12001 acquired lock:lock-group_1
>>>>>>> localhost_12001 acquired lock:lock-group_8
>>>>>>> localhost_12002 acquired lock:lock-group_6
>>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>>> localhost_12001 acquired lock:lock-group_5
>>>>>>> lockName acquired By
>>>>>>> ======================================
>>>>>>> lock-group_0 localhost_12000
>>>>>>> lock-group_1 localhost_12001
>>>>>>> lock-group_10 localhost_12002
>>>>>>> lock-group_11 localhost_12000
>>>>>>> lock-group_2 localhost_12001
>>>>>>> lock-group_3 localhost_12002
>>>>>>> lock-group_4 localhost_12000
>>>>>>> lock-group_5 localhost_12001
>>>>>>> lock-group_6 localhost_12002
>>>>>>> lock-group_7 localhost_12000
>>>>>>> lock-group_8 localhost_12001
>>>>>>> lock-group_9 localhost_12002
>>>>>>> Stopping localhost_12000
>>>>>>> localhost_12000Interrupted
>>>>>>> localhost_12002 acquired lock:lock-group_4
>>>>>>> localhost_12001 acquired lock:lock-group_11
>>>>>>> localhost_12002 acquired lock:lock-group_7
>>>>>>> localhost_12001 acquired lock:lock-group_0
>>>>>>> lockName acquired By
>>>>>>> ======================================
>>>>>>> lock-group_0 localhost_12001
>>>>>>> lock-group_1 localhost_12001
>>>>>>> lock-group_10 localhost_12002
>>>>>>> lock-group_11 localhost_12001
>>>>>>> lock-group_2 localhost_12001
>>>>>>> lock-group_3 localhost_12002
>>>>>>> lock-group_4 localhost_12002
>>>>>>> lock-group_5 localhost_12001
>>>>>>> lock-group_6 localhost_12002
>>>>>>> lock-group_7 localhost_12002
>>>>>>> lock-group_8 localhost_12001
>>>>>>> lock-group_9 localhost_12002
>>>>>>> ===Starting localhost_12000
>>>>>>> STARTING localhost_12000
>>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>>> STARTED localhost_12000
>>>>>>> localhost_12001 releasing lock:lock-group_0
>>>>>>> localhost_12001 releasing lock:lock-group_11
>>>>>>> localhost_12002 releasing lock:lock-group_4
>>>>>>> localhost_12002 releasing lock:lock-group_7
>>>>>>> lockName acquired By
>>>>>>> ======================================
>>>>>>> lock-group_0 localhost_12000
>>>>>>> lock-group_1 localhost_12001
>>>>>>> lock-group_10 localhost_12002
>>>>>>> lock-group_11 localhost_12000
>>>>>>> lock-group_2 localhost_12001
>>>>>>> lock-group_3 localhost_12002
>>>>>>> lock-group_4 localhost_12000
>>>>>>> lock-group_5 localhost_12001
>>>>>>> lock-group_6 localhost_12002
>>>>>>> lock-group_7 localhost_12000
>>>>>>> lock-group_8 localhost_12001
>>>>>>> lock-group_9 localhost_12002
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Lei Xia
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Lei Xia
>>>>
>>>
>>>
>>
>

Re: Unable to get Rebalance Delay to work using the distributed lock manager recipe

Posted by Utsav Kanani <ut...@gmail.com>.

Ignore what is said it works with

idealState.setRebalancerClassName(DelayedAutoRebalancer.class.getName());



On Thu, Mar 8, 2018 at 6:08 PM, Utsav Kanani <ut...@gmail.com> wrote:

> sorry for the late response
> hey guys this does not help either
> do you guys want me to send you my code files?
>
> this is the code change i made in the ZKHelixAdmin class
> tried with both
>
> idealState.setRebalancerClassName(DelayedAutoRebalancer.class.getName());
>
> and withoug
>
> @Override
> public void addResource(String clusterName, String resourceName, int partitions,
>     String stateModelRef, String rebalancerMode, String rebalanceStrategy, int bucketSize,
>     int maxPartitionsPerInstance) {
>   if (!ZKUtil.isClusterSetup(clusterName, _zkClient)) {
>     throw new HelixException("cluster " + clusterName + " is not setup yet");
>   }
>
>   IdealState idealState = new IdealState(resourceName);
>   idealState.setNumPartitions(partitions);
>   idealState.setStateModelDefRef(stateModelRef);
>   RebalanceMode mode =
>       idealState.rebalanceModeFromString(rebalancerMode, RebalanceMode.SEMI_AUTO);
>   idealState.setRebalanceMode(mode);
>   idealState.setRebalanceStrategy(rebalanceStrategy);
>   -------->idealState.setMinActiveReplicas(0);
>   idealState.setStateModelFactoryName(HelixConstants.DEFAULT_STATE_MODEL_FACTORY);
>   idealState.setRebalanceDelay(100000);
>   idealState.setDelayRebalanceEnabled(true);
>   //idealState.setRebalancerClassName(DelayedAutoRebalancer.class.getName());
>
>   if (maxPartitionsPerInstance > 0 && maxPartitionsPerInstance < Integer.MAX_VALUE) {
>     idealState.setMaxPartitionsPerInstance(maxPartitionsPerInstance);
>   }
>   if (bucketSize > 0) {
>     idealState.setBucketSize(bucketSize);
>   }
>   addResource(clusterName, resourceName, idealState);
> }
>
>
>
> On Thu, Mar 1, 2018 at 11:20 AM, kishore g <g....@gmail.com> wrote:
>
>> We should have a recipe for delayed rebalancer
>>
>> On Thu, Mar 1, 2018 at 9:39 AM, Lei Xia <xi...@gmail.com> wrote:
>>
>>> Hi, Utsav
>>>
>>>   Sorry to get back to you late.  There is one more thing to config,
>>>
>>>     idealstate.setMinActiveReplicas(0);
>>>
>>>  This tell Helix the minimal replica it needs to maintain,  by default
>>> is set to 1, it means Helix needs to maintain at least 1 replica
>>> irregardless of delayed rebalancing.  For your case, you want to set it to
>>> 0.
>>>
>>>
>>> Lei
>>>
>>> On Mon, Feb 26, 2018 at 11:38 AM, Utsav Kanani <ut...@gmail.com>
>>> wrote:
>>>
>>>> Hi Lei,
>>>>
>>>> That did not work
>>>> Seeing the same behavior
>>>> Added the following method to ZKHelixAdmin Class
>>>>
>>>> public void enableClusterDelayMode(String clusterName) {
>>>>   ConfigAccessor configAccessor = new ConfigAccessor(_zkClient);
>>>>   ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
>>>>   clusterConfig.setDelayRebalaceEnabled(true);
>>>>   clusterConfig.setRebalanceDelayTime(100000);
>>>>   configAccessor.setClusterConfig(clusterName, clusterConfig);
>>>> }
>>>>
>>>> and calling it in the demo class
>>>>
>>>> HelixAdmin admin = new ZKHelixAdmin(zkAddress);
>>>> admin.addCluster(clusterName, true);
>>>> ---->((ZKHelixAdmin)admin).enableClusterDelayMode(clusterName);
>>>> StateModelConfigGenerator generator = new StateModelConfigGenerator();
>>>> admin.addStateModelDef(clusterName, "OnlineOffline",
>>>>     new StateModelDefinition(generator.generateConfigForOnlineOffline()));
>>>>
>>>> admin.addResource(clusterName, lockGroupName, numPartitions, "OnlineOffline",
>>>>     RebalanceMode.FULL_AUTO.toString());
>>>> admin.rebalance(clusterName, lockGroupName, 1);
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> STARTING localhost_12000
>>>> STARTING localhost_12001
>>>> STARTING localhost_12002
>>>> STARTED localhost_12000
>>>> STARTED localhost_12002
>>>> STARTED localhost_12001
>>>> localhost_12000 acquired lock:lock-group_0
>>>> localhost_12002 acquired lock:lock-group_3
>>>> localhost_12002 acquired lock:lock-group_9
>>>> localhost_12001 acquired lock:lock-group_2
>>>> localhost_12001 acquired lock:lock-group_5
>>>> localhost_12000 acquired lock:lock-group_11
>>>> localhost_12002 acquired lock:lock-group_6
>>>> localhost_12000 acquired lock:lock-group_7
>>>> localhost_12002 acquired lock:lock-group_10
>>>> localhost_12001 acquired lock:lock-group_8
>>>> localhost_12001 acquired lock:lock-group_1
>>>> localhost_12000 acquired lock:lock-group_4
>>>> lockName acquired By
>>>> ======================================
>>>> lock-group_0 localhost_12000
>>>> lock-group_1 localhost_12001
>>>> lock-group_10 localhost_12002
>>>> lock-group_11 localhost_12000
>>>> lock-group_2 localhost_12001
>>>> lock-group_3 localhost_12002
>>>> lock-group_4 localhost_12000
>>>> lock-group_5 localhost_12001
>>>> lock-group_6 localhost_12002
>>>> lock-group_7 localhost_12000
>>>> lock-group_8 localhost_12001
>>>> lock-group_9 localhost_12002
>>>> Stopping localhost_12000
>>>> localhost_12000Interrupted
>>>> localhost_12001 acquired lock:lock-group_11
>>>> localhost_12001 acquired lock:lock-group_0
>>>> localhost_12002 acquired lock:lock-group_7
>>>> localhost_12002 acquired lock:lock-group_4
>>>> lockName acquired By
>>>> ======================================
>>>> lock-group_0 localhost_12001
>>>> lock-group_1 localhost_12001
>>>> lock-group_10 localhost_12002
>>>> lock-group_11 localhost_12001
>>>> lock-group_2 localhost_12001
>>>> lock-group_3 localhost_12002
>>>> lock-group_4 localhost_12002
>>>> lock-group_5 localhost_12001
>>>> lock-group_6 localhost_12002
>>>> lock-group_7 localhost_12002
>>>> lock-group_8 localhost_12001
>>>> lock-group_9 localhost_12002
>>>> ===Starting localhost_12000
>>>> STARTING localhost_12000
>>>> localhost_12000 acquired lock:lock-group_11
>>>> localhost_12000 acquired lock:lock-group_0
>>>> STARTED localhost_12000
>>>> localhost_12000 acquired lock:lock-group_7
>>>> localhost_12000 acquired lock:lock-group_4
>>>> localhost_12001 releasing lock:lock-group_11
>>>> localhost_12001 releasing lock:lock-group_0
>>>> localhost_12002 releasing lock:lock-group_7
>>>> localhost_12002 releasing lock:lock-group_4
>>>> lockName acquired By
>>>> ======================================
>>>> lock-group_0 localhost_12000
>>>> lock-group_1 localhost_12001
>>>> lock-group_10 localhost_12002
>>>> lock-group_11 localhost_12000
>>>> lock-group_2 localhost_12001
>>>> lock-group_3 localhost_12002
>>>> lock-group_4 localhost_12000
>>>> lock-group_5 localhost_12001
>>>> lock-group_6 localhost_12002
>>>> lock-group_7 localhost_12000
>>>> lock-group_8 localhost_12001
>>>> lock-group_9 localhost_12002
>>>>
>>>>
>>>> On Sat, Feb 24, 2018 at 8:26 PM, Lei Xia <xi...@gmail.com> wrote:
>>>>
>>>>> Hi, Utsav
>>>>>
>>>>>   Delay rebalancer by default is disabled in cluster level (this is to
>>>>> keep back-compatible somehow), you need to enable it in the clusterConfig,
>>>>> e.g
>>>>>
>>>>> ConfigAccessor configAccessor = new ConfigAccessor(zkClient);
>>>>> ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
>>>>> clusterConfig.setDelayRebalaceEnabled(enabled);
>>>>> configAccessor.setClusterConfig(clusterName, clusterConfig);
>>>>>
>>>>>
>>>>>   Could you please have a try and let me know whether it works or
>>>>> not?   Thanks
>>>>>
>>>>>
>>>>> Lei
>>>>>
>>>>>
>>>>> On Fri, Feb 23, 2018 at 2:33 PM, Utsav Kanani <ut...@gmail.com>
>>>>>  wrote:
>>>>>
>>>>>> I am trying to expand the Lockmanager example http://helix.apache.or
>>>>>> g/0.6.2-incubating-docs/recipes/lock_manager.html to introduce delay
>>>>>>
>>>>>> tried doing something like this
>>>>>> IdealState state = admin.getResourceIdealState(clusterName,
>>>>>> lockGroupName);
>>>>>>      state.setRebalanceDelay(100000);
>>>>>>      state.setDelayRebalanceEnabled(true);
>>>>>>      state.setRebalancerClassName(DelayedAutoRebalancer.class.ge
>>>>>> tName());
>>>>>>      admin.setResourceIdealState(clusterName, lockGroupName, state);
>>>>>>      admin.rebalance(clusterName, lockGroupName, 1);
>>>>>>
>>>>>> On killing a node there is immediate rebalancing which takes place. I
>>>>>> was hoping for a delay of 100 seconds before rebalancing but i am not
>>>>>> seeing that behavior
>>>>>>
>>>>>>
>>>>>> On Stopping localhost_12000 the locks are acquired immediately by
>>>>>> localhost_12001 and localhost_12002
>>>>>>
>>>>>> on STARTING localhost_12000 the rebalance is again immediate.
>>>>>>
>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>> STARTED localhost_12000
>>>>>> localhost_12001 releasing lock:lock-group_0
>>>>>> localhost_12001 releasing lock:lock-group_11
>>>>>> localhost_12002 releasing lock:lock-group_4
>>>>>> localhost_12002 releasing lock:lock-group_7
>>>>>>
>>>>>>
>>>>>> Here is the output
>>>>>> =========================================
>>>>>>
>>>>>> STARTING localhost_12000
>>>>>> STARTING localhost_12001
>>>>>> STARTING localhost_12002
>>>>>> STARTED localhost_12001
>>>>>> STARTED localhost_12002
>>>>>> STARTED localhost_12000
>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>> localhost_12002 acquired lock:lock-group_10
>>>>>> localhost_12002 acquired lock:lock-group_9
>>>>>> localhost_12002 acquired lock:lock-group_3
>>>>>> localhost_12001 acquired lock:lock-group_2
>>>>>> localhost_12001 acquired lock:lock-group_1
>>>>>> localhost_12001 acquired lock:lock-group_8
>>>>>> localhost_12002 acquired lock:lock-group_6
>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>> localhost_12001 acquired lock:lock-group_5
>>>>>> lockName acquired By
>>>>>> ======================================
>>>>>> lock-group_0 localhost_12000
>>>>>> lock-group_1 localhost_12001
>>>>>> lock-group_10 localhost_12002
>>>>>> lock-group_11 localhost_12000
>>>>>> lock-group_2 localhost_12001
>>>>>> lock-group_3 localhost_12002
>>>>>> lock-group_4 localhost_12000
>>>>>> lock-group_5 localhost_12001
>>>>>> lock-group_6 localhost_12002
>>>>>> lock-group_7 localhost_12000
>>>>>> lock-group_8 localhost_12001
>>>>>> lock-group_9 localhost_12002
>>>>>> Stopping localhost_12000
>>>>>> localhost_12000Interrupted
>>>>>> localhost_12002 acquired lock:lock-group_4
>>>>>> localhost_12001 acquired lock:lock-group_11
>>>>>> localhost_12002 acquired lock:lock-group_7
>>>>>> localhost_12001 acquired lock:lock-group_0
>>>>>> lockName acquired By
>>>>>> ======================================
>>>>>> lock-group_0 localhost_12001
>>>>>> lock-group_1 localhost_12001
>>>>>> lock-group_10 localhost_12002
>>>>>> lock-group_11 localhost_12001
>>>>>> lock-group_2 localhost_12001
>>>>>> lock-group_3 localhost_12002
>>>>>> lock-group_4 localhost_12002
>>>>>> lock-group_5 localhost_12001
>>>>>> lock-group_6 localhost_12002
>>>>>> lock-group_7 localhost_12002
>>>>>> lock-group_8 localhost_12001
>>>>>> lock-group_9 localhost_12002
>>>>>> ===Starting localhost_12000
>>>>>> STARTING localhost_12000
>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>> STARTED localhost_12000
>>>>>> localhost_12001 releasing lock:lock-group_0
>>>>>> localhost_12001 releasing lock:lock-group_11
>>>>>> localhost_12002 releasing lock:lock-group_4
>>>>>> localhost_12002 releasing lock:lock-group_7
>>>>>> lockName acquired By
>>>>>> ======================================
>>>>>> lock-group_0 localhost_12000
>>>>>> lock-group_1 localhost_12001
>>>>>> lock-group_10 localhost_12002
>>>>>> lock-group_11 localhost_12000
>>>>>> lock-group_2 localhost_12001
>>>>>> lock-group_3 localhost_12002
>>>>>> lock-group_4 localhost_12000
>>>>>> lock-group_5 localhost_12001
>>>>>> lock-group_6 localhost_12002
>>>>>> lock-group_7 localhost_12000
>>>>>> lock-group_8 localhost_12001
>>>>>> lock-group_9 localhost_12002
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Lei Xia
>>>>>
>>>>
>>>>
>>>>
>>>> On Sat, Feb 24, 2018 at 8:26 PM, Lei Xia <xi...@gmail.com> wrote:
>>>>
>>>>> Hi, Utsav
>>>>>
>>>>>   Delay rebalancer by default is disabled in cluster level (this is to
>>>>> keep back-compatible somehow), you need to enable it in the clusterConfig,
>>>>> e.g
>>>>>
>>>>> ConfigAccessor configAccessor = new ConfigAccessor(zkClient);
>>>>> ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
>>>>> clusterConfig.setDelayRebalaceEnabled(enabled);
>>>>> configAccessor.setClusterConfig(clusterName, clusterConfig);
>>>>>
>>>>>
>>>>>   Could you please have a try and let me know whether it works or
>>>>> not?   Thanks
>>>>>
>>>>>
>>>>> Lei
>>>>>
>>>>>
>>>>> On Fri, Feb 23, 2018 at 2:33 PM, Utsav Kanani <ut...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I am trying to expand the Lockmanager example
>>>>>> http://helix.apache.org/0.6.2-incubating-docs/recipes/lock_m
>>>>>> anager.html to introduce delay
>>>>>>
>>>>>> tried doing something like this
>>>>>> IdealState state = admin.getResourceIdealState(clusterName,
>>>>>> lockGroupName);
>>>>>>      state.setRebalanceDelay(100000);
>>>>>>      state.setDelayRebalanceEnabled(true);
>>>>>>      state.setRebalancerClassName(DelayedAutoRebalancer.class.ge
>>>>>> tName());
>>>>>>      admin.setResourceIdealState(clusterName, lockGroupName, state);
>>>>>>      admin.rebalance(clusterName, lockGroupName, 1);
>>>>>>
>>>>>> On killing a node there is immediate rebalancing which takes place. I
>>>>>> was hoping for a delay of 100 seconds before rebalancing but i am not
>>>>>> seeing that behavior
>>>>>>
>>>>>>
>>>>>> On Stopping localhost_12000 the locks are acquired immediately by
>>>>>> localhost_12001 and localhost_12002
>>>>>>
>>>>>> on STARTING localhost_12000 the rebalance is again immediate.
>>>>>>
>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>> STARTED localhost_12000
>>>>>> localhost_12001 releasing lock:lock-group_0
>>>>>> localhost_12001 releasing lock:lock-group_11
>>>>>> localhost_12002 releasing lock:lock-group_4
>>>>>> localhost_12002 releasing lock:lock-group_7
>>>>>>
>>>>>>
>>>>>> Here is the output
>>>>>> =========================================
>>>>>>
>>>>>> STARTING localhost_12000
>>>>>> STARTING localhost_12001
>>>>>> STARTING localhost_12002
>>>>>> STARTED localhost_12001
>>>>>> STARTED localhost_12002
>>>>>> STARTED localhost_12000
>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>> localhost_12002 acquired lock:lock-group_10
>>>>>> localhost_12002 acquired lock:lock-group_9
>>>>>> localhost_12002 acquired lock:lock-group_3
>>>>>> localhost_12001 acquired lock:lock-group_2
>>>>>> localhost_12001 acquired lock:lock-group_1
>>>>>> localhost_12001 acquired lock:lock-group_8
>>>>>> localhost_12002 acquired lock:lock-group_6
>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>> localhost_12001 acquired lock:lock-group_5
>>>>>> lockName acquired By
>>>>>> ======================================
>>>>>> lock-group_0 localhost_12000
>>>>>> lock-group_1 localhost_12001
>>>>>> lock-group_10 localhost_12002
>>>>>> lock-group_11 localhost_12000
>>>>>> lock-group_2 localhost_12001
>>>>>> lock-group_3 localhost_12002
>>>>>> lock-group_4 localhost_12000
>>>>>> lock-group_5 localhost_12001
>>>>>> lock-group_6 localhost_12002
>>>>>> lock-group_7 localhost_12000
>>>>>> lock-group_8 localhost_12001
>>>>>> lock-group_9 localhost_12002
>>>>>> Stopping localhost_12000
>>>>>> localhost_12000Interrupted
>>>>>> localhost_12002 acquired lock:lock-group_4
>>>>>> localhost_12001 acquired lock:lock-group_11
>>>>>> localhost_12002 acquired lock:lock-group_7
>>>>>> localhost_12001 acquired lock:lock-group_0
>>>>>> lockName acquired By
>>>>>> ======================================
>>>>>> lock-group_0 localhost_12001
>>>>>> lock-group_1 localhost_12001
>>>>>> lock-group_10 localhost_12002
>>>>>> lock-group_11 localhost_12001
>>>>>> lock-group_2 localhost_12001
>>>>>> lock-group_3 localhost_12002
>>>>>> lock-group_4 localhost_12002
>>>>>> lock-group_5 localhost_12001
>>>>>> lock-group_6 localhost_12002
>>>>>> lock-group_7 localhost_12002
>>>>>> lock-group_8 localhost_12001
>>>>>> lock-group_9 localhost_12002
>>>>>> ===Starting localhost_12000
>>>>>> STARTING localhost_12000
>>>>>> localhost_12000 acquired lock:lock-group_11
>>>>>> localhost_12000 acquired lock:lock-group_7
>>>>>> localhost_12000 acquired lock:lock-group_0
>>>>>> localhost_12000 acquired lock:lock-group_4
>>>>>> STARTED localhost_12000
>>>>>> localhost_12001 releasing lock:lock-group_0
>>>>>> localhost_12001 releasing lock:lock-group_11
>>>>>> localhost_12002 releasing lock:lock-group_4
>>>>>> localhost_12002 releasing lock:lock-group_7
>>>>>> lockName acquired By
>>>>>> ======================================
>>>>>> lock-group_0 localhost_12000
>>>>>> lock-group_1 localhost_12001
>>>>>> lock-group_10 localhost_12002
>>>>>> lock-group_11 localhost_12000
>>>>>> lock-group_2 localhost_12001
>>>>>> lock-group_3 localhost_12002
>>>>>> lock-group_4 localhost_12000
>>>>>> lock-group_5 localhost_12001
>>>>>> lock-group_6 localhost_12002
>>>>>> lock-group_7 localhost_12000
>>>>>> lock-group_8 localhost_12001
>>>>>> lock-group_9 localhost_12002
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Lei Xia
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Lei Xia
>>>
>>
>>
>

Re: Unable to get Rebalance Delay to work using the distributed lock manager recipe

Posted by Utsav Kanani <ut...@gmail.com>.

sorry for the late response
hey guys this does not help either
do you guys want me to send you my code files?

this is the code change i made in the ZKHelixAdmin class
tried with both

idealState.setRebalancerClassName(DelayedAutoRebalancer.class.getName());

and withoug

@Override
public void addResource(String clusterName, String resourceName, int partitions,
    String stateModelRef, String rebalancerMode, String
rebalanceStrategy, int bucketSize,
    int maxPartitionsPerInstance) {
  if (!ZKUtil.isClusterSetup(clusterName, _zkClient)) {
    throw new HelixException("cluster " + clusterName + " is not setup yet");
  }

  IdealState idealState = new IdealState(resourceName);
  idealState.setNumPartitions(partitions);
  idealState.setStateModelDefRef(stateModelRef);
  RebalanceMode mode =
      idealState.rebalanceModeFromString(rebalancerMode,
RebalanceMode.SEMI_AUTO);
  idealState.setRebalanceMode(mode);
  idealState.setRebalanceStrategy(rebalanceStrategy);
  -------->idealState.setMinActiveReplicas(0);
  idealState.setStateModelFactoryName(HelixConstants.DEFAULT_STATE_MODEL_FACTORY);
  idealState.setRebalanceDelay(100000);
  idealState.setDelayRebalanceEnabled(true);
  //idealState.setRebalancerClassName(DelayedAutoRebalancer.class.getName());

  if (maxPartitionsPerInstance > 0 && maxPartitionsPerInstance <
Integer.MAX_VALUE) {
    idealState.setMaxPartitionsPerInstance(maxPartitionsPerInstance);
  }
  if (bucketSize > 0) {
    idealState.setBucketSize(bucketSize);
  }
  addResource(clusterName, resourceName, idealState);
}



On Thu, Mar 1, 2018 at 11:20 AM, kishore g <g....@gmail.com> wrote:

> We should have a recipe for delayed rebalancer
>
> On Thu, Mar 1, 2018 at 9:39 AM, Lei Xia <xi...@gmail.com> wrote:
>
>> Hi, Utsav
>>
>>   Sorry to get back to you late.  There is one more thing to config,
>>
>>     idealstate.setMinActiveReplicas(0);
>>
>>  This tell Helix the minimal replica it needs to maintain,  by default is
>> set to 1, it means Helix needs to maintain at least 1 replica irregardless
>> of delayed rebalancing.  For your case, you want to set it to 0.
>>
>>
>> Lei
>>
>> On Mon, Feb 26, 2018 at 11:38 AM, Utsav Kanani <ut...@gmail.com>
>> wrote:
>>
>>> Hi Lei,
>>>
>>> That did not work
>>> Seeing the same behavior
>>> Added the following method to ZKHelixAdmin Class
>>>
>>> public void enableClusterDelayMode(String clusterName) {
>>>   ConfigAccessor configAccessor = new ConfigAccessor(_zkClient);
>>>   ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
>>>   clusterConfig.setDelayRebalaceEnabled(true);
>>>   clusterConfig.setRebalanceDelayTime(100000);
>>>   configAccessor.setClusterConfig(clusterName, clusterConfig);
>>> }
>>>
>>> and calling it in the demo class
>>>
>>> HelixAdmin admin = new ZKHelixAdmin(zkAddress);
>>> admin.addCluster(clusterName, true);
>>> ---->((ZKHelixAdmin)admin).enableClusterDelayMode(clusterName);
>>> StateModelConfigGenerator generator = new StateModelConfigGenerator();
>>> admin.addStateModelDef(clusterName, "OnlineOffline",
>>>     new StateModelDefinition(generator.generateConfigForOnlineOffline()));
>>>
>>> admin.addResource(clusterName, lockGroupName, numPartitions, "OnlineOffline",
>>>     RebalanceMode.FULL_AUTO.toString());
>>> admin.rebalance(clusterName, lockGroupName, 1);
>>>
>>>
>>>
>>>
>>>
>>> STARTING localhost_12000
>>> STARTING localhost_12001
>>> STARTING localhost_12002
>>> STARTED localhost_12000
>>> STARTED localhost_12002
>>> STARTED localhost_12001
>>> localhost_12000 acquired lock:lock-group_0
>>> localhost_12002 acquired lock:lock-group_3
>>> localhost_12002 acquired lock:lock-group_9
>>> localhost_12001 acquired lock:lock-group_2
>>> localhost_12001 acquired lock:lock-group_5
>>> localhost_12000 acquired lock:lock-group_11
>>> localhost_12002 acquired lock:lock-group_6
>>> localhost_12000 acquired lock:lock-group_7
>>> localhost_12002 acquired lock:lock-group_10
>>> localhost_12001 acquired lock:lock-group_8
>>> localhost_12001 acquired lock:lock-group_1
>>> localhost_12000 acquired lock:lock-group_4
>>> lockName acquired By
>>> ======================================
>>> lock-group_0 localhost_12000
>>> lock-group_1 localhost_12001
>>> lock-group_10 localhost_12002
>>> lock-group_11 localhost_12000
>>> lock-group_2 localhost_12001
>>> lock-group_3 localhost_12002
>>> lock-group_4 localhost_12000
>>> lock-group_5 localhost_12001
>>> lock-group_6 localhost_12002
>>> lock-group_7 localhost_12000
>>> lock-group_8 localhost_12001
>>> lock-group_9 localhost_12002
>>> Stopping localhost_12000
>>> localhost_12000Interrupted
>>> localhost_12001 acquired lock:lock-group_11
>>> localhost_12001 acquired lock:lock-group_0
>>> localhost_12002 acquired lock:lock-group_7
>>> localhost_12002 acquired lock:lock-group_4
>>> lockName acquired By
>>> ======================================
>>> lock-group_0 localhost_12001
>>> lock-group_1 localhost_12001
>>> lock-group_10 localhost_12002
>>> lock-group_11 localhost_12001
>>> lock-group_2 localhost_12001
>>> lock-group_3 localhost_12002
>>> lock-group_4 localhost_12002
>>> lock-group_5 localhost_12001
>>> lock-group_6 localhost_12002
>>> lock-group_7 localhost_12002
>>> lock-group_8 localhost_12001
>>> lock-group_9 localhost_12002
>>> ===Starting localhost_12000
>>> STARTING localhost_12000
>>> localhost_12000 acquired lock:lock-group_11
>>> localhost_12000 acquired lock:lock-group_0
>>> STARTED localhost_12000
>>> localhost_12000 acquired lock:lock-group_7
>>> localhost_12000 acquired lock:lock-group_4
>>> localhost_12001 releasing lock:lock-group_11
>>> localhost_12001 releasing lock:lock-group_0
>>> localhost_12002 releasing lock:lock-group_7
>>> localhost_12002 releasing lock:lock-group_4
>>> lockName acquired By
>>> ======================================
>>> lock-group_0 localhost_12000
>>> lock-group_1 localhost_12001
>>> lock-group_10 localhost_12002
>>> lock-group_11 localhost_12000
>>> lock-group_2 localhost_12001
>>> lock-group_3 localhost_12002
>>> lock-group_4 localhost_12000
>>> lock-group_5 localhost_12001
>>> lock-group_6 localhost_12002
>>> lock-group_7 localhost_12000
>>> lock-group_8 localhost_12001
>>> lock-group_9 localhost_12002
>>>
>>>
>>> On Sat, Feb 24, 2018 at 8:26 PM, Lei Xia <xi...@gmail.com> wrote:
>>>
>>>> Hi, Utsav
>>>>
>>>>   Delay rebalancer by default is disabled in cluster level (this is to
>>>> keep back-compatible somehow), you need to enable it in the clusterConfig,
>>>> e.g
>>>>
>>>> ConfigAccessor configAccessor = new ConfigAccessor(zkClient);
>>>> ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
>>>> clusterConfig.setDelayRebalaceEnabled(enabled);
>>>> configAccessor.setClusterConfig(clusterName, clusterConfig);
>>>>
>>>>
>>>>   Could you please have a try and let me know whether it works or
>>>> not?   Thanks
>>>>
>>>>
>>>> Lei
>>>>
>>>>
>>>> On Fri, Feb 23, 2018 at 2:33 PM, Utsav Kanani <ut...@gmail.com>
>>>> wrote:
>>>>
>>>>> I am trying to expand the Lockmanager example http://helix.apache.or
>>>>> g/0.6.2-incubating-docs/recipes/lock_manager.html to introduce delay
>>>>>
>>>>> tried doing something like this
>>>>> IdealState state = admin.getResourceIdealState(clusterName,
>>>>> lockGroupName);
>>>>>      state.setRebalanceDelay(100000);
>>>>>      state.setDelayRebalanceEnabled(true);
>>>>>      state.setRebalancerClassName(DelayedAutoRebalancer.class.ge
>>>>> tName());
>>>>>      admin.setResourceIdealState(clusterName, lockGroupName, state);
>>>>>      admin.rebalance(clusterName, lockGroupName, 1);
>>>>>
>>>>> On killing a node there is immediate rebalancing which takes place. I
>>>>> was hoping for a delay of 100 seconds before rebalancing but i am not
>>>>> seeing that behavior
>>>>>
>>>>>
>>>>> On Stopping localhost_12000 the locks are acquired immediately by
>>>>> localhost_12001 and localhost_12002
>>>>>
>>>>> on STARTING localhost_12000 the rebalance is again immediate.
>>>>>
>>>>> localhost_12000 acquired lock:lock-group_11
>>>>> localhost_12000 acquired lock:lock-group_7
>>>>> localhost_12000 acquired lock:lock-group_0
>>>>> localhost_12000 acquired lock:lock-group_4
>>>>> STARTED localhost_12000
>>>>> localhost_12001 releasing lock:lock-group_0
>>>>> localhost_12001 releasing lock:lock-group_11
>>>>> localhost_12002 releasing lock:lock-group_4
>>>>> localhost_12002 releasing lock:lock-group_7
>>>>>
>>>>>
>>>>> Here is the output
>>>>> =========================================
>>>>>
>>>>> STARTING localhost_12000
>>>>> STARTING localhost_12001
>>>>> STARTING localhost_12002
>>>>> STARTED localhost_12001
>>>>> STARTED localhost_12002
>>>>> STARTED localhost_12000
>>>>> localhost_12000 acquired lock:lock-group_11
>>>>> localhost_12002 acquired lock:lock-group_10
>>>>> localhost_12002 acquired lock:lock-group_9
>>>>> localhost_12002 acquired lock:lock-group_3
>>>>> localhost_12001 acquired lock:lock-group_2
>>>>> localhost_12001 acquired lock:lock-group_1
>>>>> localhost_12001 acquired lock:lock-group_8
>>>>> localhost_12002 acquired lock:lock-group_6
>>>>> localhost_12000 acquired lock:lock-group_4
>>>>> localhost_12000 acquired lock:lock-group_0
>>>>> localhost_12000 acquired lock:lock-group_7
>>>>> localhost_12001 acquired lock:lock-group_5
>>>>> lockName acquired By
>>>>> ======================================
>>>>> lock-group_0 localhost_12000
>>>>> lock-group_1 localhost_12001
>>>>> lock-group_10 localhost_12002
>>>>> lock-group_11 localhost_12000
>>>>> lock-group_2 localhost_12001
>>>>> lock-group_3 localhost_12002
>>>>> lock-group_4 localhost_12000
>>>>> lock-group_5 localhost_12001
>>>>> lock-group_6 localhost_12002
>>>>> lock-group_7 localhost_12000
>>>>> lock-group_8 localhost_12001
>>>>> lock-group_9 localhost_12002
>>>>> Stopping localhost_12000
>>>>> localhost_12000Interrupted
>>>>> localhost_12002 acquired lock:lock-group_4
>>>>> localhost_12001 acquired lock:lock-group_11
>>>>> localhost_12002 acquired lock:lock-group_7
>>>>> localhost_12001 acquired lock:lock-group_0
>>>>> lockName acquired By
>>>>> ======================================
>>>>> lock-group_0 localhost_12001
>>>>> lock-group_1 localhost_12001
>>>>> lock-group_10 localhost_12002
>>>>> lock-group_11 localhost_12001
>>>>> lock-group_2 localhost_12001
>>>>> lock-group_3 localhost_12002
>>>>> lock-group_4 localhost_12002
>>>>> lock-group_5 localhost_12001
>>>>> lock-group_6 localhost_12002
>>>>> lock-group_7 localhost_12002
>>>>> lock-group_8 localhost_12001
>>>>> lock-group_9 localhost_12002
>>>>> ===Starting localhost_12000
>>>>> STARTING localhost_12000
>>>>> localhost_12000 acquired lock:lock-group_11
>>>>> localhost_12000 acquired lock:lock-group_7
>>>>> localhost_12000 acquired lock:lock-group_0
>>>>> localhost_12000 acquired lock:lock-group_4
>>>>> STARTED localhost_12000
>>>>> localhost_12001 releasing lock:lock-group_0
>>>>> localhost_12001 releasing lock:lock-group_11
>>>>> localhost_12002 releasing lock:lock-group_4
>>>>> localhost_12002 releasing lock:lock-group_7
>>>>> lockName acquired By
>>>>> ======================================
>>>>> lock-group_0 localhost_12000
>>>>> lock-group_1 localhost_12001
>>>>> lock-group_10 localhost_12002
>>>>> lock-group_11 localhost_12000
>>>>> lock-group_2 localhost_12001
>>>>> lock-group_3 localhost_12002
>>>>> lock-group_4 localhost_12000
>>>>> lock-group_5 localhost_12001
>>>>> lock-group_6 localhost_12002
>>>>> lock-group_7 localhost_12000
>>>>> lock-group_8 localhost_12001
>>>>> lock-group_9 localhost_12002
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Lei Xia
>>>>
>>>
>>>
>>>
>>> On Sat, Feb 24, 2018 at 8:26 PM, Lei Xia <xi...@gmail.com> wrote:
>>>
>>>> Hi, Utsav
>>>>
>>>>   Delay rebalancer by default is disabled in cluster level (this is to
>>>> keep back-compatible somehow), you need to enable it in the clusterConfig,
>>>> e.g
>>>>
>>>> ConfigAccessor configAccessor = new ConfigAccessor(zkClient);
>>>> ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
>>>> clusterConfig.setDelayRebalaceEnabled(enabled);
>>>> configAccessor.setClusterConfig(clusterName, clusterConfig);
>>>>
>>>>
>>>>   Could you please have a try and let me know whether it works or
>>>> not?   Thanks
>>>>
>>>>
>>>> Lei
>>>>
>>>>
>>>> On Fri, Feb 23, 2018 at 2:33 PM, Utsav Kanani <ut...@gmail.com>
>>>> wrote:
>>>>
>>>>> I am trying to expand the Lockmanager example
>>>>> http://helix.apache.org/0.6.2-incubating-docs/recipes/lock_m
>>>>> anager.html to introduce delay
>>>>>
>>>>> tried doing something like this
>>>>> IdealState state = admin.getResourceIdealState(clusterName,
>>>>> lockGroupName);
>>>>>      state.setRebalanceDelay(100000);
>>>>>      state.setDelayRebalanceEnabled(true);
>>>>>      state.setRebalancerClassName(DelayedAutoRebalancer.class.ge
>>>>> tName());
>>>>>      admin.setResourceIdealState(clusterName, lockGroupName, state);
>>>>>      admin.rebalance(clusterName, lockGroupName, 1);
>>>>>
>>>>> On killing a node there is immediate rebalancing which takes place. I
>>>>> was hoping for a delay of 100 seconds before rebalancing but i am not
>>>>> seeing that behavior
>>>>>
>>>>>
>>>>> On Stopping localhost_12000 the locks are acquired immediately by
>>>>> localhost_12001 and localhost_12002
>>>>>
>>>>> on STARTING localhost_12000 the rebalance is again immediate.
>>>>>
>>>>> localhost_12000 acquired lock:lock-group_11
>>>>> localhost_12000 acquired lock:lock-group_7
>>>>> localhost_12000 acquired lock:lock-group_0
>>>>> localhost_12000 acquired lock:lock-group_4
>>>>> STARTED localhost_12000
>>>>> localhost_12001 releasing lock:lock-group_0
>>>>> localhost_12001 releasing lock:lock-group_11
>>>>> localhost_12002 releasing lock:lock-group_4
>>>>> localhost_12002 releasing lock:lock-group_7
>>>>>
>>>>>
>>>>> Here is the output
>>>>> =========================================
>>>>>
>>>>> STARTING localhost_12000
>>>>> STARTING localhost_12001
>>>>> STARTING localhost_12002
>>>>> STARTED localhost_12001
>>>>> STARTED localhost_12002
>>>>> STARTED localhost_12000
>>>>> localhost_12000 acquired lock:lock-group_11
>>>>> localhost_12002 acquired lock:lock-group_10
>>>>> localhost_12002 acquired lock:lock-group_9
>>>>> localhost_12002 acquired lock:lock-group_3
>>>>> localhost_12001 acquired lock:lock-group_2
>>>>> localhost_12001 acquired lock:lock-group_1
>>>>> localhost_12001 acquired lock:lock-group_8
>>>>> localhost_12002 acquired lock:lock-group_6
>>>>> localhost_12000 acquired lock:lock-group_4
>>>>> localhost_12000 acquired lock:lock-group_0
>>>>> localhost_12000 acquired lock:lock-group_7
>>>>> localhost_12001 acquired lock:lock-group_5
>>>>> lockName acquired By
>>>>> ======================================
>>>>> lock-group_0 localhost_12000
>>>>> lock-group_1 localhost_12001
>>>>> lock-group_10 localhost_12002
>>>>> lock-group_11 localhost_12000
>>>>> lock-group_2 localhost_12001
>>>>> lock-group_3 localhost_12002
>>>>> lock-group_4 localhost_12000
>>>>> lock-group_5 localhost_12001
>>>>> lock-group_6 localhost_12002
>>>>> lock-group_7 localhost_12000
>>>>> lock-group_8 localhost_12001
>>>>> lock-group_9 localhost_12002
>>>>> Stopping localhost_12000
>>>>> localhost_12000Interrupted
>>>>> localhost_12002 acquired lock:lock-group_4
>>>>> localhost_12001 acquired lock:lock-group_11
>>>>> localhost_12002 acquired lock:lock-group_7
>>>>> localhost_12001 acquired lock:lock-group_0
>>>>> lockName acquired By
>>>>> ======================================
>>>>> lock-group_0 localhost_12001
>>>>> lock-group_1 localhost_12001
>>>>> lock-group_10 localhost_12002
>>>>> lock-group_11 localhost_12001
>>>>> lock-group_2 localhost_12001
>>>>> lock-group_3 localhost_12002
>>>>> lock-group_4 localhost_12002
>>>>> lock-group_5 localhost_12001
>>>>> lock-group_6 localhost_12002
>>>>> lock-group_7 localhost_12002
>>>>> lock-group_8 localhost_12001
>>>>> lock-group_9 localhost_12002
>>>>> ===Starting localhost_12000
>>>>> STARTING localhost_12000
>>>>> localhost_12000 acquired lock:lock-group_11
>>>>> localhost_12000 acquired lock:lock-group_7
>>>>> localhost_12000 acquired lock:lock-group_0
>>>>> localhost_12000 acquired lock:lock-group_4
>>>>> STARTED localhost_12000
>>>>> localhost_12001 releasing lock:lock-group_0
>>>>> localhost_12001 releasing lock:lock-group_11
>>>>> localhost_12002 releasing lock:lock-group_4
>>>>> localhost_12002 releasing lock:lock-group_7
>>>>> lockName acquired By
>>>>> ======================================
>>>>> lock-group_0 localhost_12000
>>>>> lock-group_1 localhost_12001
>>>>> lock-group_10 localhost_12002
>>>>> lock-group_11 localhost_12000
>>>>> lock-group_2 localhost_12001
>>>>> lock-group_3 localhost_12002
>>>>> lock-group_4 localhost_12000
>>>>> lock-group_5 localhost_12001
>>>>> lock-group_6 localhost_12002
>>>>> lock-group_7 localhost_12000
>>>>> lock-group_8 localhost_12001
>>>>> lock-group_9 localhost_12002
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Lei Xia
>>>>
>>>
>>>
>>
>>
>> --
>> Lei Xia
>>
>
>

Re: Unable to get Rebalance Delay to work using the distributed lock manager recipe

Posted by kishore g <g....@gmail.com>.

We should have a recipe for delayed rebalancer

On Thu, Mar 1, 2018 at 9:39 AM, Lei Xia <xi...@gmail.com> wrote:

> Hi, Utsav
>
>   Sorry to get back to you late.  There is one more thing to config,
>
>     idealstate.setMinActiveReplicas(0);
>
>  This tell Helix the minimal replica it needs to maintain,  by default is
> set to 1, it means Helix needs to maintain at least 1 replica irregardless
> of delayed rebalancing.  For your case, you want to set it to 0.
>
>
> Lei
>
> On Mon, Feb 26, 2018 at 11:38 AM, Utsav Kanani <ut...@gmail.com>
> wrote:
>
>> Hi Lei,
>>
>> That did not work
>> Seeing the same behavior
>> Added the following method to ZKHelixAdmin Class
>>
>> public void enableClusterDelayMode(String clusterName) {
>>   ConfigAccessor configAccessor = new ConfigAccessor(_zkClient);
>>   ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
>>   clusterConfig.setDelayRebalaceEnabled(true);
>>   clusterConfig.setRebalanceDelayTime(100000);
>>   configAccessor.setClusterConfig(clusterName, clusterConfig);
>> }
>>
>> and calling it in the demo class
>>
>> HelixAdmin admin = new ZKHelixAdmin(zkAddress);
>> admin.addCluster(clusterName, true);
>> ---->((ZKHelixAdmin)admin).enableClusterDelayMode(clusterName);
>> StateModelConfigGenerator generator = new StateModelConfigGenerator();
>> admin.addStateModelDef(clusterName, "OnlineOffline",
>>     new StateModelDefinition(generator.generateConfigForOnlineOffline()));
>>
>> admin.addResource(clusterName, lockGroupName, numPartitions, "OnlineOffline",
>>     RebalanceMode.FULL_AUTO.toString());
>> admin.rebalance(clusterName, lockGroupName, 1);
>>
>>
>>
>>
>>
>> STARTING localhost_12000
>> STARTING localhost_12001
>> STARTING localhost_12002
>> STARTED localhost_12000
>> STARTED localhost_12002
>> STARTED localhost_12001
>> localhost_12000 acquired lock:lock-group_0
>> localhost_12002 acquired lock:lock-group_3
>> localhost_12002 acquired lock:lock-group_9
>> localhost_12001 acquired lock:lock-group_2
>> localhost_12001 acquired lock:lock-group_5
>> localhost_12000 acquired lock:lock-group_11
>> localhost_12002 acquired lock:lock-group_6
>> localhost_12000 acquired lock:lock-group_7
>> localhost_12002 acquired lock:lock-group_10
>> localhost_12001 acquired lock:lock-group_8
>> localhost_12001 acquired lock:lock-group_1
>> localhost_12000 acquired lock:lock-group_4
>> lockName acquired By
>> ======================================
>> lock-group_0 localhost_12000
>> lock-group_1 localhost_12001
>> lock-group_10 localhost_12002
>> lock-group_11 localhost_12000
>> lock-group_2 localhost_12001
>> lock-group_3 localhost_12002
>> lock-group_4 localhost_12000
>> lock-group_5 localhost_12001
>> lock-group_6 localhost_12002
>> lock-group_7 localhost_12000
>> lock-group_8 localhost_12001
>> lock-group_9 localhost_12002
>> Stopping localhost_12000
>> localhost_12000Interrupted
>> localhost_12001 acquired lock:lock-group_11
>> localhost_12001 acquired lock:lock-group_0
>> localhost_12002 acquired lock:lock-group_7
>> localhost_12002 acquired lock:lock-group_4
>> lockName acquired By
>> ======================================
>> lock-group_0 localhost_12001
>> lock-group_1 localhost_12001
>> lock-group_10 localhost_12002
>> lock-group_11 localhost_12001
>> lock-group_2 localhost_12001
>> lock-group_3 localhost_12002
>> lock-group_4 localhost_12002
>> lock-group_5 localhost_12001
>> lock-group_6 localhost_12002
>> lock-group_7 localhost_12002
>> lock-group_8 localhost_12001
>> lock-group_9 localhost_12002
>> ===Starting localhost_12000
>> STARTING localhost_12000
>> localhost_12000 acquired lock:lock-group_11
>> localhost_12000 acquired lock:lock-group_0
>> STARTED localhost_12000
>> localhost_12000 acquired lock:lock-group_7
>> localhost_12000 acquired lock:lock-group_4
>> localhost_12001 releasing lock:lock-group_11
>> localhost_12001 releasing lock:lock-group_0
>> localhost_12002 releasing lock:lock-group_7
>> localhost_12002 releasing lock:lock-group_4
>> lockName acquired By
>> ======================================
>> lock-group_0 localhost_12000
>> lock-group_1 localhost_12001
>> lock-group_10 localhost_12002
>> lock-group_11 localhost_12000
>> lock-group_2 localhost_12001
>> lock-group_3 localhost_12002
>> lock-group_4 localhost_12000
>> lock-group_5 localhost_12001
>> lock-group_6 localhost_12002
>> lock-group_7 localhost_12000
>> lock-group_8 localhost_12001
>> lock-group_9 localhost_12002
>>
>>
>> On Sat, Feb 24, 2018 at 8:26 PM, Lei Xia <xi...@gmail.com> wrote:
>>
>>> Hi, Utsav
>>>
>>>   Delay rebalancer by default is disabled in cluster level (this is to
>>> keep back-compatible somehow), you need to enable it in the clusterConfig,
>>> e.g
>>>
>>> ConfigAccessor configAccessor = new ConfigAccessor(zkClient);
>>> ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
>>> clusterConfig.setDelayRebalaceEnabled(enabled);
>>> configAccessor.setClusterConfig(clusterName, clusterConfig);
>>>
>>>
>>>   Could you please have a try and let me know whether it works or not?
>>> Thanks
>>>
>>>
>>> Lei
>>>
>>>
>>> On Fri, Feb 23, 2018 at 2:33 PM, Utsav Kanani <ut...@gmail.com>
>>> wrote:
>>>
>>>> I am trying to expand the Lockmanager example http://helix.apache.or
>>>> g/0.6.2-incubating-docs/recipes/lock_manager.html to introduce delay
>>>>
>>>> tried doing something like this
>>>> IdealState state = admin.getResourceIdealState(clusterName,
>>>> lockGroupName);
>>>>      state.setRebalanceDelay(100000);
>>>>      state.setDelayRebalanceEnabled(true);
>>>>      state.setRebalancerClassName(DelayedAutoRebalancer.class.ge
>>>> tName());
>>>>      admin.setResourceIdealState(clusterName, lockGroupName, state);
>>>>      admin.rebalance(clusterName, lockGroupName, 1);
>>>>
>>>> On killing a node there is immediate rebalancing which takes place. I
>>>> was hoping for a delay of 100 seconds before rebalancing but i am not
>>>> seeing that behavior
>>>>
>>>>
>>>> On Stopping localhost_12000 the locks are acquired immediately by
>>>> localhost_12001 and localhost_12002
>>>>
>>>> on STARTING localhost_12000 the rebalance is again immediate.
>>>>
>>>> localhost_12000 acquired lock:lock-group_11
>>>> localhost_12000 acquired lock:lock-group_7
>>>> localhost_12000 acquired lock:lock-group_0
>>>> localhost_12000 acquired lock:lock-group_4
>>>> STARTED localhost_12000
>>>> localhost_12001 releasing lock:lock-group_0
>>>> localhost_12001 releasing lock:lock-group_11
>>>> localhost_12002 releasing lock:lock-group_4
>>>> localhost_12002 releasing lock:lock-group_7
>>>>
>>>>
>>>> Here is the output
>>>> =========================================
>>>>
>>>> STARTING localhost_12000
>>>> STARTING localhost_12001
>>>> STARTING localhost_12002
>>>> STARTED localhost_12001
>>>> STARTED localhost_12002
>>>> STARTED localhost_12000
>>>> localhost_12000 acquired lock:lock-group_11
>>>> localhost_12002 acquired lock:lock-group_10
>>>> localhost_12002 acquired lock:lock-group_9
>>>> localhost_12002 acquired lock:lock-group_3
>>>> localhost_12001 acquired lock:lock-group_2
>>>> localhost_12001 acquired lock:lock-group_1
>>>> localhost_12001 acquired lock:lock-group_8
>>>> localhost_12002 acquired lock:lock-group_6
>>>> localhost_12000 acquired lock:lock-group_4
>>>> localhost_12000 acquired lock:lock-group_0
>>>> localhost_12000 acquired lock:lock-group_7
>>>> localhost_12001 acquired lock:lock-group_5
>>>> lockName acquired By
>>>> ======================================
>>>> lock-group_0 localhost_12000
>>>> lock-group_1 localhost_12001
>>>> lock-group_10 localhost_12002
>>>> lock-group_11 localhost_12000
>>>> lock-group_2 localhost_12001
>>>> lock-group_3 localhost_12002
>>>> lock-group_4 localhost_12000
>>>> lock-group_5 localhost_12001
>>>> lock-group_6 localhost_12002
>>>> lock-group_7 localhost_12000
>>>> lock-group_8 localhost_12001
>>>> lock-group_9 localhost_12002
>>>> Stopping localhost_12000
>>>> localhost_12000Interrupted
>>>> localhost_12002 acquired lock:lock-group_4
>>>> localhost_12001 acquired lock:lock-group_11
>>>> localhost_12002 acquired lock:lock-group_7
>>>> localhost_12001 acquired lock:lock-group_0
>>>> lockName acquired By
>>>> ======================================
>>>> lock-group_0 localhost_12001
>>>> lock-group_1 localhost_12001
>>>> lock-group_10 localhost_12002
>>>> lock-group_11 localhost_12001
>>>> lock-group_2 localhost_12001
>>>> lock-group_3 localhost_12002
>>>> lock-group_4 localhost_12002
>>>> lock-group_5 localhost_12001
>>>> lock-group_6 localhost_12002
>>>> lock-group_7 localhost_12002
>>>> lock-group_8 localhost_12001
>>>> lock-group_9 localhost_12002
>>>> ===Starting localhost_12000
>>>> STARTING localhost_12000
>>>> localhost_12000 acquired lock:lock-group_11
>>>> localhost_12000 acquired lock:lock-group_7
>>>> localhost_12000 acquired lock:lock-group_0
>>>> localhost_12000 acquired lock:lock-group_4
>>>> STARTED localhost_12000
>>>> localhost_12001 releasing lock:lock-group_0
>>>> localhost_12001 releasing lock:lock-group_11
>>>> localhost_12002 releasing lock:lock-group_4
>>>> localhost_12002 releasing lock:lock-group_7
>>>> lockName acquired By
>>>> ======================================
>>>> lock-group_0 localhost_12000
>>>> lock-group_1 localhost_12001
>>>> lock-group_10 localhost_12002
>>>> lock-group_11 localhost_12000
>>>> lock-group_2 localhost_12001
>>>> lock-group_3 localhost_12002
>>>> lock-group_4 localhost_12000
>>>> lock-group_5 localhost_12001
>>>> lock-group_6 localhost_12002
>>>> lock-group_7 localhost_12000
>>>> lock-group_8 localhost_12001
>>>> lock-group_9 localhost_12002
>>>>
>>>
>>>
>>>
>>> --
>>> Lei Xia
>>>
>>
>>
>>
>> On Sat, Feb 24, 2018 at 8:26 PM, Lei Xia <xi...@gmail.com> wrote:
>>
>>> Hi, Utsav
>>>
>>>   Delay rebalancer by default is disabled in cluster level (this is to
>>> keep back-compatible somehow), you need to enable it in the clusterConfig,
>>> e.g
>>>
>>> ConfigAccessor configAccessor = new ConfigAccessor(zkClient);
>>> ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
>>> clusterConfig.setDelayRebalaceEnabled(enabled);
>>> configAccessor.setClusterConfig(clusterName, clusterConfig);
>>>
>>>
>>>   Could you please have a try and let me know whether it works or not?
>>> Thanks
>>>
>>>
>>> Lei
>>>
>>>
>>> On Fri, Feb 23, 2018 at 2:33 PM, Utsav Kanani <ut...@gmail.com>
>>> wrote:
>>>
>>>> I am trying to expand the Lockmanager example
>>>> http://helix.apache.org/0.6.2-incubating-docs/recipes/lock_manager.html
>>>> to introduce delay
>>>>
>>>> tried doing something like this
>>>> IdealState state = admin.getResourceIdealState(clusterName,
>>>> lockGroupName);
>>>>      state.setRebalanceDelay(100000);
>>>>      state.setDelayRebalanceEnabled(true);
>>>>      state.setRebalancerClassName(DelayedAutoRebalancer.class.ge
>>>> tName());
>>>>      admin.setResourceIdealState(clusterName, lockGroupName, state);
>>>>      admin.rebalance(clusterName, lockGroupName, 1);
>>>>
>>>> On killing a node there is immediate rebalancing which takes place. I
>>>> was hoping for a delay of 100 seconds before rebalancing but i am not
>>>> seeing that behavior
>>>>
>>>>
>>>> On Stopping localhost_12000 the locks are acquired immediately by
>>>> localhost_12001 and localhost_12002
>>>>
>>>> on STARTING localhost_12000 the rebalance is again immediate.
>>>>
>>>> localhost_12000 acquired lock:lock-group_11
>>>> localhost_12000 acquired lock:lock-group_7
>>>> localhost_12000 acquired lock:lock-group_0
>>>> localhost_12000 acquired lock:lock-group_4
>>>> STARTED localhost_12000
>>>> localhost_12001 releasing lock:lock-group_0
>>>> localhost_12001 releasing lock:lock-group_11
>>>> localhost_12002 releasing lock:lock-group_4
>>>> localhost_12002 releasing lock:lock-group_7
>>>>
>>>>
>>>> Here is the output
>>>> =========================================
>>>>
>>>> STARTING localhost_12000
>>>> STARTING localhost_12001
>>>> STARTING localhost_12002
>>>> STARTED localhost_12001
>>>> STARTED localhost_12002
>>>> STARTED localhost_12000
>>>> localhost_12000 acquired lock:lock-group_11
>>>> localhost_12002 acquired lock:lock-group_10
>>>> localhost_12002 acquired lock:lock-group_9
>>>> localhost_12002 acquired lock:lock-group_3
>>>> localhost_12001 acquired lock:lock-group_2
>>>> localhost_12001 acquired lock:lock-group_1
>>>> localhost_12001 acquired lock:lock-group_8
>>>> localhost_12002 acquired lock:lock-group_6
>>>> localhost_12000 acquired lock:lock-group_4
>>>> localhost_12000 acquired lock:lock-group_0
>>>> localhost_12000 acquired lock:lock-group_7
>>>> localhost_12001 acquired lock:lock-group_5
>>>> lockName acquired By
>>>> ======================================
>>>> lock-group_0 localhost_12000
>>>> lock-group_1 localhost_12001
>>>> lock-group_10 localhost_12002
>>>> lock-group_11 localhost_12000
>>>> lock-group_2 localhost_12001
>>>> lock-group_3 localhost_12002
>>>> lock-group_4 localhost_12000
>>>> lock-group_5 localhost_12001
>>>> lock-group_6 localhost_12002
>>>> lock-group_7 localhost_12000
>>>> lock-group_8 localhost_12001
>>>> lock-group_9 localhost_12002
>>>> Stopping localhost_12000
>>>> localhost_12000Interrupted
>>>> localhost_12002 acquired lock:lock-group_4
>>>> localhost_12001 acquired lock:lock-group_11
>>>> localhost_12002 acquired lock:lock-group_7
>>>> localhost_12001 acquired lock:lock-group_0
>>>> lockName acquired By
>>>> ======================================
>>>> lock-group_0 localhost_12001
>>>> lock-group_1 localhost_12001
>>>> lock-group_10 localhost_12002
>>>> lock-group_11 localhost_12001
>>>> lock-group_2 localhost_12001
>>>> lock-group_3 localhost_12002
>>>> lock-group_4 localhost_12002
>>>> lock-group_5 localhost_12001
>>>> lock-group_6 localhost_12002
>>>> lock-group_7 localhost_12002
>>>> lock-group_8 localhost_12001
>>>> lock-group_9 localhost_12002
>>>> ===Starting localhost_12000
>>>> STARTING localhost_12000
>>>> localhost_12000 acquired lock:lock-group_11
>>>> localhost_12000 acquired lock:lock-group_7
>>>> localhost_12000 acquired lock:lock-group_0
>>>> localhost_12000 acquired lock:lock-group_4
>>>> STARTED localhost_12000
>>>> localhost_12001 releasing lock:lock-group_0
>>>> localhost_12001 releasing lock:lock-group_11
>>>> localhost_12002 releasing lock:lock-group_4
>>>> localhost_12002 releasing lock:lock-group_7
>>>> lockName acquired By
>>>> ======================================
>>>> lock-group_0 localhost_12000
>>>> lock-group_1 localhost_12001
>>>> lock-group_10 localhost_12002
>>>> lock-group_11 localhost_12000
>>>> lock-group_2 localhost_12001
>>>> lock-group_3 localhost_12002
>>>> lock-group_4 localhost_12000
>>>> lock-group_5 localhost_12001
>>>> lock-group_6 localhost_12002
>>>> lock-group_7 localhost_12000
>>>> lock-group_8 localhost_12001
>>>> lock-group_9 localhost_12002
>>>>
>>>
>>>
>>>
>>> --
>>> Lei Xia
>>>
>>
>>
>
>
> --
> Lei Xia
>

Re: Unable to get Rebalance Delay to work using the distributed lock manager recipe

Posted by Lei Xia <xi...@gmail.com>.

Hi, Utsav

  Sorry to get back to you late.  There is one more thing to config,

    idealstate.setMinActiveReplicas(0);

 This tell Helix the minimal replica it needs to maintain,  by default is
set to 1, it means Helix needs to maintain at least 1 replica irregardless
of delayed rebalancing.  For your case, you want to set it to 0.


Lei

On Mon, Feb 26, 2018 at 11:38 AM, Utsav Kanani <ut...@gmail.com>
wrote:

> Hi Lei,
>
> That did not work
> Seeing the same behavior
> Added the following method to ZKHelixAdmin Class
>
> public void enableClusterDelayMode(String clusterName) {
>   ConfigAccessor configAccessor = new ConfigAccessor(_zkClient);
>   ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
>   clusterConfig.setDelayRebalaceEnabled(true);
>   clusterConfig.setRebalanceDelayTime(100000);
>   configAccessor.setClusterConfig(clusterName, clusterConfig);
> }
>
> and calling it in the demo class
>
> HelixAdmin admin = new ZKHelixAdmin(zkAddress);
> admin.addCluster(clusterName, true);
> ---->((ZKHelixAdmin)admin).enableClusterDelayMode(clusterName);
> StateModelConfigGenerator generator = new StateModelConfigGenerator();
> admin.addStateModelDef(clusterName, "OnlineOffline",
>     new StateModelDefinition(generator.generateConfigForOnlineOffline()));
>
> admin.addResource(clusterName, lockGroupName, numPartitions, "OnlineOffline",
>     RebalanceMode.FULL_AUTO.toString());
> admin.rebalance(clusterName, lockGroupName, 1);
>
>
>
>
>
> STARTING localhost_12000
> STARTING localhost_12001
> STARTING localhost_12002
> STARTED localhost_12000
> STARTED localhost_12002
> STARTED localhost_12001
> localhost_12000 acquired lock:lock-group_0
> localhost_12002 acquired lock:lock-group_3
> localhost_12002 acquired lock:lock-group_9
> localhost_12001 acquired lock:lock-group_2
> localhost_12001 acquired lock:lock-group_5
> localhost_12000 acquired lock:lock-group_11
> localhost_12002 acquired lock:lock-group_6
> localhost_12000 acquired lock:lock-group_7
> localhost_12002 acquired lock:lock-group_10
> localhost_12001 acquired lock:lock-group_8
> localhost_12001 acquired lock:lock-group_1
> localhost_12000 acquired lock:lock-group_4
> lockName acquired By
> ======================================
> lock-group_0 localhost_12000
> lock-group_1 localhost_12001
> lock-group_10 localhost_12002
> lock-group_11 localhost_12000
> lock-group_2 localhost_12001
> lock-group_3 localhost_12002
> lock-group_4 localhost_12000
> lock-group_5 localhost_12001
> lock-group_6 localhost_12002
> lock-group_7 localhost_12000
> lock-group_8 localhost_12001
> lock-group_9 localhost_12002
> Stopping localhost_12000
> localhost_12000Interrupted
> localhost_12001 acquired lock:lock-group_11
> localhost_12001 acquired lock:lock-group_0
> localhost_12002 acquired lock:lock-group_7
> localhost_12002 acquired lock:lock-group_4
> lockName acquired By
> ======================================
> lock-group_0 localhost_12001
> lock-group_1 localhost_12001
> lock-group_10 localhost_12002
> lock-group_11 localhost_12001
> lock-group_2 localhost_12001
> lock-group_3 localhost_12002
> lock-group_4 localhost_12002
> lock-group_5 localhost_12001
> lock-group_6 localhost_12002
> lock-group_7 localhost_12002
> lock-group_8 localhost_12001
> lock-group_9 localhost_12002
> ===Starting localhost_12000
> STARTING localhost_12000
> localhost_12000 acquired lock:lock-group_11
> localhost_12000 acquired lock:lock-group_0
> STARTED localhost_12000
> localhost_12000 acquired lock:lock-group_7
> localhost_12000 acquired lock:lock-group_4
> localhost_12001 releasing lock:lock-group_11
> localhost_12001 releasing lock:lock-group_0
> localhost_12002 releasing lock:lock-group_7
> localhost_12002 releasing lock:lock-group_4
> lockName acquired By
> ======================================
> lock-group_0 localhost_12000
> lock-group_1 localhost_12001
> lock-group_10 localhost_12002
> lock-group_11 localhost_12000
> lock-group_2 localhost_12001
> lock-group_3 localhost_12002
> lock-group_4 localhost_12000
> lock-group_5 localhost_12001
> lock-group_6 localhost_12002
> lock-group_7 localhost_12000
> lock-group_8 localhost_12001
> lock-group_9 localhost_12002
>
>
> On Sat, Feb 24, 2018 at 8:26 PM, Lei Xia <xi...@gmail.com> wrote:
>
>> Hi, Utsav
>>
>>   Delay rebalancer by default is disabled in cluster level (this is to
>> keep back-compatible somehow), you need to enable it in the clusterConfig,
>> e.g
>>
>> ConfigAccessor configAccessor = new ConfigAccessor(zkClient);
>> ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
>> clusterConfig.setDelayRebalaceEnabled(enabled);
>> configAccessor.setClusterConfig(clusterName, clusterConfig);
>>
>>
>>   Could you please have a try and let me know whether it works or not?
>> Thanks
>>
>>
>> Lei
>>
>>
>> On Fri, Feb 23, 2018 at 2:33 PM, Utsav Kanani <ut...@gmail.com>
>> wrote:
>>
>>> I am trying to expand the Lockmanager example http://helix.apache.
>>> org/0.6.2-incubating-docs/recipes/lock_manager.html to introduce delay
>>>
>>> tried doing something like this
>>> IdealState state = admin.getResourceIdealState(clusterName,
>>> lockGroupName);
>>>      state.setRebalanceDelay(100000);
>>>      state.setDelayRebalanceEnabled(true);
>>>      state.setRebalancerClassName(DelayedAutoRebalancer.class.ge
>>> tName());
>>>      admin.setResourceIdealState(clusterName, lockGroupName, state);
>>>      admin.rebalance(clusterName, lockGroupName, 1);
>>>
>>> On killing a node there is immediate rebalancing which takes place. I
>>> was hoping for a delay of 100 seconds before rebalancing but i am not
>>> seeing that behavior
>>>
>>>
>>> On Stopping localhost_12000 the locks are acquired immediately by
>>> localhost_12001 and localhost_12002
>>>
>>> on STARTING localhost_12000 the rebalance is again immediate.
>>>
>>> localhost_12000 acquired lock:lock-group_11
>>> localhost_12000 acquired lock:lock-group_7
>>> localhost_12000 acquired lock:lock-group_0
>>> localhost_12000 acquired lock:lock-group_4
>>> STARTED localhost_12000
>>> localhost_12001 releasing lock:lock-group_0
>>> localhost_12001 releasing lock:lock-group_11
>>> localhost_12002 releasing lock:lock-group_4
>>> localhost_12002 releasing lock:lock-group_7
>>>
>>>
>>> Here is the output
>>> =========================================
>>>
>>> STARTING localhost_12000
>>> STARTING localhost_12001
>>> STARTING localhost_12002
>>> STARTED localhost_12001
>>> STARTED localhost_12002
>>> STARTED localhost_12000
>>> localhost_12000 acquired lock:lock-group_11
>>> localhost_12002 acquired lock:lock-group_10
>>> localhost_12002 acquired lock:lock-group_9
>>> localhost_12002 acquired lock:lock-group_3
>>> localhost_12001 acquired lock:lock-group_2
>>> localhost_12001 acquired lock:lock-group_1
>>> localhost_12001 acquired lock:lock-group_8
>>> localhost_12002 acquired lock:lock-group_6
>>> localhost_12000 acquired lock:lock-group_4
>>> localhost_12000 acquired lock:lock-group_0
>>> localhost_12000 acquired lock:lock-group_7
>>> localhost_12001 acquired lock:lock-group_5
>>> lockName acquired By
>>> ======================================
>>> lock-group_0 localhost_12000
>>> lock-group_1 localhost_12001
>>> lock-group_10 localhost_12002
>>> lock-group_11 localhost_12000
>>> lock-group_2 localhost_12001
>>> lock-group_3 localhost_12002
>>> lock-group_4 localhost_12000
>>> lock-group_5 localhost_12001
>>> lock-group_6 localhost_12002
>>> lock-group_7 localhost_12000
>>> lock-group_8 localhost_12001
>>> lock-group_9 localhost_12002
>>> Stopping localhost_12000
>>> localhost_12000Interrupted
>>> localhost_12002 acquired lock:lock-group_4
>>> localhost_12001 acquired lock:lock-group_11
>>> localhost_12002 acquired lock:lock-group_7
>>> localhost_12001 acquired lock:lock-group_0
>>> lockName acquired By
>>> ======================================
>>> lock-group_0 localhost_12001
>>> lock-group_1 localhost_12001
>>> lock-group_10 localhost_12002
>>> lock-group_11 localhost_12001
>>> lock-group_2 localhost_12001
>>> lock-group_3 localhost_12002
>>> lock-group_4 localhost_12002
>>> lock-group_5 localhost_12001
>>> lock-group_6 localhost_12002
>>> lock-group_7 localhost_12002
>>> lock-group_8 localhost_12001
>>> lock-group_9 localhost_12002
>>> ===Starting localhost_12000
>>> STARTING localhost_12000
>>> localhost_12000 acquired lock:lock-group_11
>>> localhost_12000 acquired lock:lock-group_7
>>> localhost_12000 acquired lock:lock-group_0
>>> localhost_12000 acquired lock:lock-group_4
>>> STARTED localhost_12000
>>> localhost_12001 releasing lock:lock-group_0
>>> localhost_12001 releasing lock:lock-group_11
>>> localhost_12002 releasing lock:lock-group_4
>>> localhost_12002 releasing lock:lock-group_7
>>> lockName acquired By
>>> ======================================
>>> lock-group_0 localhost_12000
>>> lock-group_1 localhost_12001
>>> lock-group_10 localhost_12002
>>> lock-group_11 localhost_12000
>>> lock-group_2 localhost_12001
>>> lock-group_3 localhost_12002
>>> lock-group_4 localhost_12000
>>> lock-group_5 localhost_12001
>>> lock-group_6 localhost_12002
>>> lock-group_7 localhost_12000
>>> lock-group_8 localhost_12001
>>> lock-group_9 localhost_12002
>>>
>>
>>
>>
>> --
>> Lei Xia
>>
>
>
>
> On Sat, Feb 24, 2018 at 8:26 PM, Lei Xia <xi...@gmail.com> wrote:
>
>> Hi, Utsav
>>
>>   Delay rebalancer by default is disabled in cluster level (this is to
>> keep back-compatible somehow), you need to enable it in the clusterConfig,
>> e.g
>>
>> ConfigAccessor configAccessor = new ConfigAccessor(zkClient);
>> ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
>> clusterConfig.setDelayRebalaceEnabled(enabled);
>> configAccessor.setClusterConfig(clusterName, clusterConfig);
>>
>>
>>   Could you please have a try and let me know whether it works or not?
>> Thanks
>>
>>
>> Lei
>>
>>
>> On Fri, Feb 23, 2018 at 2:33 PM, Utsav Kanani <ut...@gmail.com>
>> wrote:
>>
>>> I am trying to expand the Lockmanager example
>>> http://helix.apache.org/0.6.2-incubating-docs/recipes/lock_manager.html
>>> to introduce delay
>>>
>>> tried doing something like this
>>> IdealState state = admin.getResourceIdealState(clusterName,
>>> lockGroupName);
>>>      state.setRebalanceDelay(100000);
>>>      state.setDelayRebalanceEnabled(true);
>>>      state.setRebalancerClassName(DelayedAutoRebalancer.class.ge
>>> tName());
>>>      admin.setResourceIdealState(clusterName, lockGroupName, state);
>>>      admin.rebalance(clusterName, lockGroupName, 1);
>>>
>>> On killing a node there is immediate rebalancing which takes place. I
>>> was hoping for a delay of 100 seconds before rebalancing but i am not
>>> seeing that behavior
>>>
>>>
>>> On Stopping localhost_12000 the locks are acquired immediately by
>>> localhost_12001 and localhost_12002
>>>
>>> on STARTING localhost_12000 the rebalance is again immediate.
>>>
>>> localhost_12000 acquired lock:lock-group_11
>>> localhost_12000 acquired lock:lock-group_7
>>> localhost_12000 acquired lock:lock-group_0
>>> localhost_12000 acquired lock:lock-group_4
>>> STARTED localhost_12000
>>> localhost_12001 releasing lock:lock-group_0
>>> localhost_12001 releasing lock:lock-group_11
>>> localhost_12002 releasing lock:lock-group_4
>>> localhost_12002 releasing lock:lock-group_7
>>>
>>>
>>> Here is the output
>>> =========================================
>>>
>>> STARTING localhost_12000
>>> STARTING localhost_12001
>>> STARTING localhost_12002
>>> STARTED localhost_12001
>>> STARTED localhost_12002
>>> STARTED localhost_12000
>>> localhost_12000 acquired lock:lock-group_11
>>> localhost_12002 acquired lock:lock-group_10
>>> localhost_12002 acquired lock:lock-group_9
>>> localhost_12002 acquired lock:lock-group_3
>>> localhost_12001 acquired lock:lock-group_2
>>> localhost_12001 acquired lock:lock-group_1
>>> localhost_12001 acquired lock:lock-group_8
>>> localhost_12002 acquired lock:lock-group_6
>>> localhost_12000 acquired lock:lock-group_4
>>> localhost_12000 acquired lock:lock-group_0
>>> localhost_12000 acquired lock:lock-group_7
>>> localhost_12001 acquired lock:lock-group_5
>>> lockName acquired By
>>> ======================================
>>> lock-group_0 localhost_12000
>>> lock-group_1 localhost_12001
>>> lock-group_10 localhost_12002
>>> lock-group_11 localhost_12000
>>> lock-group_2 localhost_12001
>>> lock-group_3 localhost_12002
>>> lock-group_4 localhost_12000
>>> lock-group_5 localhost_12001
>>> lock-group_6 localhost_12002
>>> lock-group_7 localhost_12000
>>> lock-group_8 localhost_12001
>>> lock-group_9 localhost_12002
>>> Stopping localhost_12000
>>> localhost_12000Interrupted
>>> localhost_12002 acquired lock:lock-group_4
>>> localhost_12001 acquired lock:lock-group_11
>>> localhost_12002 acquired lock:lock-group_7
>>> localhost_12001 acquired lock:lock-group_0
>>> lockName acquired By
>>> ======================================
>>> lock-group_0 localhost_12001
>>> lock-group_1 localhost_12001
>>> lock-group_10 localhost_12002
>>> lock-group_11 localhost_12001
>>> lock-group_2 localhost_12001
>>> lock-group_3 localhost_12002
>>> lock-group_4 localhost_12002
>>> lock-group_5 localhost_12001
>>> lock-group_6 localhost_12002
>>> lock-group_7 localhost_12002
>>> lock-group_8 localhost_12001
>>> lock-group_9 localhost_12002
>>> ===Starting localhost_12000
>>> STARTING localhost_12000
>>> localhost_12000 acquired lock:lock-group_11
>>> localhost_12000 acquired lock:lock-group_7
>>> localhost_12000 acquired lock:lock-group_0
>>> localhost_12000 acquired lock:lock-group_4
>>> STARTED localhost_12000
>>> localhost_12001 releasing lock:lock-group_0
>>> localhost_12001 releasing lock:lock-group_11
>>> localhost_12002 releasing lock:lock-group_4
>>> localhost_12002 releasing lock:lock-group_7
>>> lockName acquired By
>>> ======================================
>>> lock-group_0 localhost_12000
>>> lock-group_1 localhost_12001
>>> lock-group_10 localhost_12002
>>> lock-group_11 localhost_12000
>>> lock-group_2 localhost_12001
>>> lock-group_3 localhost_12002
>>> lock-group_4 localhost_12000
>>> lock-group_5 localhost_12001
>>> lock-group_6 localhost_12002
>>> lock-group_7 localhost_12000
>>> lock-group_8 localhost_12001
>>> lock-group_9 localhost_12002
>>>
>>
>>
>>
>> --
>> Lei Xia
>>
>
>


-- 
Lei Xia

Re: Unable to get Rebalance Delay to work using the distributed lock manager recipe

Posted by Utsav Kanani <ut...@gmail.com>.

Hi Lei,

That did not work
Seeing the same behavior
Added the following method to ZKHelixAdmin Class

public void enableClusterDelayMode(String clusterName) {
  ConfigAccessor configAccessor = new ConfigAccessor(_zkClient);
  ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
  clusterConfig.setDelayRebalaceEnabled(true);
  clusterConfig.setRebalanceDelayTime(100000);
  configAccessor.setClusterConfig(clusterName, clusterConfig);
}

and calling it in the demo class

HelixAdmin admin = new ZKHelixAdmin(zkAddress);
admin.addCluster(clusterName, true);
---->((ZKHelixAdmin)admin).enableClusterDelayMode(clusterName);
StateModelConfigGenerator generator = new StateModelConfigGenerator();
admin.addStateModelDef(clusterName, "OnlineOffline",
    new StateModelDefinition(generator.generateConfigForOnlineOffline()));

admin.addResource(clusterName, lockGroupName, numPartitions, "OnlineOffline",
    RebalanceMode.FULL_AUTO.toString());
admin.rebalance(clusterName, lockGroupName, 1);





STARTING localhost_12000
STARTING localhost_12001
STARTING localhost_12002
STARTED localhost_12000
STARTED localhost_12002
STARTED localhost_12001
localhost_12000 acquired lock:lock-group_0
localhost_12002 acquired lock:lock-group_3
localhost_12002 acquired lock:lock-group_9
localhost_12001 acquired lock:lock-group_2
localhost_12001 acquired lock:lock-group_5
localhost_12000 acquired lock:lock-group_11
localhost_12002 acquired lock:lock-group_6
localhost_12000 acquired lock:lock-group_7
localhost_12002 acquired lock:lock-group_10
localhost_12001 acquired lock:lock-group_8
localhost_12001 acquired lock:lock-group_1
localhost_12000 acquired lock:lock-group_4
lockName acquired By
======================================
lock-group_0 localhost_12000
lock-group_1 localhost_12001
lock-group_10 localhost_12002
lock-group_11 localhost_12000
lock-group_2 localhost_12001
lock-group_3 localhost_12002
lock-group_4 localhost_12000
lock-group_5 localhost_12001
lock-group_6 localhost_12002
lock-group_7 localhost_12000
lock-group_8 localhost_12001
lock-group_9 localhost_12002
Stopping localhost_12000
localhost_12000Interrupted
localhost_12001 acquired lock:lock-group_11
localhost_12001 acquired lock:lock-group_0
localhost_12002 acquired lock:lock-group_7
localhost_12002 acquired lock:lock-group_4
lockName acquired By
======================================
lock-group_0 localhost_12001
lock-group_1 localhost_12001
lock-group_10 localhost_12002
lock-group_11 localhost_12001
lock-group_2 localhost_12001
lock-group_3 localhost_12002
lock-group_4 localhost_12002
lock-group_5 localhost_12001
lock-group_6 localhost_12002
lock-group_7 localhost_12002
lock-group_8 localhost_12001
lock-group_9 localhost_12002
===Starting localhost_12000
STARTING localhost_12000
localhost_12000 acquired lock:lock-group_11
localhost_12000 acquired lock:lock-group_0
STARTED localhost_12000
localhost_12000 acquired lock:lock-group_7
localhost_12000 acquired lock:lock-group_4
localhost_12001 releasing lock:lock-group_11
localhost_12001 releasing lock:lock-group_0
localhost_12002 releasing lock:lock-group_7
localhost_12002 releasing lock:lock-group_4
lockName acquired By
======================================
lock-group_0 localhost_12000
lock-group_1 localhost_12001
lock-group_10 localhost_12002
lock-group_11 localhost_12000
lock-group_2 localhost_12001
lock-group_3 localhost_12002
lock-group_4 localhost_12000
lock-group_5 localhost_12001
lock-group_6 localhost_12002
lock-group_7 localhost_12000
lock-group_8 localhost_12001
lock-group_9 localhost_12002


On Sat, Feb 24, 2018 at 8:26 PM, Lei Xia <xi...@gmail.com> wrote:

> Hi, Utsav
>
>   Delay rebalancer by default is disabled in cluster level (this is to
> keep back-compatible somehow), you need to enable it in the clusterConfig,
> e.g
>
> ConfigAccessor configAccessor = new ConfigAccessor(zkClient);
> ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
> clusterConfig.setDelayRebalaceEnabled(enabled);
> configAccessor.setClusterConfig(clusterName, clusterConfig);
>
>
>   Could you please have a try and let me know whether it works or not?
> Thanks
>
>
> Lei
>
>
> On Fri, Feb 23, 2018 at 2:33 PM, Utsav Kanani <ut...@gmail.com>
> wrote:
>
>> I am trying to expand the Lockmanager example
>> http://helix.apache.org/0.6.2-incubating-docs/recipes/lock_manager.html to
>> introduce delay
>>
>> tried doing something like this
>> IdealState state = admin.getResourceIdealState(clusterName,
>> lockGroupName);
>>      state.setRebalanceDelay(100000);
>>      state.setDelayRebalanceEnabled(true);
>>      state.setRebalancerClassName(DelayedAutoRebalancer.class.getName());
>>      admin.setResourceIdealState(clusterName, lockGroupName, state);
>>      admin.rebalance(clusterName, lockGroupName, 1);
>>
>> On killing a node there is immediate rebalancing which takes place. I was
>> hoping for a delay of 100 seconds before rebalancing but i am not seeing
>> that behavior
>>
>>
>> On Stopping localhost_12000 the locks are acquired immediately by
>> localhost_12001 and localhost_12002
>>
>> on STARTING localhost_12000 the rebalance is again immediate.
>>
>> localhost_12000 acquired lock:lock-group_11
>> localhost_12000 acquired lock:lock-group_7
>> localhost_12000 acquired lock:lock-group_0
>> localhost_12000 acquired lock:lock-group_4
>> STARTED localhost_12000
>> localhost_12001 releasing lock:lock-group_0
>> localhost_12001 releasing lock:lock-group_11
>> localhost_12002 releasing lock:lock-group_4
>> localhost_12002 releasing lock:lock-group_7
>>
>>
>> Here is the output
>> =========================================
>>
>> STARTING localhost_12000
>> STARTING localhost_12001
>> STARTING localhost_12002
>> STARTED localhost_12001
>> STARTED localhost_12002
>> STARTED localhost_12000
>> localhost_12000 acquired lock:lock-group_11
>> localhost_12002 acquired lock:lock-group_10
>> localhost_12002 acquired lock:lock-group_9
>> localhost_12002 acquired lock:lock-group_3
>> localhost_12001 acquired lock:lock-group_2
>> localhost_12001 acquired lock:lock-group_1
>> localhost_12001 acquired lock:lock-group_8
>> localhost_12002 acquired lock:lock-group_6
>> localhost_12000 acquired lock:lock-group_4
>> localhost_12000 acquired lock:lock-group_0
>> localhost_12000 acquired lock:lock-group_7
>> localhost_12001 acquired lock:lock-group_5
>> lockName acquired By
>> ======================================
>> lock-group_0 localhost_12000
>> lock-group_1 localhost_12001
>> lock-group_10 localhost_12002
>> lock-group_11 localhost_12000
>> lock-group_2 localhost_12001
>> lock-group_3 localhost_12002
>> lock-group_4 localhost_12000
>> lock-group_5 localhost_12001
>> lock-group_6 localhost_12002
>> lock-group_7 localhost_12000
>> lock-group_8 localhost_12001
>> lock-group_9 localhost_12002
>> Stopping localhost_12000
>> localhost_12000Interrupted
>> localhost_12002 acquired lock:lock-group_4
>> localhost_12001 acquired lock:lock-group_11
>> localhost_12002 acquired lock:lock-group_7
>> localhost_12001 acquired lock:lock-group_0
>> lockName acquired By
>> ======================================
>> lock-group_0 localhost_12001
>> lock-group_1 localhost_12001
>> lock-group_10 localhost_12002
>> lock-group_11 localhost_12001
>> lock-group_2 localhost_12001
>> lock-group_3 localhost_12002
>> lock-group_4 localhost_12002
>> lock-group_5 localhost_12001
>> lock-group_6 localhost_12002
>> lock-group_7 localhost_12002
>> lock-group_8 localhost_12001
>> lock-group_9 localhost_12002
>> ===Starting localhost_12000
>> STARTING localhost_12000
>> localhost_12000 acquired lock:lock-group_11
>> localhost_12000 acquired lock:lock-group_7
>> localhost_12000 acquired lock:lock-group_0
>> localhost_12000 acquired lock:lock-group_4
>> STARTED localhost_12000
>> localhost_12001 releasing lock:lock-group_0
>> localhost_12001 releasing lock:lock-group_11
>> localhost_12002 releasing lock:lock-group_4
>> localhost_12002 releasing lock:lock-group_7
>> lockName acquired By
>> ======================================
>> lock-group_0 localhost_12000
>> lock-group_1 localhost_12001
>> lock-group_10 localhost_12002
>> lock-group_11 localhost_12000
>> lock-group_2 localhost_12001
>> lock-group_3 localhost_12002
>> lock-group_4 localhost_12000
>> lock-group_5 localhost_12001
>> lock-group_6 localhost_12002
>> lock-group_7 localhost_12000
>> lock-group_8 localhost_12001
>> lock-group_9 localhost_12002
>>
>
>
>
> --
> Lei Xia
>



On Sat, Feb 24, 2018 at 8:26 PM, Lei Xia <xi...@gmail.com> wrote:

> Hi, Utsav
>
>   Delay rebalancer by default is disabled in cluster level (this is to
> keep back-compatible somehow), you need to enable it in the clusterConfig,
> e.g
>
> ConfigAccessor configAccessor = new ConfigAccessor(zkClient);
> ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
> clusterConfig.setDelayRebalaceEnabled(enabled);
> configAccessor.setClusterConfig(clusterName, clusterConfig);
>
>
>   Could you please have a try and let me know whether it works or not?
> Thanks
>
>
> Lei
>
>
> On Fri, Feb 23, 2018 at 2:33 PM, Utsav Kanani <ut...@gmail.com>
> wrote:
>
>> I am trying to expand the Lockmanager example
>> http://helix.apache.org/0.6.2-incubating-docs/recipes/lock_manager.html
>> to introduce delay
>>
>> tried doing something like this
>> IdealState state = admin.getResourceIdealState(clusterName,
>> lockGroupName);
>>      state.setRebalanceDelay(100000);
>>      state.setDelayRebalanceEnabled(true);
>>      state.setRebalancerClassName(DelayedAutoRebalancer.class.getName());
>>      admin.setResourceIdealState(clusterName, lockGroupName, state);
>>      admin.rebalance(clusterName, lockGroupName, 1);
>>
>> On killing a node there is immediate rebalancing which takes place. I was
>> hoping for a delay of 100 seconds before rebalancing but i am not seeing
>> that behavior
>>
>>
>> On Stopping localhost_12000 the locks are acquired immediately by
>> localhost_12001 and localhost_12002
>>
>> on STARTING localhost_12000 the rebalance is again immediate.
>>
>> localhost_12000 acquired lock:lock-group_11
>> localhost_12000 acquired lock:lock-group_7
>> localhost_12000 acquired lock:lock-group_0
>> localhost_12000 acquired lock:lock-group_4
>> STARTED localhost_12000
>> localhost_12001 releasing lock:lock-group_0
>> localhost_12001 releasing lock:lock-group_11
>> localhost_12002 releasing lock:lock-group_4
>> localhost_12002 releasing lock:lock-group_7
>>
>>
>> Here is the output
>> =========================================
>>
>> STARTING localhost_12000
>> STARTING localhost_12001
>> STARTING localhost_12002
>> STARTED localhost_12001
>> STARTED localhost_12002
>> STARTED localhost_12000
>> localhost_12000 acquired lock:lock-group_11
>> localhost_12002 acquired lock:lock-group_10
>> localhost_12002 acquired lock:lock-group_9
>> localhost_12002 acquired lock:lock-group_3
>> localhost_12001 acquired lock:lock-group_2
>> localhost_12001 acquired lock:lock-group_1
>> localhost_12001 acquired lock:lock-group_8
>> localhost_12002 acquired lock:lock-group_6
>> localhost_12000 acquired lock:lock-group_4
>> localhost_12000 acquired lock:lock-group_0
>> localhost_12000 acquired lock:lock-group_7
>> localhost_12001 acquired lock:lock-group_5
>> lockName acquired By
>> ======================================
>> lock-group_0 localhost_12000
>> lock-group_1 localhost_12001
>> lock-group_10 localhost_12002
>> lock-group_11 localhost_12000
>> lock-group_2 localhost_12001
>> lock-group_3 localhost_12002
>> lock-group_4 localhost_12000
>> lock-group_5 localhost_12001
>> lock-group_6 localhost_12002
>> lock-group_7 localhost_12000
>> lock-group_8 localhost_12001
>> lock-group_9 localhost_12002
>> Stopping localhost_12000
>> localhost_12000Interrupted
>> localhost_12002 acquired lock:lock-group_4
>> localhost_12001 acquired lock:lock-group_11
>> localhost_12002 acquired lock:lock-group_7
>> localhost_12001 acquired lock:lock-group_0
>> lockName acquired By
>> ======================================
>> lock-group_0 localhost_12001
>> lock-group_1 localhost_12001
>> lock-group_10 localhost_12002
>> lock-group_11 localhost_12001
>> lock-group_2 localhost_12001
>> lock-group_3 localhost_12002
>> lock-group_4 localhost_12002
>> lock-group_5 localhost_12001
>> lock-group_6 localhost_12002
>> lock-group_7 localhost_12002
>> lock-group_8 localhost_12001
>> lock-group_9 localhost_12002
>> ===Starting localhost_12000
>> STARTING localhost_12000
>> localhost_12000 acquired lock:lock-group_11
>> localhost_12000 acquired lock:lock-group_7
>> localhost_12000 acquired lock:lock-group_0
>> localhost_12000 acquired lock:lock-group_4
>> STARTED localhost_12000
>> localhost_12001 releasing lock:lock-group_0
>> localhost_12001 releasing lock:lock-group_11
>> localhost_12002 releasing lock:lock-group_4
>> localhost_12002 releasing lock:lock-group_7
>> lockName acquired By
>> ======================================
>> lock-group_0 localhost_12000
>> lock-group_1 localhost_12001
>> lock-group_10 localhost_12002
>> lock-group_11 localhost_12000
>> lock-group_2 localhost_12001
>> lock-group_3 localhost_12002
>> lock-group_4 localhost_12000
>> lock-group_5 localhost_12001
>> lock-group_6 localhost_12002
>> lock-group_7 localhost_12000
>> lock-group_8 localhost_12001
>> lock-group_9 localhost_12002
>>
>
>
>
> --
> Lei Xia
>

Re: Unable to get Rebalance Delay to work using the distributed lock manager recipe

Posted by Lei Xia <xi...@gmail.com>.

Hi, Utsav

  Delay rebalancer by default is disabled in cluster level (this is to keep
back-compatible somehow), you need to enable it in the clusterConfig, e.g

ConfigAccessor configAccessor = new ConfigAccessor(zkClient);
ClusterConfig clusterConfig = configAccessor.getClusterConfig(clusterName);
clusterConfig.setDelayRebalaceEnabled(enabled);
configAccessor.setClusterConfig(clusterName, clusterConfig);


  Could you please have a try and let me know whether it works or not?
Thanks


Lei


On Fri, Feb 23, 2018 at 2:33 PM, Utsav Kanani <ut...@gmail.com>
wrote:

> I am trying to expand the Lockmanager example
> http://helix.apache.org/0.6.2-incubating-docs/recipes/lock_manager.html
> to introduce delay
>
> tried doing something like this
> IdealState state = admin.getResourceIdealState(clusterName,
> lockGroupName);
>      state.setRebalanceDelay(100000);
>      state.setDelayRebalanceEnabled(true);
>      state.setRebalancerClassName(DelayedAutoRebalancer.class.getName());
>      admin.setResourceIdealState(clusterName, lockGroupName, state);
>      admin.rebalance(clusterName, lockGroupName, 1);
>
> On killing a node there is immediate rebalancing which takes place. I was
> hoping for a delay of 100 seconds before rebalancing but i am not seeing
> that behavior
>
>
> On Stopping localhost_12000 the locks are acquired immediately by
> localhost_12001 and localhost_12002
>
> on STARTING localhost_12000 the rebalance is again immediate.
>
> localhost_12000 acquired lock:lock-group_11
> localhost_12000 acquired lock:lock-group_7
> localhost_12000 acquired lock:lock-group_0
> localhost_12000 acquired lock:lock-group_4
> STARTED localhost_12000
> localhost_12001 releasing lock:lock-group_0
> localhost_12001 releasing lock:lock-group_11
> localhost_12002 releasing lock:lock-group_4
> localhost_12002 releasing lock:lock-group_7
>
>
> Here is the output
> =========================================
>
> STARTING localhost_12000
> STARTING localhost_12001
> STARTING localhost_12002
> STARTED localhost_12001
> STARTED localhost_12002
> STARTED localhost_12000
> localhost_12000 acquired lock:lock-group_11
> localhost_12002 acquired lock:lock-group_10
> localhost_12002 acquired lock:lock-group_9
> localhost_12002 acquired lock:lock-group_3
> localhost_12001 acquired lock:lock-group_2
> localhost_12001 acquired lock:lock-group_1
> localhost_12001 acquired lock:lock-group_8
> localhost_12002 acquired lock:lock-group_6
> localhost_12000 acquired lock:lock-group_4
> localhost_12000 acquired lock:lock-group_0
> localhost_12000 acquired lock:lock-group_7
> localhost_12001 acquired lock:lock-group_5
> lockName acquired By
> ======================================
> lock-group_0 localhost_12000
> lock-group_1 localhost_12001
> lock-group_10 localhost_12002
> lock-group_11 localhost_12000
> lock-group_2 localhost_12001
> lock-group_3 localhost_12002
> lock-group_4 localhost_12000
> lock-group_5 localhost_12001
> lock-group_6 localhost_12002
> lock-group_7 localhost_12000
> lock-group_8 localhost_12001
> lock-group_9 localhost_12002
> Stopping localhost_12000
> localhost_12000Interrupted
> localhost_12002 acquired lock:lock-group_4
> localhost_12001 acquired lock:lock-group_11
> localhost_12002 acquired lock:lock-group_7
> localhost_12001 acquired lock:lock-group_0
> lockName acquired By
> ======================================
> lock-group_0 localhost_12001
> lock-group_1 localhost_12001
> lock-group_10 localhost_12002
> lock-group_11 localhost_12001
> lock-group_2 localhost_12001
> lock-group_3 localhost_12002
> lock-group_4 localhost_12002
> lock-group_5 localhost_12001
> lock-group_6 localhost_12002
> lock-group_7 localhost_12002
> lock-group_8 localhost_12001
> lock-group_9 localhost_12002
> ===Starting localhost_12000
> STARTING localhost_12000
> localhost_12000 acquired lock:lock-group_11
> localhost_12000 acquired lock:lock-group_7
> localhost_12000 acquired lock:lock-group_0
> localhost_12000 acquired lock:lock-group_4
> STARTED localhost_12000
> localhost_12001 releasing lock:lock-group_0
> localhost_12001 releasing lock:lock-group_11
> localhost_12002 releasing lock:lock-group_4
> localhost_12002 releasing lock:lock-group_7
> lockName acquired By
> ======================================
> lock-group_0 localhost_12000
> lock-group_1 localhost_12001
> lock-group_10 localhost_12002
> lock-group_11 localhost_12000
> lock-group_2 localhost_12001
> lock-group_3 localhost_12002
> lock-group_4 localhost_12000
> lock-group_5 localhost_12001
> lock-group_6 localhost_12002
> lock-group_7 localhost_12000
> lock-group_8 localhost_12001
> lock-group_9 localhost_12002
>



-- 
Lei Xia