You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@helix.apache.org by Vinoth Chandar <vi...@uber.com> on 2015/02/17 02:22:34 UTC

Changing the number of partitions in a resource

Hey guys,

I have a rather unusual case, where the number of partitions in my resource
could change dynamically.. Is there a way to handle this without dropping
and adding the resource again?

What kind of cleanup should I do if I drop and re-add? (e.g.: reinit state
model objects?)..

Thanks
Vinoth

Re: Changing the number of partitions in a resource

Posted by Vinoth Chandar <vi...@uber.com>.

Thanks guys.. I will try that out..

On Tue, Feb 17, 2015 at 10:16 AM, Zhen Zhang <zz...@linkedin.com> wrote:

>  Hi Vinoth, you can use HelixAdmin#setResourceIdealState(), passing the
> new ideal-state.
>
>  Thanks,
> Jason
>
>  ------------------------------
> *From:* Vinoth Chandar [vinoth@uber.com]
> *Sent:* Monday, February 16, 2015 5:54 PM
> *To:* user
> *Subject:* Re: Changing the number of partitions in a resource
>
>   Thanks for the response, Kishore.
>
>  I am basically mimicking the rabbitmq consumer group example..
>
>  helixAdmin.addResource(CLUSTER_NAME, RESOURCE_NAME, 10, STATE_MODEL_NAME,
>       RebalanceMode.FULL_AUTO.toString)
>
>  So, I simple do another addResource(..) call with 15 instead of 10?
>
>  Thanks
> Vinoth
>
>
>
>
> On Mon, Feb 16, 2015 at 5:41 PM, kishore g <g....@gmail.com> wrote:
>
>> Hi Vinoth,
>>
>>  The number of partitions can be changed dynamically. The changes
>> required from your side depends on idealstate mode.
>>
>>  FULL_AUTO
>> -- Simply change the number of partitions. Helix should assign the new
>> partition to one of the nodes automatically. Lets say you have 10
>> partitions, if you change the number of partitions to 15. Partitions 11 to
>> 15 will be distributed among the nodes.
>>
>>  -- SEMI_AUTO
>> -- In this case, you have to first change the number of partitions and
>> invoke the rebalance command. This will rebalance the partitions. There
>> should be an option to use the existing assignment so that shuffling of
>> partitions is minimized.
>>
>>  -- CUSTOM
>> -- Change the ideal state either manually or dynamically using user
>> defined rebalancer
>>
>>  Hope that helps
>> thanks,
>> Kishore G
>>
>>
>> On Mon, Feb 16, 2015 at 5:22 PM, Vinoth Chandar <vi...@uber.com> wrote:
>>
>>> Hey guys,
>>>
>>>  I have a rather unusual case, where the number of partitions in my
>>> resource could change dynamically.. Is there a way to handle this without
>>> dropping and adding the resource again?
>>>
>>>  What kind of cleanup should I do if I drop and re-add? (e.g.: reinit
>>> state model objects?)..
>>>
>>>  Thanks
>>>  Vinoth
>>>
>>>
>>>
>>
>

RE: Changing the number of partitions in a resource

Posted by Zhen Zhang <zz...@linkedin.com>.

Hi Vinoth, you can use HelixAdmin#setResourceIdealState(), passing the new ideal-state.

Thanks,
Jason

________________________________
From: Vinoth Chandar [vinoth@uber.com]
Sent: Monday, February 16, 2015 5:54 PM
To: user
Subject: Re: Changing the number of partitions in a resource

Thanks for the response, Kishore.

I am basically mimicking the rabbitmq consumer group example..

helixAdmin.addResource(CLUSTER_NAME, RESOURCE_NAME, 10, STATE_MODEL_NAME,
      RebalanceMode.FULL_AUTO.toString)

So, I simple do another addResource(..) call with 15 instead of 10?

Thanks
Vinoth

On Mon, Feb 16, 2015 at 5:41 PM, kishore g <g....@gmail.com>> wrote:
Hi Vinoth,

The number of partitions can be changed dynamically. The changes required from your side depends on idealstate mode.

FULL_AUTO
-- Simply change the number of partitions. Helix should assign the new partition to one of the nodes automatically. Lets say you have 10 partitions, if you change the number of partitions to 15. Partitions 11 to 15 will be distributed among the nodes.

-- SEMI_AUTO
-- In this case, you have to first change the number of partitions and invoke the rebalance command. This will rebalance the partitions. There should be an option to use the existing assignment so that shuffling of partitions is minimized.

-- CUSTOM
-- Change the ideal state either manually or dynamically using user defined rebalancer

Hope that helps
thanks,
Kishore G

On Mon, Feb 16, 2015 at 5:22 PM, Vinoth Chandar <vi...@uber.com>> wrote:
Hey guys,

I have a rather unusual case, where the number of partitions in my resource could change dynamically.. Is there a way to handle this without dropping and adding the resource again?

What kind of cleanup should I do if I drop and re-add? (e.g.: reinit state model objects?)..

Thanks
Vinoth

Re: Changing the number of partitions in a resource

Posted by Vinoth Chandar <vi...@uber.com>.

Thanks for the response, Kishore.

I am basically mimicking the rabbitmq consumer group example..

helixAdmin.addResource(CLUSTER_NAME, RESOURCE_NAME, 10, STATE_MODEL_NAME,
      RebalanceMode.FULL_AUTO.toString)

So, I simple do another addResource(..) call with 15 instead of 10?

Thanks
Vinoth




On Mon, Feb 16, 2015 at 5:41 PM, kishore g <g....@gmail.com> wrote:

> Hi Vinoth,
>
> The number of partitions can be changed dynamically. The changes required
> from your side depends on idealstate mode.
>
> FULL_AUTO
> -- Simply change the number of partitions. Helix should assign the new
> partition to one of the nodes automatically. Lets say you have 10
> partitions, if you change the number of partitions to 15. Partitions 11 to
> 15 will be distributed among the nodes.
>
> -- SEMI_AUTO
> -- In this case, you have to first change the number of partitions and
> invoke the rebalance command. This will rebalance the partitions. There
> should be an option to use the existing assignment so that shuffling of
> partitions is minimized.
>
> -- CUSTOM
> -- Change the ideal state either manually or dynamically using user
> defined rebalancer
>
> Hope that helps
> thanks,
> Kishore G
>
>
> On Mon, Feb 16, 2015 at 5:22 PM, Vinoth Chandar <vi...@uber.com> wrote:
>
>> Hey guys,
>>
>> I have a rather unusual case, where the number of partitions in my
>> resource could change dynamically.. Is there a way to handle this without
>> dropping and adding the resource again?
>>
>> What kind of cleanup should I do if I drop and re-add? (e.g.: reinit
>> state model objects?)..
>>
>> Thanks
>> Vinoth
>>
>>
>>
>

Re: Changing the number of partitions in a resource

Posted by kishore g <g....@gmail.com>.

Hi Vinoth,

The number of partitions can be changed dynamically. The changes required
from your side depends on idealstate mode.

FULL_AUTO
-- Simply change the number of partitions. Helix should assign the new
partition to one of the nodes automatically. Lets say you have 10
partitions, if you change the number of partitions to 15. Partitions 11 to
15 will be distributed among the nodes.

-- SEMI_AUTO
-- In this case, you have to first change the number of partitions and
invoke the rebalance command. This will rebalance the partitions. There
should be an option to use the existing assignment so that shuffling of
partitions is minimized.

-- CUSTOM
-- Change the ideal state either manually or dynamically using user defined
rebalancer

Hope that helps
thanks,
Kishore G


On Mon, Feb 16, 2015 at 5:22 PM, Vinoth Chandar <vi...@uber.com> wrote:

> Hey guys,
>
> I have a rather unusual case, where the number of partitions in my
> resource could change dynamically.. Is there a way to handle this without
> dropping and adding the resource again?
>
> What kind of cleanup should I do if I drop and re-add? (e.g.: reinit state
> model objects?)..
>
> Thanks
> Vinoth
>
>
>