Dryrun mode or staging mode for updating idealState

[Hang Qi <ha...@gmail.com>]

Hi,

We are using the SEMI_AUTO mode as we want to control where to place the
partitions. And we change the ideal state when we need to perform some
operations, like moving partitions or increasing partitions.

So my question is that could we have a dryrun mode or staging mode for
updating ideal state, which only shows corresponding state transitions
without actually executing them? The reason I am asking for it is that once
the ideal state has been updated and the controller has been sent out the
transition messages, it is hard to stop them/revoke them.

What do you think?

Thanks
Hang Qi

Re: Dryrun mode or staging mode for updating idealState

[Hang Qi <ha...@gmail.com>]

Hi Kishore,

My thought was trying to reuse the logic of controller.

Controller is designed to use one general pipeline rebalancePipeline to
handle various changes, including configuration changes, membership
changes, and transition message completion changes, etc. Basically, I was
thinking to have a separate copy of rebalancePipeline pipeline and change
the last stage from sending out messages to participants to sending back
messages to issuer (helix-admin), and using a separate path to store the
staging idealState.

Also I think it is reasonable to make the assumption that only one
helix-admin is running, thus there should have no other configuration
changes happened concurrently.

However, there are still some parts I am not sure about

   - How to move the pipeline forward?
      - Originally, the pipeline will be triggered by previous messages
      completion to generate new messages for next step. For staging
ideal state,
      we do not send out messages to participants, though we can mark those
      messages are done in helix-admin, but how to deal with other ongoing
      changes, some node may go down or back to normal when handling staging
      ideal state? Another option is only compute the first step, that may be
      enough?
   - How to communicate between helix-admin and controller?
      - Currently helix relies on ZK as a communication channel. Given that
      4M data per node limitation of ZK, we may need extra work to passing all
      the messages from controller to helix-admin

Thanks
Hang Qi

On Fri, May 22, 2015 at 4:31 PM, kishore g <g....@gmail.com> wrote:

> yes, thats a good idea. Do you have any suggestions on how we can go about
> implementing it?
>
> On Fri, May 22, 2015 at 4:16 PM, Hang Qi <ha...@gmail.com> wrote:
>
>> Hi,
>>
>> We are using the SEMI_AUTO mode as we want to control where to place the
>> partitions. And we change the ideal state when we need to perform some
>> operations, like moving partitions or increasing partitions.
>>
>> So my question is that could we have a dryrun mode or staging mode for
>> updating ideal state, which only shows corresponding state transitions
>> without actually executing them? The reason I am asking for it is that once
>> the ideal state has been updated and the controller has been sent out the
>> transition messages, it is hard to stop them/revoke them.
>>
>> What do you think?
>>
>> Thanks
>> Hang Qi
>>
>
>


-- 
Qi hang

Re: Dryrun mode or staging mode for updating idealState

[kishore g <g....@gmail.com>]

yes, thats a good idea. Do you have any suggestions on how we can go about
implementing it?

On Fri, May 22, 2015 at 4:16 PM, Hang Qi <ha...@gmail.com> wrote:

> Hi,
>
> We are using the SEMI_AUTO mode as we want to control where to place the
> partitions. And we change the ideal state when we need to perform some
> operations, like moving partitions or increasing partitions.
>
> So my question is that could we have a dryrun mode or staging mode for
> updating ideal state, which only shows corresponding state transitions
> without actually executing them? The reason I am asking for it is that once
> the ideal state has been updated and the controller has been sent out the
> transition messages, it is hard to stop them/revoke them.
>
> What do you think?
>
> Thanks
> Hang Qi
>