You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@helix.apache.org by Sandeep Nayak <os...@gmail.com> on 2014/02/21 06:58:24 UTC
Helix 0.6.2: Resetting or removing a resource
Hi guys,
Wanted to check what is the correct API for removing a resource so
that it is unavailable on any of the nodes?
I see HelixAdmin.dropResource and HelixAdmin.resetResource. What is
the right API to use to remove a resource entirely from the cluster?
Thanks,
Sandeep
Re: Helix 0.6.2: Resetting or removing a resource
Posted by kishore g <g....@gmail.com>.
Yes, the state transition method is invoked when a resource X is assigned
to B. Available on IRC if you need help.
On Thu, Feb 20, 2014 at 11:39 PM, Sandeep Nayak <os...@gmail.com> wrote:
> Sorry to keep using the same thread but is there a callback which gets
> invoked when say a resource is added?
>
> So example
>
> T1: 3 nodes : A, B and C
> T2: resource X gets added on A with 1 partition and 2 replicas
> T3: resource X is available on say A and B
>
> Is there a callback that can be registered on B so that the statemodel
> or some object will receive a message when X is added to B?
>
>
> On Thu, Feb 20, 2014 at 11:31 PM, kishore g <g....@gmail.com> wrote:
> > Agree with that, there are some cases where the api's need to be async.
> We
> > have taskScheduler api that is well suited for such usecases. However, I
> > think there are some gaps in the design/ implementation. Fixing that will
> > allow us to easily support such operations.
> >
> > Yes, agree the API needs to be marked to throw a proper exception.
> >
> > thanks,
> > Kishore G
> >
> >
> > On Thu, Feb 20, 2014 at 11:17 PM, Sandeep Nayak <os...@gmail.com>
> wrote:
> >>
> >> The dropResource works I imagine but adding the resource back
> >> immediately does not. I will try and play with delays to see if the
> >> cleanup works, currently I catch HelixException and ignore it.
> >>
> >> Thinking out aloud, I imagine a call to drop should return after some
> >> internal state tracks that the resource is removed and maybe queues a
> >> task for background propagation/removal across nodes. I imagine the
> >> task should fall on a queue/executor, a subsequent task would be for
> >> resource add. But both calls to remove and add should return as soon
> >> as the internal state is tracked to mark the state correctly.
> >> Eventually I imagine the remove task will execute on the background
> >> and upon acks from all other participants move on to the next task
> >> i.e. the tasks execute sequentially.
> >>
> >> Apologies on the above blurb, just sharing my thoughts I am sure you
> >> guys are well aware of what I am talking of.
> >>
> >> On a separate note maybe this API also needs to be marked on a ticket
> >> to throw a proper exception.
> >>
> >> Thanks,
> >>
> >> Sandeep
> >>
> >> On Thu, Feb 20, 2014 at 11:05 PM, kishore g <g....@gmail.com>
> wrote:
> >> > Great! Thanks for your patience, I know it was not easy. We really
> need
> >> > lot
> >> > more work on documentation. Suggestions/feedback/contributions are
> >> > welcome.
> >> >
> >> > Did dropResource succeed, can you wait until it dissappears from
> >> > idealstates
> >> > and external view. You might want to give some time for the resource
> to
> >> > be
> >> > cleaned up. We can probably add additional safety check to addResource
> >> > back
> >> > when the resources are still active.
> >> >
> >> > thanks,
> >> > Kishore G
> >> >
> >> >
> >> > On Thu, Feb 20, 2014 at 10:16 PM, Sandeep Nayak <os...@gmail.com>
> >> > wrote:
> >> >>
> >> >> Btw other than the remove, I have the resources working and
> >> >> distributing the way I want. So just wanted to say good work, it
> works
> >> >> nicely :-)
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Sandeep
> >> >>
> >> >> On Thu, Feb 20, 2014 at 10:06 PM, Sandeep Nayak <os...@gmail.com>
> >> >> wrote:
> >> >> > Does it need time to remove the resource? I tried dropResource and
> >> >> > the
> >> >> > very next call I invoke addResource to see if I can add it again
> and
> >> >> > the add call fails with the exception
> >> >> >
> >> >> > org.apache.helix.HelixException: Skip the operation. Resource ideal
> >> >> > state directory already exists:/cluster/IDEALSTATES/resource-name
> >> >> >
> >> >> > On Thu, Feb 20, 2014 at 10:03 PM, Kanak Biscuitwala
> >> >> > <ka...@hotmail.com> wrote:
> >> >> >> Hi Sandeep,
> >> >> >>
> >> >> >> dropResource is correct. resetResource takes resources in error
> >> >> >> state
> >> >> >> and resets them to the initial state.
> >> >> >>
> >> >> >> Kanak
> >> >> >> ----------------------------------------
> >> >> >>> Date: Thu, 20 Feb 2014 21:58:24 -0800
> >> >> >>> Subject: Helix 0.6.2: Resetting or removing a resource
> >> >> >>> From: osgigeek@gmail.com
> >> >> >>> To: user@helix.apache.org
> >> >> >>>
> >> >> >>> Hi guys,
> >> >> >>>
> >> >> >>> Wanted to check what is the correct API for removing a resource
> so
> >> >> >>> that it is unavailable on any of the nodes?
> >> >> >>>
> >> >> >>> I see HelixAdmin.dropResource and HelixAdmin.resetResource. What
> is
> >> >> >>> the right API to use to remove a resource entirely from the
> >> >> >>> cluster?
> >> >> >>>
> >> >> >>> Thanks,
> >> >> >>>
> >> >> >>> Sandeep
> >> >
> >> >
> >
> >
>
Re: Helix 0.6.2: Resetting or removing a resource
Posted by Sandeep Nayak <os...@gmail.com>.
Sorry to keep using the same thread but is there a callback which gets
invoked when say a resource is added?
So example
T1: 3 nodes : A, B and C
T2: resource X gets added on A with 1 partition and 2 replicas
T3: resource X is available on say A and B
Is there a callback that can be registered on B so that the statemodel
or some object will receive a message when X is added to B?
On Thu, Feb 20, 2014 at 11:31 PM, kishore g <g....@gmail.com> wrote:
> Agree with that, there are some cases where the api's need to be async. We
> have taskScheduler api that is well suited for such usecases. However, I
> think there are some gaps in the design/ implementation. Fixing that will
> allow us to easily support such operations.
>
> Yes, agree the API needs to be marked to throw a proper exception.
>
> thanks,
> Kishore G
>
>
> On Thu, Feb 20, 2014 at 11:17 PM, Sandeep Nayak <os...@gmail.com> wrote:
>>
>> The dropResource works I imagine but adding the resource back
>> immediately does not. I will try and play with delays to see if the
>> cleanup works, currently I catch HelixException and ignore it.
>>
>> Thinking out aloud, I imagine a call to drop should return after some
>> internal state tracks that the resource is removed and maybe queues a
>> task for background propagation/removal across nodes. I imagine the
>> task should fall on a queue/executor, a subsequent task would be for
>> resource add. But both calls to remove and add should return as soon
>> as the internal state is tracked to mark the state correctly.
>> Eventually I imagine the remove task will execute on the background
>> and upon acks from all other participants move on to the next task
>> i.e. the tasks execute sequentially.
>>
>> Apologies on the above blurb, just sharing my thoughts I am sure you
>> guys are well aware of what I am talking of.
>>
>> On a separate note maybe this API also needs to be marked on a ticket
>> to throw a proper exception.
>>
>> Thanks,
>>
>> Sandeep
>>
>> On Thu, Feb 20, 2014 at 11:05 PM, kishore g <g....@gmail.com> wrote:
>> > Great! Thanks for your patience, I know it was not easy. We really need
>> > lot
>> > more work on documentation. Suggestions/feedback/contributions are
>> > welcome.
>> >
>> > Did dropResource succeed, can you wait until it dissappears from
>> > idealstates
>> > and external view. You might want to give some time for the resource to
>> > be
>> > cleaned up. We can probably add additional safety check to addResource
>> > back
>> > when the resources are still active.
>> >
>> > thanks,
>> > Kishore G
>> >
>> >
>> > On Thu, Feb 20, 2014 at 10:16 PM, Sandeep Nayak <os...@gmail.com>
>> > wrote:
>> >>
>> >> Btw other than the remove, I have the resources working and
>> >> distributing the way I want. So just wanted to say good work, it works
>> >> nicely :-)
>> >>
>> >> Thanks,
>> >>
>> >> Sandeep
>> >>
>> >> On Thu, Feb 20, 2014 at 10:06 PM, Sandeep Nayak <os...@gmail.com>
>> >> wrote:
>> >> > Does it need time to remove the resource? I tried dropResource and
>> >> > the
>> >> > very next call I invoke addResource to see if I can add it again and
>> >> > the add call fails with the exception
>> >> >
>> >> > org.apache.helix.HelixException: Skip the operation. Resource ideal
>> >> > state directory already exists:/cluster/IDEALSTATES/resource-name
>> >> >
>> >> > On Thu, Feb 20, 2014 at 10:03 PM, Kanak Biscuitwala
>> >> > <ka...@hotmail.com> wrote:
>> >> >> Hi Sandeep,
>> >> >>
>> >> >> dropResource is correct. resetResource takes resources in error
>> >> >> state
>> >> >> and resets them to the initial state.
>> >> >>
>> >> >> Kanak
>> >> >> ----------------------------------------
>> >> >>> Date: Thu, 20 Feb 2014 21:58:24 -0800
>> >> >>> Subject: Helix 0.6.2: Resetting or removing a resource
>> >> >>> From: osgigeek@gmail.com
>> >> >>> To: user@helix.apache.org
>> >> >>>
>> >> >>> Hi guys,
>> >> >>>
>> >> >>> Wanted to check what is the correct API for removing a resource so
>> >> >>> that it is unavailable on any of the nodes?
>> >> >>>
>> >> >>> I see HelixAdmin.dropResource and HelixAdmin.resetResource. What is
>> >> >>> the right API to use to remove a resource entirely from the
>> >> >>> cluster?
>> >> >>>
>> >> >>> Thanks,
>> >> >>>
>> >> >>> Sandeep
>> >
>> >
>
>
Re: Helix 0.6.2: Resetting or removing a resource
Posted by kishore g <g....@gmail.com>.
Agree with that, there are some cases where the api's need to be async. We
have taskScheduler api that is well suited for such usecases. However, I
think there are some gaps in the design/ implementation. Fixing that will
allow us to easily support such operations.
Yes, agree the API needs to be marked to throw a proper exception.
thanks,
Kishore G
On Thu, Feb 20, 2014 at 11:17 PM, Sandeep Nayak <os...@gmail.com> wrote:
> The dropResource works I imagine but adding the resource back
> immediately does not. I will try and play with delays to see if the
> cleanup works, currently I catch HelixException and ignore it.
>
> Thinking out aloud, I imagine a call to drop should return after some
> internal state tracks that the resource is removed and maybe queues a
> task for background propagation/removal across nodes. I imagine the
> task should fall on a queue/executor, a subsequent task would be for
> resource add. But both calls to remove and add should return as soon
> as the internal state is tracked to mark the state correctly.
> Eventually I imagine the remove task will execute on the background
> and upon acks from all other participants move on to the next task
> i.e. the tasks execute sequentially.
>
> Apologies on the above blurb, just sharing my thoughts I am sure you
> guys are well aware of what I am talking of.
>
> On a separate note maybe this API also needs to be marked on a ticket
> to throw a proper exception.
>
> Thanks,
>
> Sandeep
>
> On Thu, Feb 20, 2014 at 11:05 PM, kishore g <g....@gmail.com> wrote:
> > Great! Thanks for your patience, I know it was not easy. We really need
> lot
> > more work on documentation. Suggestions/feedback/contributions are
> welcome.
> >
> > Did dropResource succeed, can you wait until it dissappears from
> idealstates
> > and external view. You might want to give some time for the resource to
> be
> > cleaned up. We can probably add additional safety check to addResource
> back
> > when the resources are still active.
> >
> > thanks,
> > Kishore G
> >
> >
> > On Thu, Feb 20, 2014 at 10:16 PM, Sandeep Nayak <os...@gmail.com>
> wrote:
> >>
> >> Btw other than the remove, I have the resources working and
> >> distributing the way I want. So just wanted to say good work, it works
> >> nicely :-)
> >>
> >> Thanks,
> >>
> >> Sandeep
> >>
> >> On Thu, Feb 20, 2014 at 10:06 PM, Sandeep Nayak <os...@gmail.com>
> >> wrote:
> >> > Does it need time to remove the resource? I tried dropResource and the
> >> > very next call I invoke addResource to see if I can add it again and
> >> > the add call fails with the exception
> >> >
> >> > org.apache.helix.HelixException: Skip the operation. Resource ideal
> >> > state directory already exists:/cluster/IDEALSTATES/resource-name
> >> >
> >> > On Thu, Feb 20, 2014 at 10:03 PM, Kanak Biscuitwala
> >> > <ka...@hotmail.com> wrote:
> >> >> Hi Sandeep,
> >> >>
> >> >> dropResource is correct. resetResource takes resources in error state
> >> >> and resets them to the initial state.
> >> >>
> >> >> Kanak
> >> >> ----------------------------------------
> >> >>> Date: Thu, 20 Feb 2014 21:58:24 -0800
> >> >>> Subject: Helix 0.6.2: Resetting or removing a resource
> >> >>> From: osgigeek@gmail.com
> >> >>> To: user@helix.apache.org
> >> >>>
> >> >>> Hi guys,
> >> >>>
> >> >>> Wanted to check what is the correct API for removing a resource so
> >> >>> that it is unavailable on any of the nodes?
> >> >>>
> >> >>> I see HelixAdmin.dropResource and HelixAdmin.resetResource. What is
> >> >>> the right API to use to remove a resource entirely from the cluster?
> >> >>>
> >> >>> Thanks,
> >> >>>
> >> >>> Sandeep
> >
> >
>
Re: Helix 0.6.2: Resetting or removing a resource
Posted by Sandeep Nayak <os...@gmail.com>.
The dropResource works I imagine but adding the resource back
immediately does not. I will try and play with delays to see if the
cleanup works, currently I catch HelixException and ignore it.
Thinking out aloud, I imagine a call to drop should return after some
internal state tracks that the resource is removed and maybe queues a
task for background propagation/removal across nodes. I imagine the
task should fall on a queue/executor, a subsequent task would be for
resource add. But both calls to remove and add should return as soon
as the internal state is tracked to mark the state correctly.
Eventually I imagine the remove task will execute on the background
and upon acks from all other participants move on to the next task
i.e. the tasks execute sequentially.
Apologies on the above blurb, just sharing my thoughts I am sure you
guys are well aware of what I am talking of.
On a separate note maybe this API also needs to be marked on a ticket
to throw a proper exception.
Thanks,
Sandeep
On Thu, Feb 20, 2014 at 11:05 PM, kishore g <g....@gmail.com> wrote:
> Great! Thanks for your patience, I know it was not easy. We really need lot
> more work on documentation. Suggestions/feedback/contributions are welcome.
>
> Did dropResource succeed, can you wait until it dissappears from idealstates
> and external view. You might want to give some time for the resource to be
> cleaned up. We can probably add additional safety check to addResource back
> when the resources are still active.
>
> thanks,
> Kishore G
>
>
> On Thu, Feb 20, 2014 at 10:16 PM, Sandeep Nayak <os...@gmail.com> wrote:
>>
>> Btw other than the remove, I have the resources working and
>> distributing the way I want. So just wanted to say good work, it works
>> nicely :-)
>>
>> Thanks,
>>
>> Sandeep
>>
>> On Thu, Feb 20, 2014 at 10:06 PM, Sandeep Nayak <os...@gmail.com>
>> wrote:
>> > Does it need time to remove the resource? I tried dropResource and the
>> > very next call I invoke addResource to see if I can add it again and
>> > the add call fails with the exception
>> >
>> > org.apache.helix.HelixException: Skip the operation. Resource ideal
>> > state directory already exists:/cluster/IDEALSTATES/resource-name
>> >
>> > On Thu, Feb 20, 2014 at 10:03 PM, Kanak Biscuitwala
>> > <ka...@hotmail.com> wrote:
>> >> Hi Sandeep,
>> >>
>> >> dropResource is correct. resetResource takes resources in error state
>> >> and resets them to the initial state.
>> >>
>> >> Kanak
>> >> ----------------------------------------
>> >>> Date: Thu, 20 Feb 2014 21:58:24 -0800
>> >>> Subject: Helix 0.6.2: Resetting or removing a resource
>> >>> From: osgigeek@gmail.com
>> >>> To: user@helix.apache.org
>> >>>
>> >>> Hi guys,
>> >>>
>> >>> Wanted to check what is the correct API for removing a resource so
>> >>> that it is unavailable on any of the nodes?
>> >>>
>> >>> I see HelixAdmin.dropResource and HelixAdmin.resetResource. What is
>> >>> the right API to use to remove a resource entirely from the cluster?
>> >>>
>> >>> Thanks,
>> >>>
>> >>> Sandeep
>
>
Re: Helix 0.6.2: Resetting or removing a resource
Posted by kishore g <g....@gmail.com>.
Great! Thanks for your patience, I know it was not easy. We really need
lot more work on documentation. Suggestions/feedback/contributions are
welcome.
Did dropResource succeed, can you wait until it dissappears from
idealstates and external view. You might want to give some time for the
resource to be cleaned up. We can probably add additional safety check to
addResource back when the resources are still active.
thanks,
Kishore G
On Thu, Feb 20, 2014 at 10:16 PM, Sandeep Nayak <os...@gmail.com> wrote:
> Btw other than the remove, I have the resources working and
> distributing the way I want. So just wanted to say good work, it works
> nicely :-)
>
> Thanks,
>
> Sandeep
>
> On Thu, Feb 20, 2014 at 10:06 PM, Sandeep Nayak <os...@gmail.com>
> wrote:
> > Does it need time to remove the resource? I tried dropResource and the
> > very next call I invoke addResource to see if I can add it again and
> > the add call fails with the exception
> >
> > org.apache.helix.HelixException: Skip the operation. Resource ideal
> > state directory already exists:/cluster/IDEALSTATES/resource-name
> >
> > On Thu, Feb 20, 2014 at 10:03 PM, Kanak Biscuitwala <ka...@hotmail.com>
> wrote:
> >> Hi Sandeep,
> >>
> >> dropResource is correct. resetResource takes resources in error state
> and resets them to the initial state.
> >>
> >> Kanak
> >> ----------------------------------------
> >>> Date: Thu, 20 Feb 2014 21:58:24 -0800
> >>> Subject: Helix 0.6.2: Resetting or removing a resource
> >>> From: osgigeek@gmail.com
> >>> To: user@helix.apache.org
> >>>
> >>> Hi guys,
> >>>
> >>> Wanted to check what is the correct API for removing a resource so
> >>> that it is unavailable on any of the nodes?
> >>>
> >>> I see HelixAdmin.dropResource and HelixAdmin.resetResource. What is
> >>> the right API to use to remove a resource entirely from the cluster?
> >>>
> >>> Thanks,
> >>>
> >>> Sandeep
>
Re: Helix 0.6.2: Resetting or removing a resource
Posted by Sandeep Nayak <os...@gmail.com>.
Btw other than the remove, I have the resources working and
distributing the way I want. So just wanted to say good work, it works
nicely :-)
Thanks,
Sandeep
On Thu, Feb 20, 2014 at 10:06 PM, Sandeep Nayak <os...@gmail.com> wrote:
> Does it need time to remove the resource? I tried dropResource and the
> very next call I invoke addResource to see if I can add it again and
> the add call fails with the exception
>
> org.apache.helix.HelixException: Skip the operation. Resource ideal
> state directory already exists:/cluster/IDEALSTATES/resource-name
>
> On Thu, Feb 20, 2014 at 10:03 PM, Kanak Biscuitwala <ka...@hotmail.com> wrote:
>> Hi Sandeep,
>>
>> dropResource is correct. resetResource takes resources in error state and resets them to the initial state.
>>
>> Kanak
>> ----------------------------------------
>>> Date: Thu, 20 Feb 2014 21:58:24 -0800
>>> Subject: Helix 0.6.2: Resetting or removing a resource
>>> From: osgigeek@gmail.com
>>> To: user@helix.apache.org
>>>
>>> Hi guys,
>>>
>>> Wanted to check what is the correct API for removing a resource so
>>> that it is unavailable on any of the nodes?
>>>
>>> I see HelixAdmin.dropResource and HelixAdmin.resetResource. What is
>>> the right API to use to remove a resource entirely from the cluster?
>>>
>>> Thanks,
>>>
>>> Sandeep
Re: Helix 0.6.2: Resetting or removing a resource
Posted by Sandeep Nayak <os...@gmail.com>.
Does it need time to remove the resource? I tried dropResource and the
very next call I invoke addResource to see if I can add it again and
the add call fails with the exception
org.apache.helix.HelixException: Skip the operation. Resource ideal
state directory already exists:/cluster/IDEALSTATES/resource-name
On Thu, Feb 20, 2014 at 10:03 PM, Kanak Biscuitwala <ka...@hotmail.com> wrote:
> Hi Sandeep,
>
> dropResource is correct. resetResource takes resources in error state and resets them to the initial state.
>
> Kanak
> ----------------------------------------
>> Date: Thu, 20 Feb 2014 21:58:24 -0800
>> Subject: Helix 0.6.2: Resetting or removing a resource
>> From: osgigeek@gmail.com
>> To: user@helix.apache.org
>>
>> Hi guys,
>>
>> Wanted to check what is the correct API for removing a resource so
>> that it is unavailable on any of the nodes?
>>
>> I see HelixAdmin.dropResource and HelixAdmin.resetResource. What is
>> the right API to use to remove a resource entirely from the cluster?
>>
>> Thanks,
>>
>> Sandeep
RE: Helix 0.6.2: Resetting or removing a resource
Posted by Kanak Biscuitwala <ka...@hotmail.com>.
Hi Sandeep,
dropResource is correct. resetResource takes resources in error state and resets them to the initial state.
Kanak
----------------------------------------
> Date: Thu, 20 Feb 2014 21:58:24 -0800
> Subject: Helix 0.6.2: Resetting or removing a resource
> From: osgigeek@gmail.com
> To: user@helix.apache.org
>
> Hi guys,
>
> Wanted to check what is the correct API for removing a resource so
> that it is unavailable on any of the nodes?
>
> I see HelixAdmin.dropResource and HelixAdmin.resetResource. What is
> the right API to use to remove a resource entirely from the cluster?
>
> Thanks,
>
> Sandeep
Re: Helix 0.6.2: Resetting or removing a resource
Posted by kishore g <g....@gmail.com>.
DropResource. This will send transitions to the nodes to clean up the
resource. In you case you are in LEADER state, it will first send
LEADER->STANDBY and the STANDBY->OFFLINE and then OFFLINE->DROPPED. In the
OFFLINE-DROPPED, you can safely assume that this node no longer needs to
hold this resource and you can do the necessary clean up.
thanks,
Kishore G
On Thu, Feb 20, 2014 at 9:58 PM, Sandeep Nayak <os...@gmail.com> wrote:
> Hi guys,
>
> Wanted to check what is the correct API for removing a resource so
> that it is unavailable on any of the nodes?
>
> I see HelixAdmin.dropResource and HelixAdmin.resetResource. What is
> the right API to use to remove a resource entirely from the cluster?
>
> Thanks,
>
> Sandeep
>