You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by Saminda Wijeratne <sa...@gmail.com> on 2014/04/21 15:06:43 UTC
Implementing the cancel/terminate
Hi All,
After looking at the current design and doing some trial and error I
thought of implementing the cancellation as follows.
- Cancellation of an experiment requested by a gateway requires
cancellation request to go through several layers. (Orchestrator > GFac >
GFac Provider)
- Each layer is responsible for handling cancellation relevant for that
layer (Orchestration cancels experiment, GFac cancels Task, GFac Provider
cancels Job)
- What I thought is, each layer will listen to cancellation request made
to the layer above and perform its cancellation actions accordingly. (GFac
will see the experiment is having the status "canceling" for an experiment
id and it will perform cancellation of the tasks relevant for that
experiment)
- Effectively the Orchestrator will be
- updating the status of the experiment in registry with the status
"canceling"
- publish a message which will be caught by GFac instance which
handles its Tasks.
- GFac will perform the same and the correct GFac Provider
instance will catch the message and perform the actual job cancellation.
- Once the job cancellation is done the statuses at each layer will be
updated (to "canceled") in similar fashion.
- We allow the API call of cancellation to be asynchronous
- I'm hoping to use the MonitorPublisher implemented by Lahiru to
publish the messages.
wdyt?
Thanks,
Saminda
Re: Implementing the cancel/terminate
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Yes You are correct !
Regards
Lahiru
On Wed, Apr 30, 2014 at 10:49 PM, Danushka Menikkumbura <
danushka.menikkumbura@gmail.com> wrote:
> Hi Lahiru,
>
> That makes perfect sense.
>
> Basically I see 3 main phases of job
> execution.Pre-process(In-Handlers),Process(Job itself) and
> Post-process(Out-Handlers).The new operation that you mentioned could be
> used to revoke changes done by the In/Out-Handlers.Probably the new type of
> handler that you mentioned would be able to revoke whatever done by the
> jobs?
>
> /Danushka
>
>
> On Wednesday, April 30, 2014, Lahiru Gunathilake <gl...@gmail.com>
> wrote:
> > Hi Danushka,
> > I am thinking of having another operation for a handler which will be
> invoked during a cancel process. If each handler wants to reset what it did
> handlers can implement that logic to the same handler class, if there
> nothing much can be done handler can leave the cancel invocation blank.
> Unless we do this its going to be hard to reset the operations did in each
> handler.
> > We have handlers to run in daemon mode too, so for particular
> implementation developers can use them to perform more higher level clean
> up task which could not make sense to implement in each handler. So GFac
> framework can provide handler level clean up but developers will not be
> completely bound to use that for their clean up implementation.
> > Regards
> > Lahiru
> >
> >
> > On Tue, Apr 29, 2014 at 4:58 PM, Marlon Pierce <ma...@iu.edu> wrote:
> >
> > I assumed there would be a lot of overlap between cancel-handlers and
> > out-handlers but not complete overlap. Cleanup operations need to be
> > done in either case, but "move the successfully created output files to
> > a safe storage location" would not be done.
> >
> > Marlon
> >
> > On 4/29/14 4:49 PM, Danushka Menikkumbura wrote:
> >> To me this sounds a little more complicated.If we take the Whirr-based
> >> cloud bursting impl for instance,we have an in handler to launch a
> cluster
> >> and an out handler to shut it down.When a job is cancelled in a case
> like
> >> this, the corresponding cluster should be shut down and for that the out
> >> handler needs to be notified.
> >>
> >> /Danushka
> >>
> >> On Tuesday, April 29, 2014, Lahiru Gunathilake <gl...@gmail.com>
> wrote:
> >>> Hi Marlon,
> >>>
> >>> On Mon, Apr 28, 2014 at 3:59 PM, Marlon Pierce <ma...@iu.edu>
> wrote:
> >>>> Are any handlers called when canceling? Should they be? Seems like
> you
> >>>> would need to typically do some cleanup.
> >>> +1 thats the plan to have pre/post handlers for terminate operation
> like
> >> we have to submit.
> >>> Lahiru
> >>>> Marlon
> >>>>
> >>>> On 4/23/14 2:35 PM, Saminda Wijeratne wrote:
> >>>>> Got it Suresh. Thanks for putting it clearly :)
> >>>>>
> >>>>>
> >>>>> On Tue, Apr 22, 2014 at 5:48 PM, Suresh Marru <sm...@apache.org>
> >> wrote:
> >>>>>> On Tue, Apr 22, 2014 at 6:26 PM, Saminda Wijeratne <
> samindaw@gmail.com
> >>> wrote:
> >>>>>>> You mean like a monitoring client at client end triggering a
> >> listener?
> >>>>>> Thats ideal, but different story altogether. I mean triggering a
> >> listener
> >>>>>> for all major status changes including canceling.
> >>>>>>
> >>>>>> In this context I mean once the client gets Canceling at experiment
> >> level,
> >>>>>> once all lower level statues gets updated, we should ensure
> subsequent
> >>>>>> client query to experiment returned Cancelled. I see your plan
> already
> >>>>>> accounts for it, was just restating its simpler for gateways to wait
> >> on
> >>>>>> experiment status instead of expecting to navigate through Task->Job
> >> to
> >>>>>> verify the success.
> >>>>>>
> >>>>>> Suresh
> >>>>>>
> >>>>>>
> >>>>>>> On Tue, Apr 22, 2014 at 5:22 PM, Suresh Marru <sm...@apache.org>
> >> wrote:
> >>>>>>>> Hi Saminda,
> >>>>>>>>
> >>>>>>>> This like a good plan, I think we should make sure the API server
> >>>>>>>> properly reports these statuses back to the clients.
> >>>>>>>>
> >>>>>>>> Suresh
> >>>>>>>>
> >>>>>>>> On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <
> samindaw@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi All,
> >>>>>>>>
> >>>>>>>> After looking at the current design and doing some trial and
> error I
> >>>>>>>> thought of implementing the cancellation as follows.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> - Cancellation of an experiment requested by a gateway requires
> >>>>>>>> cancellation request to go through several layers.
> (Orchestrator
> >>> GFac >
> >>>>>>>> GFac Provider)
> >>>>>>>> - Each layer is responsible for handling cancellation
>
--
System Analyst Programmer
PTI Lab
Indiana University
Re: Implementing the cancel/terminate
Posted by Danushka Menikkumbura <da...@gmail.com>.
Hi Lahiru,
That makes perfect sense.
Basically I see 3 main phases of job
execution.Pre-process(In-Handlers),Process(Job itself) and
Post-process(Out-Handlers).The new operation that you mentioned could be
used to revoke changes done by the In/Out-Handlers.Probably the new type of
handler that you mentioned would be able to revoke whatever done by the
jobs?
/Danushka
On Wednesday, April 30, 2014, Lahiru Gunathilake <gl...@gmail.com> wrote:
> Hi Danushka,
> I am thinking of having another operation for a handler which will be
invoked during a cancel process. If each handler wants to reset what it did
handlers can implement that logic to the same handler class, if there
nothing much can be done handler can leave the cancel invocation blank.
Unless we do this its going to be hard to reset the operations did in each
handler.
> We have handlers to run in daemon mode too, so for particular
implementation developers can use them to perform more higher level clean
up task which could not make sense to implement in each handler. So GFac
framework can provide handler level clean up but developers will not be
completely bound to use that for their clean up implementation.
> Regards
> Lahiru
>
>
> On Tue, Apr 29, 2014 at 4:58 PM, Marlon Pierce <ma...@iu.edu> wrote:
>
> I assumed there would be a lot of overlap between cancel-handlers and
> out-handlers but not complete overlap. Cleanup operations need to be
> done in either case, but "move the successfully created output files to
> a safe storage location" would not be done.
>
> Marlon
>
> On 4/29/14 4:49 PM, Danushka Menikkumbura wrote:
>> To me this sounds a little more complicated.If we take the Whirr-based
>> cloud bursting impl for instance,we have an in handler to launch a
cluster
>> and an out handler to shut it down.When a job is cancelled in a case like
>> this, the corresponding cluster should be shut down and for that the out
>> handler needs to be notified.
>>
>> /Danushka
>>
>> On Tuesday, April 29, 2014, Lahiru Gunathilake <gl...@gmail.com> wrote:
>>> Hi Marlon,
>>>
>>> On Mon, Apr 28, 2014 at 3:59 PM, Marlon Pierce <ma...@iu.edu> wrote:
>>>> Are any handlers called when canceling? Should they be? Seems like you
>>>> would need to typically do some cleanup.
>>> +1 thats the plan to have pre/post handlers for terminate operation like
>> we have to submit.
>>> Lahiru
>>>> Marlon
>>>>
>>>> On 4/23/14 2:35 PM, Saminda Wijeratne wrote:
>>>>> Got it Suresh. Thanks for putting it clearly :)
>>>>>
>>>>>
>>>>> On Tue, Apr 22, 2014 at 5:48 PM, Suresh Marru <sm...@apache.org>
>> wrote:
>>>>>> On Tue, Apr 22, 2014 at 6:26 PM, Saminda Wijeratne <
samindaw@gmail.com
>>> wrote:
>>>>>>> You mean like a monitoring client at client end triggering a
>> listener?
>>>>>> Thats ideal, but different story altogether. I mean triggering a
>> listener
>>>>>> for all major status changes including canceling.
>>>>>>
>>>>>> In this context I mean once the client gets Canceling at experiment
>> level,
>>>>>> once all lower level statues gets updated, we should ensure
subsequent
>>>>>> client query to experiment returned Cancelled. I see your plan
already
>>>>>> accounts for it, was just restating its simpler for gateways to wait
>> on
>>>>>> experiment status instead of expecting to navigate through Task->Job
>> to
>>>>>> verify the success.
>>>>>>
>>>>>> Suresh
>>>>>>
>>>>>>
>>>>>>> On Tue, Apr 22, 2014 at 5:22 PM, Suresh Marru <sm...@apache.org>
>> wrote:
>>>>>>>> Hi Saminda,
>>>>>>>>
>>>>>>>> This like a good plan, I think we should make sure the API server
>>>>>>>> properly reports these statuses back to the clients.
>>>>>>>>
>>>>>>>> Suresh
>>>>>>>>
>>>>>>>> On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <sa...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> After looking at the current design and doing some trial and error
I
>>>>>>>> thought of implementing the cancellation as follows.
>>>>>>>>
>>>>>>>>
>>>>>>>> - Cancellation of an experiment requested by a gateway requires
>>>>>>>> cancellation request to go through several layers. (Orchestrator
>>> GFac >
>>>>>>>> GFac Provider)
>>>>>>>> - Each layer is responsible for handling cancellation
Re: Implementing the cancel/terminate
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi Danushka,
I am thinking of having another operation for a handler which will be
invoked during a cancel process. If each handler wants to reset what it did
handlers can implement that logic to the same handler class, if there
nothing much can be done handler can leave the cancel invocation blank.
Unless we do this its going to be hard to reset the operations did in each
handler.
We have handlers to run in daemon mode too, so for particular
implementation developers can use them to perform more higher level clean
up task which could not make sense to implement in each handler. So GFac
framework can provide handler level clean up but developers will not be
completely bound to use that for their clean up implementation.
Regards
Lahiru
On Tue, Apr 29, 2014 at 4:58 PM, Marlon Pierce <ma...@iu.edu> wrote:
> I assumed there would be a lot of overlap between cancel-handlers and
> out-handlers but not complete overlap. Cleanup operations need to be
> done in either case, but "move the successfully created output files to
> a safe storage location" would not be done.
>
> Marlon
>
> On 4/29/14 4:49 PM, Danushka Menikkumbura wrote:
> > To me this sounds a little more complicated.If we take the Whirr-based
> > cloud bursting impl for instance,we have an in handler to launch a
> cluster
> > and an out handler to shut it down.When a job is cancelled in a case like
> > this, the corresponding cluster should be shut down and for that the out
> > handler needs to be notified.
> >
> > /Danushka
> >
> > On Tuesday, April 29, 2014, Lahiru Gunathilake <gl...@gmail.com>
> wrote:
> >> Hi Marlon,
> >>
> >> On Mon, Apr 28, 2014 at 3:59 PM, Marlon Pierce <ma...@iu.edu> wrote:
> >>> Are any handlers called when canceling? Should they be? Seems like you
> >>> would need to typically do some cleanup.
> >> +1 thats the plan to have pre/post handlers for terminate operation like
> > we have to submit.
> >> Lahiru
> >>> Marlon
> >>>
> >>> On 4/23/14 2:35 PM, Saminda Wijeratne wrote:
> >>>> Got it Suresh. Thanks for putting it clearly :)
> >>>>
> >>>>
> >>>> On Tue, Apr 22, 2014 at 5:48 PM, Suresh Marru <sm...@apache.org>
> > wrote:
> >>>>> On Tue, Apr 22, 2014 at 6:26 PM, Saminda Wijeratne <
> samindaw@gmail.com
> >> wrote:
> >>>>>> You mean like a monitoring client at client end triggering a
> > listener?
> >>>>> Thats ideal, but different story altogether. I mean triggering a
> > listener
> >>>>> for all major status changes including canceling.
> >>>>>
> >>>>> In this context I mean once the client gets Canceling at experiment
> > level,
> >>>>> once all lower level statues gets updated, we should ensure
> subsequent
> >>>>> client query to experiment returned Cancelled. I see your plan
> already
> >>>>> accounts for it, was just restating its simpler for gateways to wait
> > on
> >>>>> experiment status instead of expecting to navigate through Task->Job
> > to
> >>>>> verify the success.
> >>>>>
> >>>>> Suresh
> >>>>>
> >>>>>
> >>>>>> On Tue, Apr 22, 2014 at 5:22 PM, Suresh Marru <sm...@apache.org>
> > wrote:
> >>>>>>> Hi Saminda,
> >>>>>>>
> >>>>>>> This like a good plan, I think we should make sure the API server
> >>>>>>> properly reports these statuses back to the clients.
> >>>>>>>
> >>>>>>> Suresh
> >>>>>>>
> >>>>>>> On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <samindaw@gmail.com
> >
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> Hi All,
> >>>>>>>
> >>>>>>> After looking at the current design and doing some trial and error
> I
> >>>>>>> thought of implementing the cancellation as follows.
> >>>>>>>
> >>>>>>>
> >>>>>>> - Cancellation of an experiment requested by a gateway requires
> >>>>>>> cancellation request to go through several layers. (Orchestrator
> >> GFac >
> >>>>>>> GFac Provider)
> >>>>>>> - Each layer is responsible for handling cancellation relevant
> > for
> >>>>>>> that layer (Orchestration cancels experiment, GFac cancels Task,
> > GFac
> >>>>>>> Provider cancels Job)
> >>>>>>> - What I thought is, each layer will listen to cancellation
> > request
> >>>>>>> made to the layer above and perform its cancellation actions
> > accordingly.
> >>>>>>> (GFac will see the experiment is having the status "canceling"
> > for an
> >>>>>>> experiment id and it will perform cancellation of the tasks
> > relevant for
> >>>>>>> that experiment)
> >>>>>>> - Effectively the Orchestrator will be
> >>>>>>> - updating the status of the experiment in registry with the
> >>>>>>> status "canceling"
> >>>>>>> - publish a message which will be caught by GFac instance
> >>>>>>> which handles its Tasks.
> >>>>>>> - GFac will perform the same and the correct GFac Provider
> >>>>>>> instance will catch the message and perform the actual job
> > cancellation.
> >>>>>>> - Once the job cancellation is done the statuses at each layer
> > will
> >>>>>>> be updated (to "canceled") in similar fashion.
> >>>>>>> - We allow the API call of cancellation to be asynchronous
> >>>>>>> - I'm hoping to use the MonitorPublisher implemented by Lahiru
> to
> >>>>>>> publish the messages.
> >>>>>>>
> >>>>>>> wdyt?
> >>>>>>>
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>> Saminda
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>
> >>
> >> --
> >> System Analyst Programmer
> >> PTI Lab
> >> Indiana University
>
>
--
System Analyst Programmer
PTI Lab
Indiana University
Re: Implementing the cancel/terminate
Posted by Marlon Pierce <ma...@iu.edu>.
I assumed there would be a lot of overlap between cancel-handlers and
out-handlers but not complete overlap. Cleanup operations need to be
done in either case, but "move the successfully created output files to
a safe storage location" would not be done.
Marlon
On 4/29/14 4:49 PM, Danushka Menikkumbura wrote:
> To me this sounds a little more complicated.If we take the Whirr-based
> cloud bursting impl for instance,we have an in handler to launch a cluster
> and an out handler to shut it down.When a job is cancelled in a case like
> this, the corresponding cluster should be shut down and for that the out
> handler needs to be notified.
>
> /Danushka
>
> On Tuesday, April 29, 2014, Lahiru Gunathilake <gl...@gmail.com> wrote:
>> Hi Marlon,
>>
>> On Mon, Apr 28, 2014 at 3:59 PM, Marlon Pierce <ma...@iu.edu> wrote:
>>> Are any handlers called when canceling? Should they be? Seems like you
>>> would need to typically do some cleanup.
>> +1 thats the plan to have pre/post handlers for terminate operation like
> we have to submit.
>> Lahiru
>>> Marlon
>>>
>>> On 4/23/14 2:35 PM, Saminda Wijeratne wrote:
>>>> Got it Suresh. Thanks for putting it clearly :)
>>>>
>>>>
>>>> On Tue, Apr 22, 2014 at 5:48 PM, Suresh Marru <sm...@apache.org>
> wrote:
>>>>> On Tue, Apr 22, 2014 at 6:26 PM, Saminda Wijeratne <samindaw@gmail.com
>> wrote:
>>>>>> You mean like a monitoring client at client end triggering a
> listener?
>>>>> Thats ideal, but different story altogether. I mean triggering a
> listener
>>>>> for all major status changes including canceling.
>>>>>
>>>>> In this context I mean once the client gets Canceling at experiment
> level,
>>>>> once all lower level statues gets updated, we should ensure subsequent
>>>>> client query to experiment returned Cancelled. I see your plan already
>>>>> accounts for it, was just restating its simpler for gateways to wait
> on
>>>>> experiment status instead of expecting to navigate through Task->Job
> to
>>>>> verify the success.
>>>>>
>>>>> Suresh
>>>>>
>>>>>
>>>>>> On Tue, Apr 22, 2014 at 5:22 PM, Suresh Marru <sm...@apache.org>
> wrote:
>>>>>>> Hi Saminda,
>>>>>>>
>>>>>>> This like a good plan, I think we should make sure the API server
>>>>>>> properly reports these statuses back to the clients.
>>>>>>>
>>>>>>> Suresh
>>>>>>>
>>>>>>> On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <sa...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> After looking at the current design and doing some trial and error I
>>>>>>> thought of implementing the cancellation as follows.
>>>>>>>
>>>>>>>
>>>>>>> - Cancellation of an experiment requested by a gateway requires
>>>>>>> cancellation request to go through several layers. (Orchestrator
>> GFac >
>>>>>>> GFac Provider)
>>>>>>> - Each layer is responsible for handling cancellation relevant
> for
>>>>>>> that layer (Orchestration cancels experiment, GFac cancels Task,
> GFac
>>>>>>> Provider cancels Job)
>>>>>>> - What I thought is, each layer will listen to cancellation
> request
>>>>>>> made to the layer above and perform its cancellation actions
> accordingly.
>>>>>>> (GFac will see the experiment is having the status "canceling"
> for an
>>>>>>> experiment id and it will perform cancellation of the tasks
> relevant for
>>>>>>> that experiment)
>>>>>>> - Effectively the Orchestrator will be
>>>>>>> - updating the status of the experiment in registry with the
>>>>>>> status "canceling"
>>>>>>> - publish a message which will be caught by GFac instance
>>>>>>> which handles its Tasks.
>>>>>>> - GFac will perform the same and the correct GFac Provider
>>>>>>> instance will catch the message and perform the actual job
> cancellation.
>>>>>>> - Once the job cancellation is done the statuses at each layer
> will
>>>>>>> be updated (to "canceled") in similar fashion.
>>>>>>> - We allow the API call of cancellation to be asynchronous
>>>>>>> - I'm hoping to use the MonitorPublisher implemented by Lahiru to
>>>>>>> publish the messages.
>>>>>>>
>>>>>>> wdyt?
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Saminda
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>
>>
>> --
>> System Analyst Programmer
>> PTI Lab
>> Indiana University
Re: Implementing the cancel/terminate
Posted by Danushka Menikkumbura <da...@gmail.com>.
To me this sounds a little more complicated.If we take the Whirr-based
cloud bursting impl for instance,we have an in handler to launch a cluster
and an out handler to shut it down.When a job is cancelled in a case like
this, the corresponding cluster should be shut down and for that the out
handler needs to be notified.
/Danushka
On Tuesday, April 29, 2014, Lahiru Gunathilake <gl...@gmail.com> wrote:
> Hi Marlon,
>
> On Mon, Apr 28, 2014 at 3:59 PM, Marlon Pierce <ma...@iu.edu> wrote:
>>
>> Are any handlers called when canceling? Should they be? Seems like you
>> would need to typically do some cleanup.
>
> +1 thats the plan to have pre/post handlers for terminate operation like
we have to submit.
> Lahiru
>>
>> Marlon
>>
>> On 4/23/14 2:35 PM, Saminda Wijeratne wrote:
>> > Got it Suresh. Thanks for putting it clearly :)
>> >
>> >
>> > On Tue, Apr 22, 2014 at 5:48 PM, Suresh Marru <sm...@apache.org>
wrote:
>> >
>> >> On Tue, Apr 22, 2014 at 6:26 PM, Saminda Wijeratne <samindaw@gmail.com
>wrote:
>> >>
>> >>> You mean like a monitoring client at client end triggering a
listener?
>> >>>
>> >> Thats ideal, but different story altogether. I mean triggering a
listener
>> >> for all major status changes including canceling.
>> >>
>> >> In this context I mean once the client gets Canceling at experiment
level,
>> >> once all lower level statues gets updated, we should ensure subsequent
>> >> client query to experiment returned Cancelled. I see your plan already
>> >> accounts for it, was just restating its simpler for gateways to wait
on
>> >> experiment status instead of expecting to navigate through Task->Job
to
>> >> verify the success.
>> >>
>> >> Suresh
>> >>
>> >>
>> >>>
>> >>> On Tue, Apr 22, 2014 at 5:22 PM, Suresh Marru <sm...@apache.org>
wrote:
>> >>>
>> >>>> Hi Saminda,
>> >>>>
>> >>>> This like a good plan, I think we should make sure the API server
>> >>>> properly reports these statuses back to the clients.
>> >>>>
>> >>>> Suresh
>> >>>>
>> >>>> On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <sa...@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>> Hi All,
>> >>>>
>> >>>> After looking at the current design and doing some trial and error I
>> >>>> thought of implementing the cancellation as follows.
>> >>>>
>> >>>>
>> >>>> - Cancellation of an experiment requested by a gateway requires
>> >>>> cancellation request to go through several layers. (Orchestrator
> GFac >
>> >>>> GFac Provider)
>> >>>> - Each layer is responsible for handling cancellation relevant
for
>> >>>> that layer (Orchestration cancels experiment, GFac cancels Task,
GFac
>> >>>> Provider cancels Job)
>> >>>> - What I thought is, each layer will listen to cancellation
request
>> >>>> made to the layer above and perform its cancellation actions
accordingly.
>> >>>> (GFac will see the experiment is having the status "canceling"
for an
>> >>>> experiment id and it will perform cancellation of the tasks
relevant for
>> >>>> that experiment)
>> >>>> - Effectively the Orchestrator will be
>> >>>> - updating the status of the experiment in registry with the
>> >>>> status "canceling"
>> >>>> - publish a message which will be caught by GFac instance
>> >>>> which handles its Tasks.
>> >>>> - GFac will perform the same and the correct GFac Provider
>> >>>> instance will catch the message and perform the actual job
cancellation.
>> >>>> - Once the job cancellation is done the statuses at each layer
will
>> >>>> be updated (to "canceled") in similar fashion.
>> >>>> - We allow the API call of cancellation to be asynchronous
>> >>>> - I'm hoping to use the MonitorPublisher implemented by Lahiru to
>> >>>> publish the messages.
>> >>>>
>> >>>> wdyt?
>> >>>>
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> Saminda
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>>
>
>
>
> --
> System Analyst Programmer
> PTI Lab
> Indiana University
Re: Implementing the cancel/terminate
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi Marlon,
On Mon, Apr 28, 2014 at 3:59 PM, Marlon Pierce <ma...@iu.edu> wrote:
> Are any handlers called when canceling? Should they be? Seems like you
> would need to typically do some cleanup.
>
+1 thats the plan to have pre/post handlers for terminate operation like we
have to submit.
Lahiru
>
> Marlon
>
> On 4/23/14 2:35 PM, Saminda Wijeratne wrote:
> > Got it Suresh. Thanks for putting it clearly :)
> >
> >
> > On Tue, Apr 22, 2014 at 5:48 PM, Suresh Marru <sm...@apache.org> wrote:
> >
> >> On Tue, Apr 22, 2014 at 6:26 PM, Saminda Wijeratne <samindaw@gmail.com
> >wrote:
> >>
> >>> You mean like a monitoring client at client end triggering a listener?
> >>>
> >> Thats ideal, but different story altogether. I mean triggering a
> listener
> >> for all major status changes including canceling.
> >>
> >> In this context I mean once the client gets Canceling at experiment
> level,
> >> once all lower level statues gets updated, we should ensure subsequent
> >> client query to experiment returned Cancelled. I see your plan already
> >> accounts for it, was just restating its simpler for gateways to wait on
> >> experiment status instead of expecting to navigate through Task->Job to
> >> verify the success.
> >>
> >> Suresh
> >>
> >>
> >>>
> >>> On Tue, Apr 22, 2014 at 5:22 PM, Suresh Marru <sm...@apache.org>
> wrote:
> >>>
> >>>> Hi Saminda,
> >>>>
> >>>> This like a good plan, I think we should make sure the API server
> >>>> properly reports these statuses back to the clients.
> >>>>
> >>>> Suresh
> >>>>
> >>>> On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <sa...@gmail.com>
> >>>> wrote:
> >>>>
> >>>> Hi All,
> >>>>
> >>>> After looking at the current design and doing some trial and error I
> >>>> thought of implementing the cancellation as follows.
> >>>>
> >>>>
> >>>> - Cancellation of an experiment requested by a gateway requires
> >>>> cancellation request to go through several layers. (Orchestrator >
> GFac >
> >>>> GFac Provider)
> >>>> - Each layer is responsible for handling cancellation relevant for
> >>>> that layer (Orchestration cancels experiment, GFac cancels Task,
> GFac
> >>>> Provider cancels Job)
> >>>> - What I thought is, each layer will listen to cancellation request
> >>>> made to the layer above and perform its cancellation actions
> accordingly.
> >>>> (GFac will see the experiment is having the status "canceling" for
> an
> >>>> experiment id and it will perform cancellation of the tasks
> relevant for
> >>>> that experiment)
> >>>> - Effectively the Orchestrator will be
> >>>> - updating the status of the experiment in registry with the
> >>>> status "canceling"
> >>>> - publish a message which will be caught by GFac instance
> >>>> which handles its Tasks.
> >>>> - GFac will perform the same and the correct GFac Provider
> >>>> instance will catch the message and perform the actual job
> cancellation.
> >>>> - Once the job cancellation is done the statuses at each layer will
> >>>> be updated (to "canceled") in similar fashion.
> >>>> - We allow the API call of cancellation to be asynchronous
> >>>> - I'm hoping to use the MonitorPublisher implemented by Lahiru to
> >>>> publish the messages.
> >>>>
> >>>> wdyt?
> >>>>
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Saminda
> >>>>
> >>>>
> >>>>
> >>>>
>
>
--
System Analyst Programmer
PTI Lab
Indiana University
Re: Implementing the cancel/terminate
Posted by Marlon Pierce <ma...@iu.edu>.
Are any handlers called when canceling? Should they be? Seems like you
would need to typically do some cleanup.
Marlon
On 4/23/14 2:35 PM, Saminda Wijeratne wrote:
> Got it Suresh. Thanks for putting it clearly :)
>
>
> On Tue, Apr 22, 2014 at 5:48 PM, Suresh Marru <sm...@apache.org> wrote:
>
>> On Tue, Apr 22, 2014 at 6:26 PM, Saminda Wijeratne <sa...@gmail.com>wrote:
>>
>>> You mean like a monitoring client at client end triggering a listener?
>>>
>> Thats ideal, but different story altogether. I mean triggering a listener
>> for all major status changes including canceling.
>>
>> In this context I mean once the client gets Canceling at experiment level,
>> once all lower level statues gets updated, we should ensure subsequent
>> client query to experiment returned Cancelled. I see your plan already
>> accounts for it, was just restating its simpler for gateways to wait on
>> experiment status instead of expecting to navigate through Task->Job to
>> verify the success.
>>
>> Suresh
>>
>>
>>>
>>> On Tue, Apr 22, 2014 at 5:22 PM, Suresh Marru <sm...@apache.org> wrote:
>>>
>>>> Hi Saminda,
>>>>
>>>> This like a good plan, I think we should make sure the API server
>>>> properly reports these statuses back to the clients.
>>>>
>>>> Suresh
>>>>
>>>> On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <sa...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> After looking at the current design and doing some trial and error I
>>>> thought of implementing the cancellation as follows.
>>>>
>>>>
>>>> - Cancellation of an experiment requested by a gateway requires
>>>> cancellation request to go through several layers. (Orchestrator > GFac >
>>>> GFac Provider)
>>>> - Each layer is responsible for handling cancellation relevant for
>>>> that layer (Orchestration cancels experiment, GFac cancels Task, GFac
>>>> Provider cancels Job)
>>>> - What I thought is, each layer will listen to cancellation request
>>>> made to the layer above and perform its cancellation actions accordingly.
>>>> (GFac will see the experiment is having the status "canceling" for an
>>>> experiment id and it will perform cancellation of the tasks relevant for
>>>> that experiment)
>>>> - Effectively the Orchestrator will be
>>>> - updating the status of the experiment in registry with the
>>>> status "canceling"
>>>> - publish a message which will be caught by GFac instance
>>>> which handles its Tasks.
>>>> - GFac will perform the same and the correct GFac Provider
>>>> instance will catch the message and perform the actual job cancellation.
>>>> - Once the job cancellation is done the statuses at each layer will
>>>> be updated (to "canceled") in similar fashion.
>>>> - We allow the API call of cancellation to be asynchronous
>>>> - I'm hoping to use the MonitorPublisher implemented by Lahiru to
>>>> publish the messages.
>>>>
>>>> wdyt?
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Saminda
>>>>
>>>>
>>>>
>>>>
Re: Implementing the cancel/terminate
Posted by Saminda Wijeratne <sa...@gmail.com>.
Got it Suresh. Thanks for putting it clearly :)
On Tue, Apr 22, 2014 at 5:48 PM, Suresh Marru <sm...@apache.org> wrote:
> On Tue, Apr 22, 2014 at 6:26 PM, Saminda Wijeratne <sa...@gmail.com>wrote:
>
>> You mean like a monitoring client at client end triggering a listener?
>>
>
> Thats ideal, but different story altogether. I mean triggering a listener
> for all major status changes including canceling.
>
> In this context I mean once the client gets Canceling at experiment level,
> once all lower level statues gets updated, we should ensure subsequent
> client query to experiment returned Cancelled. I see your plan already
> accounts for it, was just restating its simpler for gateways to wait on
> experiment status instead of expecting to navigate through Task->Job to
> verify the success.
>
> Suresh
>
>
>>
>>
>> On Tue, Apr 22, 2014 at 5:22 PM, Suresh Marru <sm...@apache.org> wrote:
>>
>>> Hi Saminda,
>>>
>>> This like a good plan, I think we should make sure the API server
>>> properly reports these statuses back to the clients.
>>>
>>> Suresh
>>>
>>> On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <sa...@gmail.com>
>>> wrote:
>>>
>>> Hi All,
>>>
>>> After looking at the current design and doing some trial and error I
>>> thought of implementing the cancellation as follows.
>>>
>>>
>>> - Cancellation of an experiment requested by a gateway requires
>>> cancellation request to go through several layers. (Orchestrator > GFac >
>>> GFac Provider)
>>> - Each layer is responsible for handling cancellation relevant for
>>> that layer (Orchestration cancels experiment, GFac cancels Task, GFac
>>> Provider cancels Job)
>>> - What I thought is, each layer will listen to cancellation request
>>> made to the layer above and perform its cancellation actions accordingly.
>>> (GFac will see the experiment is having the status "canceling" for an
>>> experiment id and it will perform cancellation of the tasks relevant for
>>> that experiment)
>>> - Effectively the Orchestrator will be
>>> - updating the status of the experiment in registry with the
>>> status "canceling"
>>> - publish a message which will be caught by GFac instance
>>> which handles its Tasks.
>>> - GFac will perform the same and the correct GFac Provider
>>> instance will catch the message and perform the actual job cancellation.
>>> - Once the job cancellation is done the statuses at each layer will
>>> be updated (to "canceled") in similar fashion.
>>> - We allow the API call of cancellation to be asynchronous
>>> - I'm hoping to use the MonitorPublisher implemented by Lahiru to
>>> publish the messages.
>>>
>>> wdyt?
>>>
>>>
>>> Thanks,
>>>
>>> Saminda
>>>
>>>
>>>
>>>
>>
>
Re: Implementing the cancel/terminate
Posted by Suresh Marru <sm...@apache.org>.
On Tue, Apr 22, 2014 at 6:26 PM, Saminda Wijeratne <sa...@gmail.com>wrote:
> You mean like a monitoring client at client end triggering a listener?
>
Thats ideal, but different story altogether. I mean triggering a listener
for all major status changes including canceling.
In this context I mean once the client gets Canceling at experiment level,
once all lower level statues gets updated, we should ensure subsequent
client query to experiment returned Cancelled. I see your plan already
accounts for it, was just restating its simpler for gateways to wait on
experiment status instead of expecting to navigate through Task->Job to
verify the success.
Suresh
>
>
> On Tue, Apr 22, 2014 at 5:22 PM, Suresh Marru <sm...@apache.org> wrote:
>
>> Hi Saminda,
>>
>> This like a good plan, I think we should make sure the API server
>> properly reports these statuses back to the clients.
>>
>> Suresh
>>
>> On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <sa...@gmail.com>
>> wrote:
>>
>> Hi All,
>>
>> After looking at the current design and doing some trial and error I
>> thought of implementing the cancellation as follows.
>>
>>
>> - Cancellation of an experiment requested by a gateway requires
>> cancellation request to go through several layers. (Orchestrator > GFac >
>> GFac Provider)
>> - Each layer is responsible for handling cancellation relevant for
>> that layer (Orchestration cancels experiment, GFac cancels Task, GFac
>> Provider cancels Job)
>> - What I thought is, each layer will listen to cancellation request
>> made to the layer above and perform its cancellation actions accordingly.
>> (GFac will see the experiment is having the status "canceling" for an
>> experiment id and it will perform cancellation of the tasks relevant for
>> that experiment)
>> - Effectively the Orchestrator will be
>> - updating the status of the experiment in registry with the
>> status "canceling"
>> - publish a message which will be caught by GFac instance which
>> handles its Tasks.
>> - GFac will perform the same and the correct GFac Provider
>> instance will catch the message and perform the actual job cancellation.
>> - Once the job cancellation is done the statuses at each layer will
>> be updated (to "canceled") in similar fashion.
>> - We allow the API call of cancellation to be asynchronous
>> - I'm hoping to use the MonitorPublisher implemented by Lahiru to
>> publish the messages.
>>
>> wdyt?
>>
>>
>> Thanks,
>>
>> Saminda
>>
>>
>>
>>
>
Re: Implementing the cancel/terminate
Posted by Saminda Wijeratne <sa...@gmail.com>.
You mean like a monitoring client at client end triggering a listener?
On Tue, Apr 22, 2014 at 5:22 PM, Suresh Marru <sm...@apache.org> wrote:
> Hi Saminda,
>
> This like a good plan, I think we should make sure the API server properly
> reports these statuses back to the clients.
>
> Suresh
>
> On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <sa...@gmail.com> wrote:
>
> Hi All,
>
> After looking at the current design and doing some trial and error I
> thought of implementing the cancellation as follows.
>
>
> - Cancellation of an experiment requested by a gateway requires
> cancellation request to go through several layers. (Orchestrator > GFac >
> GFac Provider)
> - Each layer is responsible for handling cancellation relevant for
> that layer (Orchestration cancels experiment, GFac cancels Task, GFac
> Provider cancels Job)
> - What I thought is, each layer will listen to cancellation request
> made to the layer above and perform its cancellation actions accordingly.
> (GFac will see the experiment is having the status "canceling" for an
> experiment id and it will perform cancellation of the tasks relevant for
> that experiment)
> - Effectively the Orchestrator will be
> - updating the status of the experiment in registry with the status
> "canceling"
> - publish a message which will be caught by GFac instance which
> handles its Tasks.
> - GFac will perform the same and the correct GFac Provider
> instance will catch the message and perform the actual job cancellation.
> - Once the job cancellation is done the statuses at each layer will be
> updated (to "canceled") in similar fashion.
> - We allow the API call of cancellation to be asynchronous
> - I'm hoping to use the MonitorPublisher implemented by Lahiru to
> publish the messages.
>
> wdyt?
>
>
> Thanks,
>
> Saminda
>
>
>
>
Re: Implementing the cancel/terminate
Posted by Suresh Marru <sm...@apache.org>.
Hi Saminda,
This like a good plan, I think we should make sure the API server properly reports these statuses back to the clients.
Suresh
On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <sa...@gmail.com> wrote:
> Hi All,
>
> After looking at the current design and doing some trial and error I thought of implementing the cancellation as follows.
>
> Cancellation of an experiment requested by a gateway requires cancellation request to go through several layers. (Orchestrator > GFac > GFac Provider)
> Each layer is responsible for handling cancellation relevant for that layer (Orchestration cancels experiment, GFac cancels Task, GFac Provider cancels Job)
> What I thought is, each layer will listen to cancellation request made to the layer above and perform its cancellation actions accordingly. (GFac will see the experiment is having the status "canceling" for an experiment id and it will perform cancellation of the tasks relevant for that experiment)
> Effectively the Orchestrator will be
> updating the status of the experiment in registry with the status "canceling"
> publish a message which will be caught by GFac instance which handles its Tasks.
> GFac will perform the same and the correct GFac Provider instance will catch the message and perform the actual job cancellation.
> Once the job cancellation is done the statuses at each layer will be updated (to "canceled") in similar fashion.
> We allow the API call of cancellation to be asynchronous
> I'm hoping to use the MonitorPublisher implemented by Lahiru to publish the messages.
> wdyt?
>
>
>
> Thanks,
>
> Saminda
>
>
Re: Implementing the cancel/terminate
Posted by Raminder Singh <ra...@gmail.com>.
Yes, its important to track those job states and make actions based on user request. Job canceling at the early stage may have few time but it need to be handled in Airavata. Can we use registry to communicate such status to input chain or provider?
Thanks
Raminder
On Apr 21, 2014, at 10:51 AM, Saminda Wijeratne <sa...@gmail.com> wrote:
> hmm... The reason why I thought the GFacProvider should handle is that the actual process of job submission involves many steps including authentications, data transfers apart from the actual job submission. Cancellation should be possible inbetween each of those steps. Thus the active provider should monitor cancellation requests and stop whatever its doing. If the job is already submitted I think the Monitoring which you mentioned could take over the cancellation. wdyt?
>
>
> On Mon, Apr 21, 2014 at 7:35 AM, Raminder Singh <ra...@gmail.com> wrote:
> Thanks for investigation the problem and working through solution. This is really required for the production gateways like Ultrascan.
>
> In the current architecture where we have job submission(provider) and monitoring separate, job cancel request need not to go to GFAC provider. Provider submits the jobs and handover the job id to the orchestrator. Orchestrator works with the job monitoring to maintain the job state. Now the cancel need to be handled by Orchestrator and Monitoring. That will change the course of action for API to cancel a job.
>
> One important requirement to take care is cleanup task after the job is canceled like updating the job status table and updating the status.
>
> Thanks
> Raminder
>
>
> On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <sa...@gmail.com> wrote:
>
>> Hi All,
>>
>> After looking at the current design and doing some trial and error I thought of implementing the cancellation as follows.
>>
>> Cancellation of an experiment requested by a gateway requires cancellation request to go through several layers. (Orchestrator > GFac > GFac Provider)
>> Each layer is responsible for handling cancellation relevant for that layer (Orchestration cancels experiment, GFac cancels Task, GFac Provider cancels Job)
>> What I thought is, each layer will listen to cancellation request made to the layer above and perform its cancellation actions accordingly. (GFac will see the experiment is having the status "canceling" for an experiment id and it will perform cancellation of the tasks relevant for that experiment)
>> Effectively the Orchestrator will be
>> updating the status of the experiment in registry with the status "canceling"
>> publish a message which will be caught by GFac instance which handles its Tasks.
>> GFac will perform the same and the correct GFac Provider instance will catch the message and perform the actual job cancellation.
>> Once the job cancellation is done the statuses at each layer will be updated (to "canceled") in similar fashion.
>> We allow the API call of cancellation to be asynchronous
>> I'm hoping to use the MonitorPublisher implemented by Lahiru to publish the messages.
>> wdyt?
>>
>>
>>
>> Thanks,
>>
>> Saminda
>>
>>
>
>
Re: Implementing the cancel/terminate
Posted by Saminda Wijeratne <sa...@gmail.com>.
hmm... The reason why I thought the GFacProvider should handle is that the
actual process of job submission involves many steps including
authentications, data transfers apart from the actual job submission.
Cancellation should be possible inbetween each of those steps. Thus the
active provider should monitor cancellation requests and stop whatever its
doing. If the job is already submitted I think the Monitoring which you
mentioned could take over the cancellation. wdyt?
On Mon, Apr 21, 2014 at 7:35 AM, Raminder Singh <ra...@gmail.com>wrote:
> Thanks for investigation the problem and working through solution. This
> is really required for the production gateways like Ultrascan.
>
> In the current architecture where we have job submission(provider) and
> monitoring separate, job cancel request need not to go to GFAC provider.
> Provider submits the jobs and handover the job id to the orchestrator.
> Orchestrator works with the job monitoring to maintain the job state. Now
> the cancel need to be handled by Orchestrator and Monitoring. That will
> change the course of action for API to cancel a job.
>
> One important requirement to take care is cleanup task after the job is
> canceled like updating the job status table and updating the status.
>
> Thanks
> Raminder
>
>
> On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <sa...@gmail.com> wrote:
>
> Hi All,
>
> After looking at the current design and doing some trial and error I
> thought of implementing the cancellation as follows.
>
>
> - Cancellation of an experiment requested by a gateway requires
> cancellation request to go through several layers. (Orchestrator > GFac >
> GFac Provider)
> - Each layer is responsible for handling cancellation relevant for
> that layer (Orchestration cancels experiment, GFac cancels Task, GFac
> Provider cancels Job)
> - What I thought is, each layer will listen to cancellation request
> made to the layer above and perform its cancellation actions accordingly.
> (GFac will see the experiment is having the status "canceling" for an
> experiment id and it will perform cancellation of the tasks relevant for
> that experiment)
> - Effectively the Orchestrator will be
> - updating the status of the experiment in registry with the status
> "canceling"
> - publish a message which will be caught by GFac instance which
> handles its Tasks.
> - GFac will perform the same and the correct GFac Provider
> instance will catch the message and perform the actual job cancellation.
> - Once the job cancellation is done the statuses at each layer will be
> updated (to "canceled") in similar fashion.
> - We allow the API call of cancellation to be asynchronous
> - I'm hoping to use the MonitorPublisher implemented by Lahiru to
> publish the messages.
>
> wdyt?
>
>
> Thanks,
>
> Saminda
>
>
>
>
Re: Implementing the cancel/terminate
Posted by Raminder Singh <ra...@gmail.com>.
Idea is not to do job canceling from monitoring but have a single state. I had experience dealing with job cancel as a asynchronous call and difficulty of dealing with job status. Resources return job as complete even user canceled the job. I will recommend to modify the monitor thread to stop listening for canceled job and update the job status not relaying on resource status.
Thanks
Raminder
On Apr 21, 2014, at 3:23 PM, Saminda Wijeratne <sa...@gmail.com> wrote:
>
>
>
> On Mon, Apr 21, 2014 at 11:51 AM, Eroma Abeysinghe <er...@gmail.com> wrote:
> Hi,
>
> If you have bottom up we will not be able to cancel unless there is a job available for that experiment right?
> Hmm... dats true... when a Job is not available the Task would be the bottom doing the preprocessing stuff for the job submission or post processing stuff after job completion.
> Also few questions;
> 1. What do we really mean by canceling? is it just a status update?
> Effectively a status update. But it will signal for interested parties what is happenning. eg: gateway will knwo that its in the process of canceling and GFacProvider knows it needs to perform actual cancellation if not already done.
> OR
> 2. Are we going to stop all file transfers, delete any data file/file path existing in Airavata for that experiment/tasks/job?
> I think thats what Raminder meant by the cleanup operations.
> 3. And also are we considering both single submission and workflows or is it just single submissions?
> Personally right now I'm focusing only on single submission. Atleast until I get my bearings on how this new design will play-out with what we want. Workflows will come next.
> If we are going to consider canceling of workflows then we need to extend cancelling to multiple tasks and jobs an experiment would have
> Yep
> 4. Also we need to define what experiments we can cancel - IMO we don't need to bother with COMPLETED, CANCELED, UNKNOWN, FAILED experiments and similar statuses in tasks and jobs
> Yes, I'll add that validation.
>
>
>
> IMHO i also don't think Job monitor should do any cancellations.
>
> Thank You,
> Best Regards,
> Eroma
>
>
>
>
>
> On Mon, Apr 21, 2014 at 2:37 PM, Saminda Wijeratne <sa...@gmail.com> wrote:
> May I finish setting up the framework for catching cancel requests? I'll finish implementing the cancel once we decide upon who will do what when cancelling a job.
>
> I just remembered the canceled notification would be handled by the status update mechanism we introduced last week. But this mechanism works only bottom up, i.e. Job status updates will trigger Task status updates and that will trigger Experiment status updates. Does it make sense to have "canceling" status also to progress likewise instead of top down (which i suggested in the first mail)?
>
>
>
> On Mon, Apr 21, 2014 at 11:11 AM, Lahiru Gunathilake <gl...@gmail.com> wrote:
> Hi Raman,
>
>
> On Mon, Apr 21, 2014 at 10:35 AM, Raminder Singh <ra...@gmail.com> wrote:
> Thanks for investigation the problem and working through solution. This is really required for the production gateways like Ultrascan.
>
> In the current architecture where we have job submission(provider) and monitoring separate, job cancel request need not to go to GFAC provider. Provider submits the jobs and handover the job id to the orchestrator. Orchestrator works with the job monitoring to maintain the job state. Now the cancel need to be handled by Orchestrator and Monitoring. That will change the course of action for API to cancel a job.
>
> I dont' think so, Orchestrator can invoke GFac Provider level job cancellation and it should simply reflect in the monitor when it try to get the status of that job( once its got to know by the monitor it should stop monitoring that job) and without modifying the monitor everything should work. There is no need to touch the monitor.
>
> I think Job cancellation should be a functionality of GFAC Provider and it should be similar to job submission where you can do pre processing and post processing after job cancellation operation.
> One important requirement to take care is cleanup task after the job is canceled like updating the job status table and updating the status.
>
> Thanks
> Raminder
>
>
> On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <sa...@gmail.com> wrote:
>
>> Hi All,
>>
>> After looking at the current design and doing some trial and error I thought of implementing the cancellation as follows.
>>
>> Cancellation of an experiment requested by a gateway requires cancellation request to go through several layers. (Orchestrator > GFac > GFac Provider)
>> Each layer is responsible for handling cancellation relevant for that layer (Orchestration cancels experiment, GFac cancels Task, GFac Provider cancels Job)
>> What I thought is, each layer will listen to cancellation request made to the layer above and perform its cancellation actions accordingly. (GFac will see the experiment is having the status "canceling" for an experiment id and it will perform cancellation of the tasks relevant for that experiment)
>> Effectively the Orchestrator will be
>> updating the status of the experiment in registry with the status "canceling"
>> publish a message which will be caught by GFac instance which handles its Tasks.
>> GFac will perform the same and the correct GFac Provider instance will catch the message and perform the actual job cancellation.
>> Once the job cancellation is done the statuses at each layer will be updated (to "canceled") in similar fashion.
>> We allow the API call of cancellation to be asynchronous
>> I'm hoping to use the MonitorPublisher implemented by Lahiru to publish the messages.
>> wdyt?
>>
>>
>>
>> Thanks,
>>
>> Saminda
>>
>>
>
>
>
>
> --
> System Analyst Programmer
> PTI Lab
> Indiana University
>
>
>
>
> --
> Thank You,
> Best Regards,
> Eroma
>
Re: Implementing the cancel/terminate
Posted by Saminda Wijeratne <sa...@gmail.com>.
On Mon, Apr 21, 2014 at 11:51 AM, Eroma Abeysinghe <
eroma.abeysinghe@gmail.com> wrote:
> Hi,
>
> If you have bottom up we will not be able to cancel unless there is a job
> available for that experiment right?
>
Hmm... dats true... when a Job is not available the Task would be the
bottom doing the preprocessing stuff for the job submission or post
processing stuff after job completion.
> Also few questions;
> 1. What do we really mean by canceling? is it just a status update?
>
Effectively a status update. But it will signal for interested parties what
is happenning. eg: gateway will knwo that its in the process of canceling
and GFacProvider knows it needs to perform actual cancellation if not
already done.
> OR
> 2. Are we going to stop all file transfers, delete any data file/file path
> existing in Airavata for that experiment/tasks/job?
>
I think thats what Raminder meant by the cleanup operations.
> 3. And also are we considering both single submission and workflows or
> is it just single submissions?
>
Personally right now I'm focusing only on single submission. Atleast until
I get my bearings on how this new design will play-out with what we want.
Workflows will come next.
> If we are going to consider canceling of workflows then we need to extend
> cancelling to multiple tasks and jobs an experiment would have
>
Yep
> 4. Also we need to define what experiments we can cancel - IMO we don't
> need to bother with COMPLETED, CANCELED, UNKNOWN, FAILED experiments and
> similar statuses in tasks and jobs
>
Yes, I'll add that validation.
>
> IMHO i also don't think Job monitor should do any cancellations.
>
> Thank You,
> Best Regards,
> Eroma
>
>
>
>
>
> On Mon, Apr 21, 2014 at 2:37 PM, Saminda Wijeratne <sa...@gmail.com>wrote:
>
>> May I finish setting up the framework for catching cancel requests? I'll
>> finish implementing the cancel once we decide upon who will do what when
>> cancelling a job.
>>
>> I just remembered the canceled notification would be handled by the
>> status update mechanism we introduced last week. But this mechanism works
>> only bottom up, i.e. Job status updates will trigger Task status updates
>> and that will trigger Experiment status updates. Does it make sense to have
>> "canceling" status also to progress likewise instead of top down (which i
>> suggested in the first mail)?
>>
>>
>>
>> On Mon, Apr 21, 2014 at 11:11 AM, Lahiru Gunathilake <gl...@gmail.com>wrote:
>>
>>> Hi Raman,
>>>
>>>
>>> On Mon, Apr 21, 2014 at 10:35 AM, Raminder Singh <
>>> raminderjsingh@gmail.com> wrote:
>>>
>>>> Thanks for investigation the problem and working through
>>>> solution. This is really required for the production gateways like
>>>> Ultrascan.
>>>>
>>>> In the current architecture where we have job submission(provider) and
>>>> monitoring separate, job cancel request need not to go to GFAC provider.
>>>> Provider submits the jobs and handover the job id to the orchestrator.
>>>> Orchestrator works with the job monitoring to maintain the job state. Now
>>>> the cancel need to be handled by Orchestrator and Monitoring. That will
>>>> change the course of action for API to cancel a job.
>>>>
>>>> I dont' think so, Orchestrator can invoke GFac Provider level job
>>> cancellation and it should simply reflect in the monitor when it try to get
>>> the status of that job( once its got to know by the monitor it should stop
>>> monitoring that job) and without modifying the monitor everything should
>>> work. There is no need to touch the monitor.
>>>
>>> I think Job cancellation should be a functionality of GFAC Provider and
>>> it should be similar to job submission where you can do pre processing and
>>> post processing after job cancellation operation.
>>>
>>>> One important requirement to take care is cleanup task after the job is
>>>> canceled like updating the job status table and updating the status.
>>>>
>>>> Thanks
>>>> Raminder
>>>>
>>>>
>>>> On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <sa...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> After looking at the current design and doing some trial and error I
>>>> thought of implementing the cancellation as follows.
>>>>
>>>>
>>>> - Cancellation of an experiment requested by a gateway requires
>>>> cancellation request to go through several layers. (Orchestrator > GFac >
>>>> GFac Provider)
>>>> - Each layer is responsible for handling cancellation relevant for
>>>> that layer (Orchestration cancels experiment, GFac cancels Task, GFac
>>>> Provider cancels Job)
>>>> - What I thought is, each layer will listen to cancellation request
>>>> made to the layer above and perform its cancellation actions accordingly.
>>>> (GFac will see the experiment is having the status "canceling" for an
>>>> experiment id and it will perform cancellation of the tasks relevant for
>>>> that experiment)
>>>> - Effectively the Orchestrator will be
>>>> - updating the status of the experiment in registry with the
>>>> status "canceling"
>>>> - publish a message which will be caught by GFac instance
>>>> which handles its Tasks.
>>>> - GFac will perform the same and the correct GFac Provider
>>>> instance will catch the message and perform the actual job cancellation.
>>>> - Once the job cancellation is done the statuses at each layer will
>>>> be updated (to "canceled") in similar fashion.
>>>> - We allow the API call of cancellation to be asynchronous
>>>> - I'm hoping to use the MonitorPublisher implemented by Lahiru to
>>>> publish the messages.
>>>>
>>>> wdyt?
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Saminda
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> System Analyst Programmer
>>> PTI Lab
>>> Indiana University
>>>
>>
>>
>
>
> --
> Thank You,
> Best Regards,
> Eroma
>
Re: Implementing the cancel/terminate
Posted by Eroma Abeysinghe <er...@gmail.com>.
Hi,
If you have bottom up we will not be able to cancel unless there is a job
available for that experiment right?
Also few questions;
1. What do we really mean by canceling? is it just a status update?
OR
2. Are we going to stop all file transfers, delete any data file/file path
existing in Airavata for that experiment/tasks/job?
3. And also are we considering both single submission and workflows or is
it just single submissions?
If we are going to consider canceling of workflows then we need to extend
cancelling to multiple tasks and jobs an experiment would have
4. Also we need to define what experiments we can cancel - IMO we don't
need to bother with COMPLETED, CANCELED, UNKNOWN, FAILED experiments and
similar statuses in tasks and jobs
IMHO i also don't think Job monitor should do any cancellations.
Thank You,
Best Regards,
Eroma
On Mon, Apr 21, 2014 at 2:37 PM, Saminda Wijeratne <sa...@gmail.com>wrote:
> May I finish setting up the framework for catching cancel requests? I'll
> finish implementing the cancel once we decide upon who will do what when
> cancelling a job.
>
> I just remembered the canceled notification would be handled by the status
> update mechanism we introduced last week. But this mechanism works only
> bottom up, i.e. Job status updates will trigger Task status updates and
> that will trigger Experiment status updates. Does it make sense to have
> "canceling" status also to progress likewise instead of top down (which i
> suggested in the first mail)?
>
>
>
> On Mon, Apr 21, 2014 at 11:11 AM, Lahiru Gunathilake <gl...@gmail.com>wrote:
>
>> Hi Raman,
>>
>>
>> On Mon, Apr 21, 2014 at 10:35 AM, Raminder Singh <
>> raminderjsingh@gmail.com> wrote:
>>
>>> Thanks for investigation the problem and working through solution. This
>>> is really required for the production gateways like Ultrascan.
>>>
>>> In the current architecture where we have job submission(provider) and
>>> monitoring separate, job cancel request need not to go to GFAC provider.
>>> Provider submits the jobs and handover the job id to the orchestrator.
>>> Orchestrator works with the job monitoring to maintain the job state. Now
>>> the cancel need to be handled by Orchestrator and Monitoring. That will
>>> change the course of action for API to cancel a job.
>>>
>>> I dont' think so, Orchestrator can invoke GFac Provider level job
>> cancellation and it should simply reflect in the monitor when it try to get
>> the status of that job( once its got to know by the monitor it should stop
>> monitoring that job) and without modifying the monitor everything should
>> work. There is no need to touch the monitor.
>>
>> I think Job cancellation should be a functionality of GFAC Provider and
>> it should be similar to job submission where you can do pre processing and
>> post processing after job cancellation operation.
>>
>>> One important requirement to take care is cleanup task after the job is
>>> canceled like updating the job status table and updating the status.
>>>
>>> Thanks
>>> Raminder
>>>
>>>
>>> On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <sa...@gmail.com>
>>> wrote:
>>>
>>> Hi All,
>>>
>>> After looking at the current design and doing some trial and error I
>>> thought of implementing the cancellation as follows.
>>>
>>>
>>> - Cancellation of an experiment requested by a gateway requires
>>> cancellation request to go through several layers. (Orchestrator > GFac >
>>> GFac Provider)
>>> - Each layer is responsible for handling cancellation relevant for
>>> that layer (Orchestration cancels experiment, GFac cancels Task, GFac
>>> Provider cancels Job)
>>> - What I thought is, each layer will listen to cancellation request
>>> made to the layer above and perform its cancellation actions accordingly.
>>> (GFac will see the experiment is having the status "canceling" for an
>>> experiment id and it will perform cancellation of the tasks relevant for
>>> that experiment)
>>> - Effectively the Orchestrator will be
>>> - updating the status of the experiment in registry with the
>>> status "canceling"
>>> - publish a message which will be caught by GFac instance
>>> which handles its Tasks.
>>> - GFac will perform the same and the correct GFac Provider
>>> instance will catch the message and perform the actual job cancellation.
>>> - Once the job cancellation is done the statuses at each layer will
>>> be updated (to "canceled") in similar fashion.
>>> - We allow the API call of cancellation to be asynchronous
>>> - I'm hoping to use the MonitorPublisher implemented by Lahiru to
>>> publish the messages.
>>>
>>> wdyt?
>>>
>>>
>>> Thanks,
>>>
>>> Saminda
>>>
>>>
>>>
>>>
>>
>>
>> --
>> System Analyst Programmer
>> PTI Lab
>> Indiana University
>>
>
>
--
Thank You,
Best Regards,
Eroma
Re: Implementing the cancel/terminate
Posted by Saminda Wijeratne <sa...@gmail.com>.
May I finish setting up the framework for catching cancel requests? I'll
finish implementing the cancel once we decide upon who will do what when
cancelling a job.
I just remembered the canceled notification would be handled by the status
update mechanism we introduced last week. But this mechanism works only
bottom up, i.e. Job status updates will trigger Task status updates and
that will trigger Experiment status updates. Does it make sense to have
"canceling" status also to progress likewise instead of top down (which i
suggested in the first mail)?
On Mon, Apr 21, 2014 at 11:11 AM, Lahiru Gunathilake <gl...@gmail.com>wrote:
> Hi Raman,
>
>
> On Mon, Apr 21, 2014 at 10:35 AM, Raminder Singh <raminderjsingh@gmail.com
> > wrote:
>
>> Thanks for investigation the problem and working through solution. This
>> is really required for the production gateways like Ultrascan.
>>
>> In the current architecture where we have job submission(provider) and
>> monitoring separate, job cancel request need not to go to GFAC provider.
>> Provider submits the jobs and handover the job id to the orchestrator.
>> Orchestrator works with the job monitoring to maintain the job state. Now
>> the cancel need to be handled by Orchestrator and Monitoring. That will
>> change the course of action for API to cancel a job.
>>
>> I dont' think so, Orchestrator can invoke GFac Provider level job
> cancellation and it should simply reflect in the monitor when it try to get
> the status of that job( once its got to know by the monitor it should stop
> monitoring that job) and without modifying the monitor everything should
> work. There is no need to touch the monitor.
>
> I think Job cancellation should be a functionality of GFAC Provider and it
> should be similar to job submission where you can do pre processing and
> post processing after job cancellation operation.
>
>> One important requirement to take care is cleanup task after the job is
>> canceled like updating the job status table and updating the status.
>>
>> Thanks
>> Raminder
>>
>>
>> On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <sa...@gmail.com>
>> wrote:
>>
>> Hi All,
>>
>> After looking at the current design and doing some trial and error I
>> thought of implementing the cancellation as follows.
>>
>>
>> - Cancellation of an experiment requested by a gateway requires
>> cancellation request to go through several layers. (Orchestrator > GFac >
>> GFac Provider)
>> - Each layer is responsible for handling cancellation relevant for
>> that layer (Orchestration cancels experiment, GFac cancels Task, GFac
>> Provider cancels Job)
>> - What I thought is, each layer will listen to cancellation request
>> made to the layer above and perform its cancellation actions accordingly.
>> (GFac will see the experiment is having the status "canceling" for an
>> experiment id and it will perform cancellation of the tasks relevant for
>> that experiment)
>> - Effectively the Orchestrator will be
>> - updating the status of the experiment in registry with the
>> status "canceling"
>> - publish a message which will be caught by GFac instance which
>> handles its Tasks.
>> - GFac will perform the same and the correct GFac Provider
>> instance will catch the message and perform the actual job cancellation.
>> - Once the job cancellation is done the statuses at each layer will
>> be updated (to "canceled") in similar fashion.
>> - We allow the API call of cancellation to be asynchronous
>> - I'm hoping to use the MonitorPublisher implemented by Lahiru to
>> publish the messages.
>>
>> wdyt?
>>
>>
>> Thanks,
>>
>> Saminda
>>
>>
>>
>>
>
>
> --
> System Analyst Programmer
> PTI Lab
> Indiana University
>
Re: Implementing the cancel/terminate
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi Raman,
On Mon, Apr 21, 2014 at 10:35 AM, Raminder Singh
<ra...@gmail.com>wrote:
> Thanks for investigation the problem and working through solution. This
> is really required for the production gateways like Ultrascan.
>
> In the current architecture where we have job submission(provider) and
> monitoring separate, job cancel request need not to go to GFAC provider.
> Provider submits the jobs and handover the job id to the orchestrator.
> Orchestrator works with the job monitoring to maintain the job state. Now
> the cancel need to be handled by Orchestrator and Monitoring. That will
> change the course of action for API to cancel a job.
>
> I dont' think so, Orchestrator can invoke GFac Provider level job
cancellation and it should simply reflect in the monitor when it try to get
the status of that job( once its got to know by the monitor it should stop
monitoring that job) and without modifying the monitor everything should
work. There is no need to touch the monitor.
I think Job cancellation should be a functionality of GFAC Provider and it
should be similar to job submission where you can do pre processing and
post processing after job cancellation operation.
> One important requirement to take care is cleanup task after the job is
> canceled like updating the job status table and updating the status.
>
> Thanks
> Raminder
>
>
> On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <sa...@gmail.com> wrote:
>
> Hi All,
>
> After looking at the current design and doing some trial and error I
> thought of implementing the cancellation as follows.
>
>
> - Cancellation of an experiment requested by a gateway requires
> cancellation request to go through several layers. (Orchestrator > GFac >
> GFac Provider)
> - Each layer is responsible for handling cancellation relevant for
> that layer (Orchestration cancels experiment, GFac cancels Task, GFac
> Provider cancels Job)
> - What I thought is, each layer will listen to cancellation request
> made to the layer above and perform its cancellation actions accordingly.
> (GFac will see the experiment is having the status "canceling" for an
> experiment id and it will perform cancellation of the tasks relevant for
> that experiment)
> - Effectively the Orchestrator will be
> - updating the status of the experiment in registry with the status
> "canceling"
> - publish a message which will be caught by GFac instance which
> handles its Tasks.
> - GFac will perform the same and the correct GFac Provider
> instance will catch the message and perform the actual job cancellation.
> - Once the job cancellation is done the statuses at each layer will be
> updated (to "canceled") in similar fashion.
> - We allow the API call of cancellation to be asynchronous
> - I'm hoping to use the MonitorPublisher implemented by Lahiru to
> publish the messages.
>
> wdyt?
>
>
> Thanks,
>
> Saminda
>
>
>
>
--
System Analyst Programmer
PTI Lab
Indiana University
Re: Implementing the cancel/terminate
Posted by Raminder Singh <ra...@gmail.com>.
Thanks for investigation the problem and working through solution. This is really required for the production gateways like Ultrascan.
In the current architecture where we have job submission(provider) and monitoring separate, job cancel request need not to go to GFAC provider. Provider submits the jobs and handover the job id to the orchestrator. Orchestrator works with the job monitoring to maintain the job state. Now the cancel need to be handled by Orchestrator and Monitoring. That will change the course of action for API to cancel a job.
One important requirement to take care is cleanup task after the job is canceled like updating the job status table and updating the status.
Thanks
Raminder
On Apr 21, 2014, at 9:06 AM, Saminda Wijeratne <sa...@gmail.com> wrote:
> Hi All,
>
> After looking at the current design and doing some trial and error I thought of implementing the cancellation as follows.
>
> Cancellation of an experiment requested by a gateway requires cancellation request to go through several layers. (Orchestrator > GFac > GFac Provider)
> Each layer is responsible for handling cancellation relevant for that layer (Orchestration cancels experiment, GFac cancels Task, GFac Provider cancels Job)
> What I thought is, each layer will listen to cancellation request made to the layer above and perform its cancellation actions accordingly. (GFac will see the experiment is having the status "canceling" for an experiment id and it will perform cancellation of the tasks relevant for that experiment)
> Effectively the Orchestrator will be
> updating the status of the experiment in registry with the status "canceling"
> publish a message which will be caught by GFac instance which handles its Tasks.
> GFac will perform the same and the correct GFac Provider instance will catch the message and perform the actual job cancellation.
> Once the job cancellation is done the statuses at each layer will be updated (to "canceled") in similar fashion.
> We allow the API call of cancellation to be asynchronous
> I'm hoping to use the MonitorPublisher implemented by Lahiru to publish the messages.
> wdyt?
>
>
>
> Thanks,
>
> Saminda
>
>