You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by Raminder Singh <ra...@gmail.com> on 2013/07/16 17:58:47 UTC
Synchronous vs Async submissions in GFAC provider
I am working on a use case where these is an asynchronous job submission script for pegasus[1] (Condor Dagman [2]). Submission scripts returns a JOBID to monitor the status. Monitor script returns the job status using the JOBID provided by submission script. Another script is available to cancel the job.
I plan to use SSHProvider to call these scripts. Currently provider only support syncronous submission and does not allow to add a wait thread using another script. According to me, this can be common pattern among other gateway use cases. This case is true for LocalProvider also. I can add another provider for my specific case but i think this is a generic case of airavata and need discussion.
If we want to add asyncronous submission and monitoring to current Local and SSH providers following changes are needed
1. Add flag to descriptor to identify execution pattern.
2. Change descriptor to allow users to add monitoring script path.
3. Incase we want to support job cancel, cancel script path also need to be added to descriptors also.
4. Monitoring pattern and status (to make decision) need to be configured using some interface.
Adding Async Local and SSH provider for this case is another option and new descriptors can be added.
Thoughts?
Thanks
Raminder
Re: Synchronous vs Async submissions in GFAC provider
Posted by Raminder Singh <ra...@gmail.com>.
Lahiru,
Async or Sync is define time step. User will know the nature of executable and can configure it while defining its application description. We need other information from user as well like which command or executable to call to monitor status and what status decisions are going to be. I can't think about a case when user want to change these properties at run time.
WDYT?
Raman
On Jul 16, 2013, at 1:57 PM, Lahiru Gunathilake <gl...@gmail.com> wrote:
>
>
>
> On Tue, Jul 16, 2013 at 11:23 PM, Amila Jayasekara <th...@gmail.com> wrote:
> +1 Raman.
> Is this going to be in 0.9 ? If so can you please update feature list for 0.9 also ?
>
>
> On Tue, Jul 16, 2013 at 11:58 AM, Raminder Singh <ra...@gmail.com> wrote:
> I am working on a use case where these is an asynchronous job submission script for pegasus[1] (Condor Dagman [2]). Submission scripts returns a JOBID to monitor the status.
>
> Does pegasus generate above job id ?
>
>
> Monitor script returns the job status using the JOBID provided by submission script. Another script is available to cancel the job.
>
> I plan to use SSHProvider to call these scripts. Currently provider only support syncronous submission and does not allow to add a wait thread using another script. According to me, this can be common pattern among other gateway use cases. This case is true for LocalProvider also. I can add another provider for my specific case but i think this is a generic case of airavata and need discussion.
>
> Are these scripts specific to pegasus ? If scripts are generic and can provide information about running jobs its good to change existing providers rather than introducing new one.
>
>
> If we want to add asyncronous submission and monitoring to current Local and SSH providers following changes are needed
>
> 1. Add flag to descriptor to identify execution pattern.
> 2. Change descriptor to allow users to add monitoring script path.
> 3. Incase we want to support job cancel, cancel script path also need to be added to descriptors also.
> 4. Monitoring pattern and status (to make decision) need to be configured using some interface.
>
> In addition to specifying specific script paths can we generate some generic scripts to do job monitoring and job cancellation. For example for monitoring we can check the process id status. For cancellation we can execute a command like "kill -TERM <jobid>".
>
> Further can we make async and sync as an option in the same provider. Rather than have a separate provider for Async operations.
> We can specify this WorkflowContext header and keep the synchronous submission as the default one.
>
> WDYT ?
>
> Lahiru
>
> Thanks
> Amila
>
> Adding Async Local and SSH provider for this case is another option and new descriptors can be added.
>
> Thoughts?
>
> Thanks
> Raminder
>
>
>
>
>
>
>
> --
> System Analyst Programmer
> PTI Lab
> Indiana University
Re: Synchronous vs Async submissions in GFAC provider
Posted by Raminder Singh <ra...@gmail.com>.
Yes, handling this in existing providers is a good idea. This scripts concept is generic and can be used for other applications. These is nothing specific to Pegasus[1]. In current script job id is generated by Pegasus but any other job id like process id should work also. Flow for Async provider call will be
1. Provider will call the executable with input parameter and read the JOBID from stdout file.
2. JobID will be saved to registry for gfac restart case.
2. Provider listener will start for job status on a poll frequency(configurable) and make a decision based on status value. Status conditions can be configurable from descriptor or configuration file.
3. On Success/Failure listener will terminate and output handler will be called.
Thanks Amila and Lahiru for your feedback.
Raminder
1. http://pegasus.isi.edu/
On Jul 16, 2013, at 2:00 PM, Amila Jayasekara <th...@gmail.com> wrote:
>
>
>
> On Tue, Jul 16, 2013 at 1:57 PM, Lahiru Gunathilake <gl...@gmail.com> wrote:
>
>
>
> On Tue, Jul 16, 2013 at 11:23 PM, Amila Jayasekara <th...@gmail.com> wrote:
> +1 Raman.
> Is this going to be in 0.9 ? If so can you please update feature list for 0.9 also ?
>
>
> On Tue, Jul 16, 2013 at 11:58 AM, Raminder Singh <ra...@gmail.com> wrote:
> I am working on a use case where these is an asynchronous job submission script for pegasus[1] (Condor Dagman [2]). Submission scripts returns a JOBID to monitor the status.
>
> Does pegasus generate above job id ?
>
>
> Monitor script returns the job status using the JOBID provided by submission script. Another script is available to cancel the job.
>
> I plan to use SSHProvider to call these scripts. Currently provider only support syncronous submission and does not allow to add a wait thread using another script. According to me, this can be common pattern among other gateway use cases. This case is true for LocalProvider also. I can add another provider for my specific case but i think this is a generic case of airavata and need discussion.
>
> Are these scripts specific to pegasus ? If scripts are generic and can provide information about running jobs its good to change existing providers rather than introducing new one.
>
>
> If we want to add asyncronous submission and monitoring to current Local and SSH providers following changes are needed
>
> 1. Add flag to descriptor to identify execution pattern.
> 2. Change descriptor to allow users to add monitoring script path.
> 3. Incase we want to support job cancel, cancel script path also need to be added to descriptors also.
> 4. Monitoring pattern and status (to make decision) need to be configured using some interface.
>
> In addition to specifying specific script paths can we generate some generic scripts to do job monitoring and job cancellation. For example for monitoring we can check the process id status. For cancellation we can execute a command like "kill -TERM <jobid>".
>
> Further can we make async and sync as an option in the same provider. Rather than have a separate provider for Async operations.
> We can specify this WorkflowContext header and keep the synchronous submission as the default one.
>
> WDYT ?
> +1
> -AJ
>
> Lahiru
>
> Thanks
> Amila
>
> Adding Async Local and SSH provider for this case is another option and new descriptors can be added.
>
> Thoughts?
>
> Thanks
> Raminder
>
>
>
>
>
>
>
> --
> System Analyst Programmer
> PTI Lab
> Indiana University
>
Re: Synchronous vs Async submissions in GFAC provider
Posted by Amila Jayasekara <th...@gmail.com>.
On Tue, Jul 16, 2013 at 1:57 PM, Lahiru Gunathilake <gl...@gmail.com>wrote:
>
>
>
> On Tue, Jul 16, 2013 at 11:23 PM, Amila Jayasekara <
> thejaka.amila@gmail.com> wrote:
>
>> +1 Raman.
>> Is this going to be in 0.9 ? If so can you please update feature list for
>> 0.9 also ?
>>
>>
>> On Tue, Jul 16, 2013 at 11:58 AM, Raminder Singh <
>> raminderjsingh@gmail.com> wrote:
>>
>>> I am working on a use case where these is an asynchronous job submission
>>> script for pegasus[1] (Condor Dagman [2]). Submission scripts returns a
>>> JOBID to monitor the status.
>>
>>
>> Does pegasus generate above job id ?
>>
>>
>>
>>> Monitor script returns the job status using the JOBID provided by
>>> submission script. Another script is available to cancel the job.
>>>
>>> I plan to use SSHProvider to call these scripts. Currently provider only
>>> support syncronous submission and does not allow to add a wait thread using
>>> another script. According to me, this can be common pattern among other
>>> gateway use cases. This case is true for LocalProvider also. I can add
>>> another provider for my specific case but i think this is a generic case of
>>> airavata and need discussion.
>>>
>>
>> Are these scripts specific to pegasus ? If scripts are generic and can
>> provide information about running jobs its good to change existing
>> providers rather than introducing new one.
>>
>>
>>>
>>> If we want to add asyncronous submission and monitoring to current Local
>>> and SSH providers following changes are needed
>>>
>>> 1. Add flag to descriptor to identify execution pattern.
>>> 2. Change descriptor to allow users to add monitoring script path.
>>> 3. Incase we want to support job cancel, cancel script path also need to
>>> be added to descriptors also.
>>> 4. Monitoring pattern and status (to make decision) need to be
>>> configured using some interface.
>>>
>>
>> In addition to specifying specific script paths can we generate some
>> generic scripts to do job monitoring and job cancellation. For example for
>> monitoring we can check the process id status. For cancellation we can
>> execute a command like "kill -TERM <jobid>".
>>
>> Further can we make async and sync as an option in the same provider.
>> Rather than have a separate provider for Async operations.
>>
> We can specify this WorkflowContext header and keep the synchronous
> submission as the default one.
>
> WDYT ?
>
+1
-AJ
>
> Lahiru
>
>>
>> Thanks
>> Amila
>>
>>>
>>> Adding Async Local and SSH provider for this case is another option and
>>> new descriptors can be added.
>>>
>>> Thoughts?
>>>
>>> Thanks
>>> Raminder
>>>
>>>
>>>
>>
>>
>>
>
>
> --
> System Analyst Programmer
> PTI Lab
> Indiana University
>
Re: Synchronous vs Async submissions in GFAC provider
Posted by Lahiru Gunathilake <gl...@gmail.com>.
On Tue, Jul 16, 2013 at 11:23 PM, Amila Jayasekara
<th...@gmail.com>wrote:
> +1 Raman.
> Is this going to be in 0.9 ? If so can you please update feature list for
> 0.9 also ?
>
>
> On Tue, Jul 16, 2013 at 11:58 AM, Raminder Singh <raminderjsingh@gmail.com
> > wrote:
>
>> I am working on a use case where these is an asynchronous job submission
>> script for pegasus[1] (Condor Dagman [2]). Submission scripts returns a
>> JOBID to monitor the status.
>
>
> Does pegasus generate above job id ?
>
>
>
>> Monitor script returns the job status using the JOBID provided by
>> submission script. Another script is available to cancel the job.
>>
>> I plan to use SSHProvider to call these scripts. Currently provider only
>> support syncronous submission and does not allow to add a wait thread using
>> another script. According to me, this can be common pattern among other
>> gateway use cases. This case is true for LocalProvider also. I can add
>> another provider for my specific case but i think this is a generic case of
>> airavata and need discussion.
>>
>
> Are these scripts specific to pegasus ? If scripts are generic and can
> provide information about running jobs its good to change existing
> providers rather than introducing new one.
>
>
>>
>> If we want to add asyncronous submission and monitoring to current Local
>> and SSH providers following changes are needed
>>
>> 1. Add flag to descriptor to identify execution pattern.
>> 2. Change descriptor to allow users to add monitoring script path.
>> 3. Incase we want to support job cancel, cancel script path also need to
>> be added to descriptors also.
>> 4. Monitoring pattern and status (to make decision) need to be configured
>> using some interface.
>>
>
> In addition to specifying specific script paths can we generate some
> generic scripts to do job monitoring and job cancellation. For example for
> monitoring we can check the process id status. For cancellation we can
> execute a command like "kill -TERM <jobid>".
>
> Further can we make async and sync as an option in the same provider.
> Rather than have a separate provider for Async operations.
>
We can specify this WorkflowContext header and keep the synchronous
submission as the default one.
WDYT ?
Lahiru
>
> Thanks
> Amila
>
>>
>> Adding Async Local and SSH provider for this case is another option and
>> new descriptors can be added.
>>
>> Thoughts?
>>
>> Thanks
>> Raminder
>>
>>
>>
>
>
>
--
System Analyst Programmer
PTI Lab
Indiana University
Re: Synchronous vs Async submissions in GFAC provider
Posted by Amila Jayasekara <th...@gmail.com>.
+1 Raman.
Is this going to be in 0.9 ? If so can you please update feature list for
0.9 also ?
On Tue, Jul 16, 2013 at 11:58 AM, Raminder Singh
<ra...@gmail.com>wrote:
> I am working on a use case where these is an asynchronous job submission
> script for pegasus[1] (Condor Dagman [2]). Submission scripts returns a
> JOBID to monitor the status.
Does pegasus generate above job id ?
> Monitor script returns the job status using the JOBID provided by
> submission script. Another script is available to cancel the job.
>
> I plan to use SSHProvider to call these scripts. Currently provider only
> support syncronous submission and does not allow to add a wait thread using
> another script. According to me, this can be common pattern among other
> gateway use cases. This case is true for LocalProvider also. I can add
> another provider for my specific case but i think this is a generic case of
> airavata and need discussion.
>
Are these scripts specific to pegasus ? If scripts are generic and can
provide information about running jobs its good to change existing
providers rather than introducing new one.
>
> If we want to add asyncronous submission and monitoring to current Local
> and SSH providers following changes are needed
>
> 1. Add flag to descriptor to identify execution pattern.
> 2. Change descriptor to allow users to add monitoring script path.
> 3. Incase we want to support job cancel, cancel script path also need to
> be added to descriptors also.
> 4. Monitoring pattern and status (to make decision) need to be configured
> using some interface.
>
In addition to specifying specific script paths can we generate some
generic scripts to do job monitoring and job cancellation. For example for
monitoring we can check the process id status. For cancellation we can
execute a command like "kill -TERM <jobid>".
Further can we make async and sync as an option in the same provider.
Rather than have a separate provider for Async operations.
Thanks
Amila
>
> Adding Async Local and SSH provider for this case is another option and
> new descriptors can be added.
>
> Thoughts?
>
> Thanks
> Raminder
>
>
>