Importance of Orchestrator Component for CIPRES+Airavata Integration

[Saminda Wijeratne <sa...@gmail.com>]

Hi Devs,

These days we are working on an initial effort of integrating Airavata to
CIPRES gateway backend. Following is a short review of CIPRES requirements
against Orchestrator component design.
(For a full list of CIPRES usecases related to Airavata integration please
view [1].)

*Single Job execution vs Workflow execution*
Orchestrator supporting the single job executions benefits gateways like
CIPRES, NSG and Ultrascan since it removes the unintended overhead of
executing and managing a workflow context which is not required for those
gateways. The status, the progress messages, IDs and commands
(launch/cancel/retry) of an experiment directly correspond to the job
submissions/executions which these gateways interested in.

*Functional Requirements*
The orchestrator component supports 2 main functions which CIPRES requires
in relating to managing its user tasks. Job submission and cancellation.
Additional requirements relating to data management and progress/status
monitoring will be further discussed in order to determine the involvement
of orchestrator component to support them or not.

*Exposed API functions*
Current AiravataAPI ExecutionManager exposes workflow execution API
functions. While the functions themselves do not distinguish this fact
(i.e. runExperiment(...)), it is evident that this could confuse the
gateway developer in understanding how to specify that he/she intends to
run a single job and not a workflow. Thus for such single jobs a seperate
launch (or run) API function is required. However the rest of the
functionalities such as cancel, retrieve progress and provenance data can
be managed via the existing API functions since the concept of experiment
ID is common for single jobs and workflows.

*Accessing the Orchestrator by the gateway*
Although we have an Orchestrator API designed at the moment thoughts are to
expose an "Orchestrator Service" for the gateway developers to work with.
This will most probably be a thrift service which would give the gateway
developers the benefit of having a thrift client of their own programming
language.

Other than what was discussed in google hangout sessions and mail threads,
I do not see any other design flaws or improvements in the Orchestrator
with relating to CIPRES needs. But we need to soon start using the
component in CIPRES in order to get more feedback. I will update this mail
once I start doing that in the coming weeks.

Thanks,
Saminda

1.
https://docs.google.com/document/d/1t0dqUgknIO9OgR-INIMccyYQP_KukzNmDMXdsVv0oSc/edit#

Re: Importance of Orchestrator Component for CIPRES+Airavata Integration

[Lahiru Gunathilake <gl...@gmail.com>]

Thank you saminda for the clear information !

Regards
Lahiru


On Fri, Jan 17, 2014 at 1:55 PM, Saminda Wijeratne <sa...@gmail.com>wrote:

> Hi Devs,
>
> These days we are working on an initial effort of integrating Airavata to
> CIPRES gateway backend. Following is a short review of CIPRES requirements
> against Orchestrator component design.
> (For a full list of CIPRES usecases related to Airavata integration please
> view [1].)
>
> *Single Job execution vs Workflow execution*
> Orchestrator supporting the single job executions benefits gateways like
> CIPRES, NSG and Ultrascan since it removes the unintended overhead of
> executing and managing a workflow context which is not required for those
> gateways. The status, the progress messages, IDs and commands
> (launch/cancel/retry) of an experiment directly correspond to the job
> submissions/executions which these gateways interested in.
>
> *Functional Requirements*
> The orchestrator component supports 2 main functions which CIPRES requires
> in relating to managing its user tasks. Job submission and cancellation.
> Additional requirements relating to data management and progress/status
> monitoring will be further discussed in order to determine the involvement
> of orchestrator component to support them or not.
>
> *Exposed API functions*
> Current AiravataAPI ExecutionManager exposes workflow execution API
> functions. While the functions themselves do not distinguish this fact
> (i.e. runExperiment(...)), it is evident that this could confuse the
> gateway developer in understanding how to specify that he/she intends to
> run a single job and not a workflow. Thus for such single jobs a seperate
> launch (or run) API function is required. However the rest of the
> functionalities such as cancel, retrieve progress and provenance data can
> be managed via the existing API functions since the concept of experiment
> ID is common for single jobs and workflows.
>
> *Accessing the Orchestrator by the gateway*
> Although we have an Orchestrator API designed at the moment thoughts are
> to expose an "Orchestrator Service" for the gateway developers to work
> with. This will most probably be a thrift service which would give the
> gateway developers the benefit of having a thrift client of their own
> programming language.
>
> Other than what was discussed in google hangout sessions and mail threads,
> I do not see any other design flaws or improvements in the Orchestrator
> with relating to CIPRES needs. But we need to soon start using the
> component in CIPRES in order to get more feedback. I will update this mail
> once I start doing that in the coming weeks.
>
> Thanks,
> Saminda
>
> 1.
> https://docs.google.com/document/d/1t0dqUgknIO9OgR-INIMccyYQP_KukzNmDMXdsVv0oSc/edit#
>



-- 
System Analyst Programmer
PTI Lab
Indiana University