Re: Airavata 0.16 Design Refactoring Suggestion?

[Shameera Rathnayaka <sh...@gmail.com>]

Hi All,

Here is how the sequence diagram looks like for experiment submit request,
with the proposed changes.



​

Thanks,
Shameera.

On Sat, May 30, 2015 at 4:55 PM, Shameera Rathnayaka <sh...@apache.org>
wrote:

> Hi Devs,
>
> As we are about to release Airavata 0.15( already cut the branch ) we will
> not add any major changes and it is in testing stage. This will give us
> time to discuss and finalize requirements for the next release , it can be
> either 0.16 or 1.0.
>
>
> As per the feedback from our user community, they need more transparent
> view of what Airavata does when they submit an experiment to run a job on
> remote computer resource. Airavata users are science gateway developers,
> they are not only interested in Experiment level and remote Job level
> status changes. They would like to know some degree of transparency about
> pre-processing and post-processing tasks performed by airavata framework,
> before and after Job submission. For example they would like to see which
> task is being executed at particular time, does scp file transferring
> succeed or not. With current Hander architecture, it is not possible to
> Airavata framework to know which handler does what. User can write and
> integrate different kind of handlers and integrate it with the execution
> chain. If Airavata Job submission failed while transferring input file to
> the compute resource. Gateway developer should be able to find the reason
> without any trouble. Current Airavata save the failure reason with
> stracktrace but that is too low level for a gateway developer.
>
> Here we are thinking of replace this static handler architecture with
> dynamic task mechanism. Here framework has different type of tasks, lets
> say for input staging we have SCP , GRIDFTP and HTTP tasks. each task
> clearly know what it need to do and how. When Airavata get an experiment
> with three inputs, one is simple string and other two are SCP and HTTP type
> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to
> the dynamic task chain. Then add another Job submission task, let's say job
> need to submit using ssh keys then Airavata add SSH job submission task. as
> same add required task for the outputs. Each task has three states
> Processing, Completed, Failed. In case of failure, framework know which
> type of works it was doing or which task failed, is it SCP file transfer
> task or GRIDFTP file transfering task. Then Airavata can provide(show) this
> details to Users by messaging. Please see following diagram to get an idea
> about different level of state transitions.
>
> Yours feedback are highly appreciate. ​
>
> ​
>
>
> Thanks,
> Shameera.
>
>


-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Airavata 0.16 Design Refactoring Suggestion?

[Shameera Rathnayaka <sh...@gmail.com>]

Hi Amila,

>
> By looking at the diagram I can ask lot of questions but I guess your
> complete design is not depicted in the picture.
>

​Yes sequence diagram is not descriptive enough to explain the whole
design. But this will help to get some idea about experiment execution
path.


> Few quick questions are as follows;
>
> 1. What is API server ? (sorry I dont know what this is) and what is the
> purpose of that ?
>

API server is already a part of existing design, which will allow different
clients to interact with Airavata components. All client requests go
through this.
​


> 2. Is experiment id generated by the PHP gateway (as per diagram that is
> the idea i got) ? (i.e. by client side not Airavata server) If so I would
> like to know how it works.
>

​This diagram only explain submit request path ( I already have mentioned
that ), experiment creation is not explain by this sequence diagram. Yes
experiment Id will be generated by server side.
​

> 3. When you first mention about tasks I was under the impression that GFac
> is going to spawn different tasks for a process. i.e. Scp task, Qsub task
> etc. but diagram doesnt exactly depict that behaviour.
>

​Dynamic task chain will be generated with process creation time at
orchestration level. Gfac will invoke those tasks. ​I agree, diagram
doesn't show those details.

4. Is monitoring also a task ?
>

​Nope, monitoring is not a task. It is a common service for all tasks which
will handle monitoring.


> 5. Seems some tasks need to share information among other components (or
> other tasks). E.g:- job id from job submission task. How are you planning
> to share those data ?
>

​There will be process context, which acts as container for that process.
each task will get this process context as a parameter.


> 6. Also diagram doesnt show how this design resolves your original
> question ("transparency"); i.e. sending feedback about each task to amqp
>

​As I have explained above, we are trying to solve two problems, and
transparency is one of them. Yes, amqp messages will be used to publish
status changes.


> 7. Also did you consider FT in the design ? It seems tasks have a pretty
> important role to acheive FT. E.g:- The job submission task
>

​Plan is to use zookeeper + work queue FT mechanism which we currently
have. But would like to talk about other alternatives too.


>
> I remember we discussing a design on FT sometime back and I believe
> Saminda came up with few nice diagrams that handle most of the failure
> cases. I suggest you looking at them if you are planning to consider FT in
> your design.
>

​Sure will have a look.


Thanks,
Shameera.
​


>
> Thanks
> -Thejaka
>
>
> On Tue, Jun 2, 2015 at 6:03 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi All,
>>
>> Here is how the sequence diagram looks like for experiment submit
>> request, with the proposed changes.
>>
>>
>>
>> ​
>>
>> Thanks,
>> Shameera.
>>
>> On Sat, May 30, 2015 at 4:55 PM, Shameera Rathnayaka <shameera@apache.org
>> > wrote:
>>
>>> Hi Devs,
>>>
>>> As we are about to release Airavata 0.15( already cut the branch ) we
>>> will not add any major changes and it is in testing stage. This will give
>>> us time to discuss and finalize requirements for the next release , it can
>>> be either 0.16 or 1.0.
>>>
>>>
>>> As per the feedback from our user community, they need more transparent
>>> view of what Airavata does when they submit an experiment to run a job on
>>> remote computer resource. Airavata users are science gateway developers,
>>> they are not only interested in Experiment level and remote Job level
>>> status changes. They would like to know some degree of transparency about
>>> pre-processing and post-processing tasks performed by airavata framework,
>>> before and after Job submission. For example they would like to see which
>>> task is being executed at particular time, does scp file transferring
>>> succeed or not. With current Hander architecture, it is not possible to
>>> Airavata framework to know which handler does what. User can write and
>>> integrate different kind of handlers and integrate it with the execution
>>> chain. If Airavata Job submission failed while transferring input file to
>>> the compute resource. Gateway developer should be able to find the reason
>>> without any trouble. Current Airavata save the failure reason with
>>> stracktrace but that is too low level for a gateway developer.
>>>
>>> Here we are thinking of replace this static handler architecture with
>>> dynamic task mechanism. Here framework has different type of tasks, lets
>>> say for input staging we have SCP , GRIDFTP and HTTP tasks. each task
>>> clearly know what it need to do and how. When Airavata get an experiment
>>> with three inputs, one is simple string and other two are SCP and HTTP type
>>> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to
>>> the dynamic task chain. Then add another Job submission task, let's say job
>>> need to submit using ssh keys then Airavata add SSH job submission task. as
>>> same add required task for the outputs. Each task has three states
>>> Processing, Completed, Failed. In case of failure, framework know which
>>> type of works it was doing or which task failed, is it SCP file transfer
>>> task or GRIDFTP file transfering task. Then Airavata can provide(show) this
>>> details to Users by messaging. Please see following diagram to get an idea
>>> about different level of state transitions.
>>>
>>> Yours feedback are highly appreciate. ​
>>>
>>> ​
>>>
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>


-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Airavata 0.16 Design Refactoring Suggestion?

[Amila Jayasekara <th...@gmail.com>]

Hi Shameera,

By looking at the diagram I can ask lot of questions but I guess your
complete design is not depicted in the picture.

Few quick questions are as follows;

1. What is API server ? (sorry I dont know what this is) and what is the
purpose of that ?
2. Is experiment id generated by the PHP gateway (as per diagram that is
the idea i got) ? (i.e. by client side not Airavata server) If so I would
like to know how it works.
3. When you first mention about tasks I was under the impression that GFac
is going to spawn different tasks for a process. i.e. Scp task, Qsub task
etc. but diagram doesnt exactly depict that behaviour.
4. Is monitoring also a task ?
5. Seems some tasks need to share information among other components (or
other tasks). E.g:- job id from job submission task. How are you planning
to share those data ?
6. Also diagram doesnt show how this design resolves your original question
("transparency"); i.e. sending feedback about each task to amqp
7. Also did you consider FT in the design ? It seems tasks have a pretty
important role to acheive FT. E.g:- The job submission task

I remember we discussing a design on FT sometime back and I believe Saminda
came up with few nice diagrams that handle most of the failure cases. I
suggest you looking at them if you are planning to consider FT in your
design.

Thanks
-Thejaka


On Tue, Jun 2, 2015 at 6:03 PM, Shameera Rathnayaka <sh...@gmail.com>
wrote:

> Hi All,
>
> Here is how the sequence diagram looks like for experiment submit request,
> with the proposed changes.
>
>
>
> ​
>
> Thanks,
> Shameera.
>
> On Sat, May 30, 2015 at 4:55 PM, Shameera Rathnayaka <sh...@apache.org>
> wrote:
>
>> Hi Devs,
>>
>> As we are about to release Airavata 0.15( already cut the branch ) we
>> will not add any major changes and it is in testing stage. This will give
>> us time to discuss and finalize requirements for the next release , it can
>> be either 0.16 or 1.0.
>>
>>
>> As per the feedback from our user community, they need more transparent
>> view of what Airavata does when they submit an experiment to run a job on
>> remote computer resource. Airavata users are science gateway developers,
>> they are not only interested in Experiment level and remote Job level
>> status changes. They would like to know some degree of transparency about
>> pre-processing and post-processing tasks performed by airavata framework,
>> before and after Job submission. For example they would like to see which
>> task is being executed at particular time, does scp file transferring
>> succeed or not. With current Hander architecture, it is not possible to
>> Airavata framework to know which handler does what. User can write and
>> integrate different kind of handlers and integrate it with the execution
>> chain. If Airavata Job submission failed while transferring input file to
>> the compute resource. Gateway developer should be able to find the reason
>> without any trouble. Current Airavata save the failure reason with
>> stracktrace but that is too low level for a gateway developer.
>>
>> Here we are thinking of replace this static handler architecture with
>> dynamic task mechanism. Here framework has different type of tasks, lets
>> say for input staging we have SCP , GRIDFTP and HTTP tasks. each task
>> clearly know what it need to do and how. When Airavata get an experiment
>> with three inputs, one is simple string and other two are SCP and HTTP type
>> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to
>> the dynamic task chain. Then add another Job submission task, let's say job
>> need to submit using ssh keys then Airavata add SSH job submission task. as
>> same add required task for the outputs. Each task has three states
>> Processing, Completed, Failed. In case of failure, framework know which
>> type of works it was doing or which task failed, is it SCP file transfer
>> task or GRIDFTP file transfering task. Then Airavata can provide(show) this
>> details to Users by messaging. Please see following diagram to get an idea
>> about different level of state transitions.
>>
>> Yours feedback are highly appreciate. ​
>>
>> ​
>>
>>
>> Thanks,
>> Shameera.
>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>

Re: Airavata 0.16 Design Refactoring Suggestion?

[Suresh Marru <sm...@apache.org>]

Hi Shameera,

This looks good. I have a cosmetic suggestion not related to the design. 

While we re-organize the code, we probably also want to rename the components so they are self-explanatory. 

The monitor in your diagram should be made more concrete like “HPC Job Monitor”. 

I think the name GFac has lived its life. This component by itself is no longer a Generic Application Factory. We should consider a new name which explains what it is doing. One though is Task Executor, other suggestions? 

Suresh

> On Jun 2, 2015, at 6:03 PM, Shameera Rathnayaka <sh...@gmail.com> wrote:
> 
> Hi All, 
> 
> Here is how the sequence diagram looks like for experiment submit request, with the proposed changes. 
> 
> 
> <Airavata 0.16 Design - Sequence Diagram.jpg>
> ​
> 
> Thanks,
> Shameera.
> 
> On Sat, May 30, 2015 at 4:55 PM, Shameera Rathnayaka <shameera@apache.org <ma...@apache.org>> wrote:
> Hi Devs, 
> 
> As we are about to release Airavata 0.15( already cut the branch ) we will not add any major changes and it is in testing stage. This will give us time to discuss and finalize requirements for the next release , it can be either 0.16 or 1.0. 
> 
> 
> As per the feedback from our user community, they need more transparent view of what Airavata does when they submit an experiment to run a job on remote computer resource. Airavata users are science gateway developers, they are not only interested in Experiment level and remote Job level status changes. They would like to know some degree of transparency about pre-processing and post-processing tasks performed by airavata framework, before and after Job submission. For example they would like to see which task is being executed at particular time, does scp file transferring succeed or not. With current Hander architecture, it is not possible to Airavata framework to know which handler does what. User can write and integrate different kind of handlers and integrate it with the execution chain. If Airavata Job submission failed while transferring input file to the compute resource. Gateway developer should be able to find the reason without any trouble. Current Airavata save the failure reason with stracktrace but that is too low level for a gateway developer. 
> 
> Here we are thinking of replace this static handler architecture with dynamic task mechanism. Here framework has different type of tasks, lets say for input staging we have SCP , GRIDFTP and HTTP tasks. each task clearly know what it need to do and how. When Airavata get an experiment with three inputs, one is simple string and other two are SCP and HTTP type file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to the dynamic task chain. Then add another Job submission task, let's say job need to submit using ssh keys then Airavata add SSH job submission task. as same add required task for the outputs. Each task has three states Processing, Completed, Failed. In case of failure, framework know which type of works it was doing or which task failed, is it SCP file transfer task or GRIDFTP file transfering task. Then Airavata can provide(show) this details to Users by messaging. Please see following diagram to get an idea about different level of state transitions.
> 
> Yours feedback are highly appreciate. ​
> <Airavata 0.16 Design.png>
> ​
> 
> 
> Thanks,
> Shameera.
> 
> 
> 
> 
> -- 
> Best Regards,
> Shameera Rathnayaka.
> 
> email: shameera AT apache.org <http://apache.org/> , shameerainfo AT gmail.com <http://gmail.com/>
> Blog : http://shameerarathnayaka.blogspot.com/ <http://shameerarathnayaka.blogspot.com/>