Airavata 0.16 Design Refactoring Suggestion?

[Shameera Rathnayaka <sh...@apache.org>]

Hi Devs,

As we are about to release Airavata 0.15( already cut the branch ) we will
not add any major changes and it is in testing stage. This will give us
time to discuss and finalize requirements for the next release , it can be
either 0.16 or 1.0.


As per the feedback from our user community, they need more transparent
view of what Airavata does when they submit an experiment to run a job on
remote computer resource. Airavata users are science gateway developers,
they are not only interested in Experiment level and remote Job level
status changes. They would like to know some degree of transparency about
pre-processing and post-processing tasks performed by airavata framework,
before and after Job submission. For example they would like to see which
task is being executed at particular time, does scp file transferring
succeed or not. With current Hander architecture, it is not possible to
Airavata framework to know which handler does what. User can write and
integrate different kind of handlers and integrate it with the execution
chain. If Airavata Job submission failed while transferring input file to
the compute resource. Gateway developer should be able to find the reason
without any trouble. Current Airavata save the failure reason with
stracktrace but that is too low level for a gateway developer.

Here we are thinking of replace this static handler architecture with
dynamic task mechanism. Here framework has different type of tasks, lets
say for input staging we have SCP , GRIDFTP and HTTP tasks. each task
clearly know what it need to do and how. When Airavata get an experiment
with three inputs, one is simple string and other two are SCP and HTTP type
file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to
the dynamic task chain. Then add another Job submission task, let's say job
need to submit using ssh keys then Airavata add SSH job submission task. as
same add required task for the outputs. Each task has three states
Processing, Completed, Failed. In case of failure, framework know which
type of works it was doing or which task failed, is it SCP file transfer
task or GRIDFTP file transfering task. Then Airavata can provide(show) this
details to Users by messaging. Please see following diagram to get an idea
about different level of state transitions.

Yours feedback are highly appreciate. ​

​


Thanks,
Shameera.

Re: Airavata 0.16 Design Refactoring Suggestion?

[Borries Demeler <de...@biochem.uthscsa.edu>]

There was no attachment.

-b.
> On Sun, May 31, 2015 at 12:01 AM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi Gary,
>>
>> I converted the image to pdf and attached here. in both mails i can see
>> the image, wonder how you all not getting that image. Please let me know
>> if
>> you still can't see it.
>>
>> Thanks,
>> Shameera.
>>
>> On Sat, May 30, 2015 at 9:43 PM, Gary E. Gorbet <ge...@gmail.com>
>> wrote:
>>
>>>
>>> > On May 30, 2015, at 5:32 PM, Shameera Rathnayaka <
>>> shameerainfo@gmail.com> wrote:
>>> >
>>> > Hi Gary,
>>> >
>>> > I inserted the diagram as image, let me attache it as attachment.
>>>
>>> I still see nothing.
>>> - Gary
>>> >
>>> > Thanks,
>>> > Shameera.
>>> >
>>> > On Sat, May 30, 2015 at 5:35 PM, Gary E. Gorbet <ge...@gmail.com>
>>> wrote:
>>> > Shameera,
>>> >
>>> > On my copy of this email there was no attachment, no following
>>> diagram.
>>> Would you please send that attachment or a URL that points to it.
>>> >
>>> > Thanks,
>>> > Gary
>>> >
>>> > > On May 30, 2015, at 3:55 PM, Shameera Rathnayaka
>>> <sh...@apache.org>
>>> wrote:
>>> > >
>>> > > Hi Devs,
>>> > >
>>> > > As we are about to release Airavata 0.15( already cut the branch )
>>> we
>>> will not add any major changes and it is in testing stage. This will
>>> give
>>> us time to discuss and finalize requirements for the next release , it
>>> can
>>> be either 0.16 or 1.0.
>>> > >
>>> > >
>>> > > As per the feedback from our user community, they need more
>>> transparent view of what Airavata does when they submit an experiment
>>> to
>>> run a job on remote computer resource. Airavata users are science
>>> gateway
>>> developers, they are not only interested in Experiment level and remote
>>> Job
>>> level status changes. They would like to know some degree of
>>> transparency
>>> about pre-processing and post-processing tasks performed by airavata
>>> framework, before and after Job submission. For example they would like
>>> to
>>> see which task is being executed at particular time, does scp file
>>> transferring succeed or not. With current Hander architecture, it is
>>> not
>>> possible to Airavata framework to know which handler does what. User
>>> can
>>> write and integrate different kind of handlers and integrate it with
>>> the
>>> execution chain. If Airavata Job submission failed while transferring
>>> input
>>> file to the compute resource. Gateway developer should be able to find
>>> the
>>> reason without any trouble. Current Airavata save the failure reason
>>> with
>>> stracktrace but that is too low level for a gateway developer.
>>> > >
>>> > > Here we are thinking of replace this static handler architecture
>>> with
>>> dynamic task mechanism. Here framework has different type of tasks,
>>> lets
>>> say for input staging we have SCP , GRIDFTP and HTTP tasks. each task
>>> clearly know what it need to do and how. When Airavata get an
>>> experiment
>>> with three inputs, one is simple string and other two are SCP and HTTP
>>> type
>>> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks
>>> to
>>> the dynamic task chain. Then add another Job submission task, let's say
>>> job
>>> need to submit using ssh keys then Airavata add SSH job submission
>>> task. as
>>> same add required task for the outputs. Each task has three states
>>> Processing, Completed, Failed. In case of failure, framework know which
>>> type of works it was doing or which task failed, is it SCP file
>>> transfer
>>> task or GRIDFTP file transfering task. Then Airavata can provide(show)
>>> this
>>> details to Users by messaging. Please see following diagram to get an
>>> idea
>>> about different level of state transitions.
>>> > >
>>> > > Yours feedback are highly appreciate. ​
>>> > >
>>> > > ​
>>> > >
>>> > >
>>> > > Thanks,
>>> > > Shameera.
>>> > >
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Best Regards,
>>> > Shameera Rathnayaka.
>>> >
>>> > email: shameera AT apache.org , shameerainfo AT gmail.com
>>> > Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>


-- 
Borries Demeler, Ph.D.
Associate Professor
The University of Texas Health Science Center at San Antonio
Dept. of Biochemistry
Email: demeler@biochem.uthscsa.edu

Re: Airavata 0.16 Design Refactoring Suggestion?

[Shameera Rathnayaka <sh...@apache.org>]

On Sun, May 31, 2015 at 12:01 AM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:

> Hi Gary,
>
> I converted the image to pdf and attached here. in both mails i can see
> the image, wonder how you all not getting that image. Please let me know if
> you still can't see it.
>
> Thanks,
> Shameera.
>
> On Sat, May 30, 2015 at 9:43 PM, Gary E. Gorbet <ge...@gmail.com>
> wrote:
>
>>
>> > On May 30, 2015, at 5:32 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>> >
>> > Hi Gary,
>> >
>> > I inserted the diagram as image, let me attache it as attachment.
>>
>> I still see nothing.
>> - Gary
>> >
>> > Thanks,
>> > Shameera.
>> >
>> > On Sat, May 30, 2015 at 5:35 PM, Gary E. Gorbet <ge...@gmail.com>
>> wrote:
>> > Shameera,
>> >
>> > On my copy of this email there was no attachment, no following diagram.
>> Would you please send that attachment or a URL that points to it.
>> >
>> > Thanks,
>> > Gary
>> >
>> > > On May 30, 2015, at 3:55 PM, Shameera Rathnayaka <sh...@apache.org>
>> wrote:
>> > >
>> > > Hi Devs,
>> > >
>> > > As we are about to release Airavata 0.15( already cut the branch ) we
>> will not add any major changes and it is in testing stage. This will give
>> us time to discuss and finalize requirements for the next release , it can
>> be either 0.16 or 1.0.
>> > >
>> > >
>> > > As per the feedback from our user community, they need more
>> transparent view of what Airavata does when they submit an experiment to
>> run a job on remote computer resource. Airavata users are science gateway
>> developers, they are not only interested in Experiment level and remote Job
>> level status changes. They would like to know some degree of transparency
>> about pre-processing and post-processing tasks performed by airavata
>> framework, before and after Job submission. For example they would like to
>> see which task is being executed at particular time, does scp file
>> transferring succeed or not. With current Hander architecture, it is not
>> possible to Airavata framework to know which handler does what. User can
>> write and integrate different kind of handlers and integrate it with the
>> execution chain. If Airavata Job submission failed while transferring input
>> file to the compute resource. Gateway developer should be able to find the
>> reason without any trouble. Current Airavata save the failure reason with
>> stracktrace but that is too low level for a gateway developer.
>> > >
>> > > Here we are thinking of replace this static handler architecture with
>> dynamic task mechanism. Here framework has different type of tasks, lets
>> say for input staging we have SCP , GRIDFTP and HTTP tasks. each task
>> clearly know what it need to do and how. When Airavata get an experiment
>> with three inputs, one is simple string and other two are SCP and HTTP type
>> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to
>> the dynamic task chain. Then add another Job submission task, let's say job
>> need to submit using ssh keys then Airavata add SSH job submission task. as
>> same add required task for the outputs. Each task has three states
>> Processing, Completed, Failed. In case of failure, framework know which
>> type of works it was doing or which task failed, is it SCP file transfer
>> task or GRIDFTP file transfering task. Then Airavata can provide(show) this
>> details to Users by messaging. Please see following diagram to get an idea
>> about different level of state transitions.
>> > >
>> > > Yours feedback are highly appreciate. ​
>> > >
>> > > ​
>> > >
>> > >
>> > > Thanks,
>> > > Shameera.
>> > >
>> >
>> >
>> >
>> >
>> > --
>> > Best Regards,
>> > Shameera Rathnayaka.
>> >
>> > email: shameera AT apache.org , shameerainfo AT gmail.com
>> > Blog : http://shameerarathnayaka.blogspot.com/
>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>

Re: Airavata 0.16 Design Refactoring Suggestion?

[Suresh Marru <sm...@apache.org>]

On Jun 1, 2015, at 10:11 AM, Amila Jayasekara <th...@gmail.com> wrote:
> 
> Hi Shameera,
> 
> I am also having little difficulty relating the problem you are stating and
> the solution approach you are proposing. This is maybe because I am
> outdated with Airavata developments.
> 
> As per my understanding (from your description) "transparency" is the
> problem. One question about the problem is what level of transparency are
> you expecting ? Is it transparency depicted in log files or feedback to
> user front-ends etc. You may need to clarify this.
> 
> Also if it is the transparency in logs or user front-ends, then the
> solution you are proposing doesn't fit exactly to the problem (to my
> understanding).
> 
> On the other hand task based approach is a nice way to handle jobs. I guess
> you need to design a schedular to execute these tasks also. Some things you
> need to be concerned about when designing such a schedular are as follows;
> 
> 1. Some tasks have dependencies and ordering. Probably dependencies and
> ordering is defined by user (just like they way handlers are defined in
> gfac config files)
> 2. Would be nice to execute tasks in parallel whenever they dont have
> dependencies
> 3. Also be concerned about FT aspects when designing such a schedular.
> State of the schedular and also state of the individual tasks may need to
> be replicated over all nodes.

Thanks Amila for enumerating these important desired characteristics. As part of the FT you mention in 3, it will be important to checkout and recover capabilities in the tasks. 

Suresh

> 
> Thanks
> -Thejaka Amila
> 
> 
> 
> 
> On Mon, Jun 1, 2015 at 2:09 AM, Sachith Withana <sw...@gmail.com> wrote:
> 
>> Hi Shameera,
>> 
>> Wouldn't a job specific distributed logging mechanism solve the problem
>> that you mentioned? Allow the user/GWDeveloper to specify which
>> logs/messages
>> they want to see ( apart from that log everything anyways).
>> 
>> 
>> 
>> On Sun, May 31, 2015 at 8:57 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com
>>> wrote:
>> 
>>> Hi Gary & Borries,
>>> 
>>> It seems architecture mailing list doesn't show(drop) attachments, here
>> is
>>> the link
>>> https://creately.com/diagram/iaadwt6t1/yWLDNUPgfz60jl4k8WVH0Up3mk%3D
>>> 
>>> @Suresh, Could you please have a look ?
>>> 
>>> Thanks,
>>> Shameera.
>>> 
>>> On Sun, May 31, 2015 at 5:58 AM, Borries Demeler <
>>> demeler@biochem.uthscsa.edu> wrote:
>>> 
>>>> I think your attachments are stripped somewhere, you would be better
>> off
>>>> posting a URL.
>>>> 
>>>> Thanks, -b.
>>>> 
>>>>> Hi Gary,
>>>>> 
>>>>> I converted the image to pdf and attached here. in both mails i can
>> see
>>>>> the
>>>>> image, wonder how you all not getting that image. Please let me know
>> if
>>>>> you
>>>>> still can't see it.
>>>>> 
>>>>> Thanks,
>>>>> Shameera.
>>>>> 
>>>>> On Sat, May 30, 2015 at 9:43 PM, Gary E. Gorbet <ge...@gmail.com>
>>>>> wrote:
>>>>> 
>>>>>> 
>>>>>>> On May 30, 2015, at 5:32 PM, Shameera Rathnayaka
>>>>>> <sh...@gmail.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>> Hi Gary,
>>>>>>> 
>>>>>>> I inserted the diagram as image, let me attache it as attachment.
>>>>>> 
>>>>>> I still see nothing.
>>>>>> - Gary
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Shameera.
>>>>>>> 
>>>>>>> On Sat, May 30, 2015 at 5:35 PM, Gary E. Gorbet <
>> gegorbet@gmail.com
>>>> 
>>>>>> wrote:
>>>>>>> Shameera,
>>>>>>> 
>>>>>>> On my copy of this email there was no attachment, no following
>>>>>> diagram.
>>>>>> Would you please send that attachment or a URL that points to it.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Gary
>>>>>>> 
>>>>>>>> On May 30, 2015, at 3:55 PM, Shameera Rathnayaka
>>>>>> <sh...@apache.org>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>> Hi Devs,
>>>>>>>> 
>>>>>>>> As we are about to release Airavata 0.15( already cut the
>> branch )
>>>>>> we
>>>>>> will not add any major changes and it is in testing stage. This will
>>>>>> give
>>>>>> us time to discuss and finalize requirements for the next release ,
>> it
>>>>>> can
>>>>>> be either 0.16 or 1.0.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> As per the feedback from our user community, they need more
>>>>>> transparent view of what Airavata does when they submit an
>> experiment
>>> to
>>>>>> run a job on remote computer resource. Airavata users are science
>>>>>> gateway
>>>>>> developers, they are not only interested in Experiment level and
>>> remote
>>>>>> Job
>>>>>> level status changes. They would like to know some degree of
>>>>>> transparency
>>>>>> about pre-processing and post-processing tasks performed by airavata
>>>>>> framework, before and after Job submission. For example they would
>>> like
>>>>>> to
>>>>>> see which task is being executed at particular time, does scp file
>>>>>> transferring succeed or not. With current Hander architecture, it is
>>> not
>>>>>> possible to Airavata framework to know which handler does what. User
>>> can
>>>>>> write and integrate different kind of handlers and integrate it with
>>> the
>>>>>> execution chain. If Airavata Job submission failed while
>> transferring
>>>>>> input
>>>>>> file to the compute resource. Gateway developer should be able to
>> find
>>>>>> the
>>>>>> reason without any trouble. Current Airavata save the failure reason
>>>>>> with
>>>>>> stracktrace but that is too low level for a gateway developer.
>>>>>>>> 
>>>>>>>> Here we are thinking of replace this static handler architecture
>>>>>> with
>>>>>> dynamic task mechanism. Here framework has different type of tasks,
>>> lets
>>>>>> say for input staging we have SCP , GRIDFTP and HTTP tasks. each
>> task
>>>>>> clearly know what it need to do and how. When Airavata get an
>>> experiment
>>>>>> with three inputs, one is simple string and other two are SCP and
>> HTTP
>>>>>> type
>>>>>> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP
>>> tasks
>>>>>> to
>>>>>> the dynamic task chain. Then add another Job submission task, let's
>>> say
>>>>>> job
>>>>>> need to submit using ssh keys then Airavata add SSH job submission
>>> task.
>>>>>> as
>>>>>> same add required task for the outputs. Each task has three states
>>>>>> Processing, Completed, Failed. In case of failure, framework know
>>> which
>>>>>> type of works it was doing or which task failed, is it SCP file
>>> transfer
>>>>>> task or GRIDFTP file transfering task. Then Airavata can
>> provide(show)
>>>>>> this
>>>>>> details to Users by messaging. Please see following diagram to get
>> an
>>>>>> idea
>>>>>> about different level of state transitions.
>>>>>>>> 
>>>>>>>> Yours feedback are highly appreciate. ​
>>>>>>>> 
>>>>>>>> ​
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Shameera.
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Shameera Rathnayaka.
>>>>>>> 
>>>>>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>>>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Best Regards,
>>>>> Shameera Rathnayaka.
>>>>> 
>>>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> Borries Demeler, Ph.D.
>>>> Associate Professor
>>>> The University of Texas Health Science Center at San Antonio
>>>> Dept. of Biochemistry
>>>> Email: demeler@biochem.uthscsa.edu
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>> 
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>> 
>> 
>> 
>> 
>> --
>> Thanks,
>> Sachith Withana
>> 

Re: Airavata 0.16 Design Refactoring Suggestion?

["Gary E. Gorbet" <ge...@gmail.com>]

I would just like to chime in that I agree with Amila’s general point that having multiple logs capture everything and the Airavata APIs able to return a specified one of them (or set of them) sounds like the most useful model.

Gary

> On Jun 1, 2015, at 9:11 AM, Amila Jayasekara <th...@gmail.com> wrote:
> 
> Hi Shameera,
> 
> I am also having little difficulty relating the problem you are stating and
> the solution approach you are proposing. This is maybe because I am
> outdated with Airavata developments.
> 
> As per my understanding (from your description) "transparency" is the
> problem. One question about the problem is what level of transparency are
> you expecting ? Is it transparency depicted in log files or feedback to
> user front-ends etc. You may need to clarify this.
> 
> Also if it is the transparency in logs or user front-ends, then the
> solution you are proposing doesn't fit exactly to the problem (to my
> understanding).
> 
> On the other hand task based approach is a nice way to handle jobs. I guess
> you need to design a schedular to execute these tasks also. Some things you
> need to be concerned about when designing such a schedular are as follows;
> 
> 1. Some tasks have dependencies and ordering. Probably dependencies and
> ordering is defined by user (just like they way handlers are defined in
> gfac config files)
> 2. Would be nice to execute tasks in parallel whenever they dont have
> dependencies
> 3. Also be concerned about FT aspects when designing such a schedular.
> State of the schedular and also state of the individual tasks may need to
> be replicated over all nodes.
> 
> Thanks
> -Thejaka Amila
> 
> 
> 
> 
> On Mon, Jun 1, 2015 at 2:09 AM, Sachith Withana <sw...@gmail.com> wrote:
> 
>> Hi Shameera,
>> 
>> Wouldn't a job specific distributed logging mechanism solve the problem
>> that you mentioned? Allow the user/GWDeveloper to specify which
>> logs/messages
>> they want to see ( apart from that log everything anyways).
>> 
>> 
>> 
>> On Sun, May 31, 2015 at 8:57 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com
>>> wrote:
>> 
>>> Hi Gary & Borries,
>>> 
>>> It seems architecture mailing list doesn't show(drop) attachments, here
>> is
>>> the link
>>> https://creately.com/diagram/iaadwt6t1/yWLDNUPgfz60jl4k8WVH0Up3mk%3D
>>> 
>>> @Suresh, Could you please have a look ?
>>> 
>>> Thanks,
>>> Shameera.
>>> 
>>> On Sun, May 31, 2015 at 5:58 AM, Borries Demeler <
>>> demeler@biochem.uthscsa.edu> wrote:
>>> 
>>>> I think your attachments are stripped somewhere, you would be better
>> off
>>>> posting a URL.
>>>> 
>>>> Thanks, -b.
>>>> 
>>>>> Hi Gary,
>>>>> 
>>>>> I converted the image to pdf and attached here. in both mails i can
>> see
>>>>> the
>>>>> image, wonder how you all not getting that image. Please let me know
>> if
>>>>> you
>>>>> still can't see it.
>>>>> 
>>>>> Thanks,
>>>>> Shameera.
>>>>> 
>>>>> On Sat, May 30, 2015 at 9:43 PM, Gary E. Gorbet <ge...@gmail.com>
>>>>> wrote:
>>>>> 
>>>>>> 
>>>>>>> On May 30, 2015, at 5:32 PM, Shameera Rathnayaka
>>>>>> <sh...@gmail.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>> Hi Gary,
>>>>>>> 
>>>>>>> I inserted the diagram as image, let me attache it as attachment.
>>>>>> 
>>>>>> I still see nothing.
>>>>>> - Gary
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Shameera.
>>>>>>> 
>>>>>>> On Sat, May 30, 2015 at 5:35 PM, Gary E. Gorbet <
>> gegorbet@gmail.com
>>>> 
>>>>>> wrote:
>>>>>>> Shameera,
>>>>>>> 
>>>>>>> On my copy of this email there was no attachment, no following
>>>>>> diagram.
>>>>>> Would you please send that attachment or a URL that points to it.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Gary
>>>>>>> 
>>>>>>>> On May 30, 2015, at 3:55 PM, Shameera Rathnayaka
>>>>>> <sh...@apache.org>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>> Hi Devs,
>>>>>>>> 
>>>>>>>> As we are about to release Airavata 0.15( already cut the
>> branch )
>>>>>> we
>>>>>> will not add any major changes and it is in testing stage. This will
>>>>>> give
>>>>>> us time to discuss and finalize requirements for the next release ,
>> it
>>>>>> can
>>>>>> be either 0.16 or 1.0.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> As per the feedback from our user community, they need more
>>>>>> transparent view of what Airavata does when they submit an
>> experiment
>>> to
>>>>>> run a job on remote computer resource. Airavata users are science
>>>>>> gateway
>>>>>> developers, they are not only interested in Experiment level and
>>> remote
>>>>>> Job
>>>>>> level status changes. They would like to know some degree of
>>>>>> transparency
>>>>>> about pre-processing and post-processing tasks performed by airavata
>>>>>> framework, before and after Job submission. For example they would
>>> like
>>>>>> to
>>>>>> see which task is being executed at particular time, does scp file
>>>>>> transferring succeed or not. With current Hander architecture, it is
>>> not
>>>>>> possible to Airavata framework to know which handler does what. User
>>> can
>>>>>> write and integrate different kind of handlers and integrate it with
>>> the
>>>>>> execution chain. If Airavata Job submission failed while
>> transferring
>>>>>> input
>>>>>> file to the compute resource. Gateway developer should be able to
>> find
>>>>>> the
>>>>>> reason without any trouble. Current Airavata save the failure reason
>>>>>> with
>>>>>> stracktrace but that is too low level for a gateway developer.
>>>>>>>> 
>>>>>>>> Here we are thinking of replace this static handler architecture
>>>>>> with
>>>>>> dynamic task mechanism. Here framework has different type of tasks,
>>> lets
>>>>>> say for input staging we have SCP , GRIDFTP and HTTP tasks. each
>> task
>>>>>> clearly know what it need to do and how. When Airavata get an
>>> experiment
>>>>>> with three inputs, one is simple string and other two are SCP and
>> HTTP
>>>>>> type
>>>>>> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP
>>> tasks
>>>>>> to
>>>>>> the dynamic task chain. Then add another Job submission task, let's
>>> say
>>>>>> job
>>>>>> need to submit using ssh keys then Airavata add SSH job submission
>>> task.
>>>>>> as
>>>>>> same add required task for the outputs. Each task has three states
>>>>>> Processing, Completed, Failed. In case of failure, framework know
>>> which
>>>>>> type of works it was doing or which task failed, is it SCP file
>>> transfer
>>>>>> task or GRIDFTP file transfering task. Then Airavata can
>> provide(show)
>>>>>> this
>>>>>> details to Users by messaging. Please see following diagram to get
>> an
>>>>>> idea
>>>>>> about different level of state transitions.
>>>>>>>> 
>>>>>>>> Yours feedback are highly appreciate. ​
>>>>>>>> 
>>>>>>>> ​
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Shameera.
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>> Shameera Rathnayaka.
>>>>>>> 
>>>>>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>>>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Best Regards,
>>>>> Shameera Rathnayaka.
>>>>> 
>>>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> Borries Demeler, Ph.D.
>>>> Associate Professor
>>>> The University of Texas Health Science Center at San Antonio
>>>> Dept. of Biochemistry
>>>> Email: demeler@biochem.uthscsa.edu
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>> 
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>> 
>> 
>> 
>> 
>> --
>> Thanks,
>> Sachith Withana
>> 

Re: Airavata 0.16 Design Refactoring Suggestion?

[Shameera Rathnayaka <sh...@gmail.com>]

Hi Amila,




On Mon, Jun 1, 2015 at 10:11 AM, Amila Jayasekara <th...@gmail.com>
wrote:

> Hi Shameera,
>
> I am also having little difficulty relating the problem you are stating and
> the solution approach you are proposing. This is maybe because I am
> outdated with Airavata developments.
>
> As per my understanding (from your description) "transparency" is the
> problem. One question about the problem is what level of transparency are
> you expecting ? Is it transparency depicted in log files or feedback to
> user front-ends etc. You may need to clarify this.
>
> Also if it is the transparency in logs or user front-ends, then the
> solution you are proposing doesn't fit exactly to the problem (to my
> understanding).
>
>
With proposed solution I am going to fix following issues. .Let me explain
the requirements clearly and why current airavata implementation failed( or
not good enough) to support those.

1. User(Gateway Admin+ Developers) should able to see more details than
what we have, currently we have experiment status which say experiment is
executing. But sometimes user may need to know more deep, whether input
staging in progress or working on job submission etc ... In failure case,
experiment failed because input staging issue or job submission issue. Log
level data is too low level for a user. In fail case , log answer the
question why it failed? but user need to know what failed eg: "scp input
stage task failed". We need to handle this in framework level. Currently
what framework know is there are set of input handlers then there is a
provider and set of output handlers to execute. Framework doesn't know what
each handler does, it is hidden from framework. But framework has enough
details to know what it need to do for a given experiment and how. eg: for
a ssh input type, framework know it need to invoke a handler which ssh
given input to the compute resource like wise.

2. Current gfac configuration file allow us to define static execution
chains. Let's say we have a static chain with SCP input staging , SSH job
submission and then SCP output staging. But in run time user need to change
one of his experiment input to use another file transfer mechanism while
rest use SCP. And user may change this behavior time to time. Current
static approach con't support this dynamic behavior unless it define two
static chains and use correct one base on inputs. If we go this way we will
have many number of execution chain for a one application. So we need
dynamic approach.

With proposed solution, framework know what it does because framework
generate the task chain. Hence framework can provide more information to
user (via amqp message) what happen with each task. Users can get different
level of information by listening to different routing keys. Framework will
send Experiment , Process and Task level status changes via amqp messages.

On the other hand task based approach is a nice way to handle jobs. I guess
> you need to design a schedular to execute these tasks also. Some things you
> need to be concerned about when designing such a schedular are as follows;
>
> 1. Some tasks have dependencies and ordering. Probably dependencies and
> ordering is defined by user (just like they way handlers are defined in
> gfac config files)
> 2. Would be nice to execute tasks in parallel whenever they dont have
> dependencies
> 3. Also be concerned about FT aspects when designing such a schedular.
> State of the schedular and also state of the individual tasks may need to
> be replicated over all nodes.
>

​For FT we will use the current zookeeper base implementation + registry
data​. Yes we need to handle task dependency like make environment before
copy the input files. and input files can be perform parallel. Once we come
up with basic functionalities these are the improvement we need to do.

Thanks,
Shameera.



>
> Thanks
> -Thejaka Amila
>
>
>
>
> On Mon, Jun 1, 2015 at 2:09 AM, Sachith Withana <sw...@gmail.com>
> wrote:
>
> > Hi Shameera,
> >
> > Wouldn't a job specific distributed logging mechanism solve the problem
> > that you mentioned? Allow the user/GWDeveloper to specify which
> > logs/messages
> > they want to see ( apart from that log everything anyways).
> >
> >
> >
> > On Sun, May 31, 2015 at 8:57 PM, Shameera Rathnayaka <
> > shameerainfo@gmail.com
> > > wrote:
> >
> > > Hi Gary & Borries,
> > >
> > > It seems architecture mailing list doesn't show(drop) attachments, here
> > is
> > > the link
> > > https://creately.com/diagram/iaadwt6t1/yWLDNUPgfz60jl4k8WVH0Up3mk%3D
> > >
> > > @Suresh, Could you please have a look ?
> > >
> > > Thanks,
> > > Shameera.
> > >
> > > On Sun, May 31, 2015 at 5:58 AM, Borries Demeler <
> > > demeler@biochem.uthscsa.edu> wrote:
> > >
> > > > I think your attachments are stripped somewhere, you would be better
> > off
> > > > posting a URL.
> > > >
> > > > Thanks, -b.
> > > >
> > > > > Hi Gary,
> > > > >
> > > > > I converted the image to pdf and attached here. in both mails i can
> > see
> > > > > the
> > > > > image, wonder how you all not getting that image. Please let me
> know
> > if
> > > > > you
> > > > > still can't see it.
> > > > >
> > > > > Thanks,
> > > > > Shameera.
> > > > >
> > > > > On Sat, May 30, 2015 at 9:43 PM, Gary E. Gorbet <
> gegorbet@gmail.com>
> > > > > wrote:
> > > > >
> > > > >>
> > > > >> > On May 30, 2015, at 5:32 PM, Shameera Rathnayaka
> > > > >> <sh...@gmail.com>
> > > > >> wrote:
> > > > >> >
> > > > >> > Hi Gary,
> > > > >> >
> > > > >> > I inserted the diagram as image, let me attache it as
> attachment.
> > > > >>
> > > > >> I still see nothing.
> > > > >> - Gary
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Shameera.
> > > > >> >
> > > > >> > On Sat, May 30, 2015 at 5:35 PM, Gary E. Gorbet <
> > gegorbet@gmail.com
> > > >
> > > > >> wrote:
> > > > >> > Shameera,
> > > > >> >
> > > > >> > On my copy of this email there was no attachment, no following
> > > > >> diagram.
> > > > >> Would you please send that attachment or a URL that points to it.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Gary
> > > > >> >
> > > > >> > > On May 30, 2015, at 3:55 PM, Shameera Rathnayaka
> > > > >> <sh...@apache.org>
> > > > >> wrote:
> > > > >> > >
> > > > >> > > Hi Devs,
> > > > >> > >
> > > > >> > > As we are about to release Airavata 0.15( already cut the
> > branch )
> > > > >> we
> > > > >> will not add any major changes and it is in testing stage. This
> will
> > > > >> give
> > > > >> us time to discuss and finalize requirements for the next release
> ,
> > it
> > > > >> can
> > > > >> be either 0.16 or 1.0.
> > > > >> > >
> > > > >> > >
> > > > >> > > As per the feedback from our user community, they need more
> > > > >> transparent view of what Airavata does when they submit an
> > experiment
> > > to
> > > > >> run a job on remote computer resource. Airavata users are science
> > > > >> gateway
> > > > >> developers, they are not only interested in Experiment level and
> > > remote
> > > > >> Job
> > > > >> level status changes. They would like to know some degree of
> > > > >> transparency
> > > > >> about pre-processing and post-processing tasks performed by
> airavata
> > > > >> framework, before and after Job submission. For example they would
> > > like
> > > > >> to
> > > > >> see which task is being executed at particular time, does scp file
> > > > >> transferring succeed or not. With current Hander architecture, it
> is
> > > not
> > > > >> possible to Airavata framework to know which handler does what.
> User
> > > can
> > > > >> write and integrate different kind of handlers and integrate it
> with
> > > the
> > > > >> execution chain. If Airavata Job submission failed while
> > transferring
> > > > >> input
> > > > >> file to the compute resource. Gateway developer should be able to
> > find
> > > > >> the
> > > > >> reason without any trouble. Current Airavata save the failure
> reason
> > > > >> with
> > > > >> stracktrace but that is too low level for a gateway developer.
> > > > >> > >
> > > > >> > > Here we are thinking of replace this static handler
> architecture
> > > > >> with
> > > > >> dynamic task mechanism. Here framework has different type of
> tasks,
> > > lets
> > > > >> say for input staging we have SCP , GRIDFTP and HTTP tasks. each
> > task
> > > > >> clearly know what it need to do and how. When Airavata get an
> > > experiment
> > > > >> with three inputs, one is simple string and other two are SCP and
> > HTTP
> > > > >> type
> > > > >> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP
> > > tasks
> > > > >> to
> > > > >> the dynamic task chain. Then add another Job submission task,
> let's
> > > say
> > > > >> job
> > > > >> need to submit using ssh keys then Airavata add SSH job submission
> > > task.
> > > > >> as
> > > > >> same add required task for the outputs. Each task has three states
> > > > >> Processing, Completed, Failed. In case of failure, framework know
> > > which
> > > > >> type of works it was doing or which task failed, is it SCP file
> > > transfer
> > > > >> task or GRIDFTP file transfering task. Then Airavata can
> > provide(show)
> > > > >> this
> > > > >> details to Users by messaging. Please see following diagram to get
> > an
> > > > >> idea
> > > > >> about different level of state transitions.
> > > > >> > >
> > > > >> > > Yours feedback are highly appreciate. ​
> > > > >> > >
> > > > >> > > ​
> > > > >> > >
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Shameera.
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> > Best Regards,
> > > > >> > Shameera Rathnayaka.
> > > > >> >
> > > > >> > email: shameera AT apache.org , shameerainfo AT gmail.com
> > > > >> > Blog : http://shameerarathnayaka.blogspot.com/
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > Best Regards,
> > > > > Shameera Rathnayaka.
> > > > >
> > > > > email: shameera AT apache.org , shameerainfo AT gmail.com
> > > > > Blog : http://shameerarathnayaka.blogspot.com/
> > > > >
> > > >
> > > >
> > > > --
> > > > Borries Demeler, Ph.D.
> > > > Associate Professor
> > > > The University of Texas Health Science Center at San Antonio
> > > > Dept. of Biochemistry
> > > > Email: demeler@biochem.uthscsa.edu
> > > >
> > >
> > >
> > >
> > > --
> > > Best Regards,
> > > Shameera Rathnayaka.
> > >
> > > email: shameera AT apache.org , shameerainfo AT gmail.com
> > > Blog : http://shameerarathnayaka.blogspot.com/
> > >
> >
> >
> >
> > --
> > Thanks,
> > Sachith Withana
> >
>



-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Airavata 0.16 Design Refactoring Suggestion?

[Amila Jayasekara <th...@gmail.com>]

Hi Shameera,

I am also having little difficulty relating the problem you are stating and
the solution approach you are proposing. This is maybe because I am
outdated with Airavata developments.

As per my understanding (from your description) "transparency" is the
problem. One question about the problem is what level of transparency are
you expecting ? Is it transparency depicted in log files or feedback to
user front-ends etc. You may need to clarify this.

Also if it is the transparency in logs or user front-ends, then the
solution you are proposing doesn't fit exactly to the problem (to my
understanding).

On the other hand task based approach is a nice way to handle jobs. I guess
you need to design a schedular to execute these tasks also. Some things you
need to be concerned about when designing such a schedular are as follows;

1. Some tasks have dependencies and ordering. Probably dependencies and
ordering is defined by user (just like they way handlers are defined in
gfac config files)
2. Would be nice to execute tasks in parallel whenever they dont have
dependencies
3. Also be concerned about FT aspects when designing such a schedular.
State of the schedular and also state of the individual tasks may need to
be replicated over all nodes.

Thanks
-Thejaka Amila




On Mon, Jun 1, 2015 at 2:09 AM, Sachith Withana <sw...@gmail.com> wrote:

> Hi Shameera,
>
> Wouldn't a job specific distributed logging mechanism solve the problem
> that you mentioned? Allow the user/GWDeveloper to specify which
> logs/messages
> they want to see ( apart from that log everything anyways).
>
>
>
> On Sun, May 31, 2015 at 8:57 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com
> > wrote:
>
> > Hi Gary & Borries,
> >
> > It seems architecture mailing list doesn't show(drop) attachments, here
> is
> > the link
> > https://creately.com/diagram/iaadwt6t1/yWLDNUPgfz60jl4k8WVH0Up3mk%3D
> >
> > @Suresh, Could you please have a look ?
> >
> > Thanks,
> > Shameera.
> >
> > On Sun, May 31, 2015 at 5:58 AM, Borries Demeler <
> > demeler@biochem.uthscsa.edu> wrote:
> >
> > > I think your attachments are stripped somewhere, you would be better
> off
> > > posting a URL.
> > >
> > > Thanks, -b.
> > >
> > > > Hi Gary,
> > > >
> > > > I converted the image to pdf and attached here. in both mails i can
> see
> > > > the
> > > > image, wonder how you all not getting that image. Please let me know
> if
> > > > you
> > > > still can't see it.
> > > >
> > > > Thanks,
> > > > Shameera.
> > > >
> > > > On Sat, May 30, 2015 at 9:43 PM, Gary E. Gorbet <ge...@gmail.com>
> > > > wrote:
> > > >
> > > >>
> > > >> > On May 30, 2015, at 5:32 PM, Shameera Rathnayaka
> > > >> <sh...@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > Hi Gary,
> > > >> >
> > > >> > I inserted the diagram as image, let me attache it as attachment.
> > > >>
> > > >> I still see nothing.
> > > >> - Gary
> > > >> >
> > > >> > Thanks,
> > > >> > Shameera.
> > > >> >
> > > >> > On Sat, May 30, 2015 at 5:35 PM, Gary E. Gorbet <
> gegorbet@gmail.com
> > >
> > > >> wrote:
> > > >> > Shameera,
> > > >> >
> > > >> > On my copy of this email there was no attachment, no following
> > > >> diagram.
> > > >> Would you please send that attachment or a URL that points to it.
> > > >> >
> > > >> > Thanks,
> > > >> > Gary
> > > >> >
> > > >> > > On May 30, 2015, at 3:55 PM, Shameera Rathnayaka
> > > >> <sh...@apache.org>
> > > >> wrote:
> > > >> > >
> > > >> > > Hi Devs,
> > > >> > >
> > > >> > > As we are about to release Airavata 0.15( already cut the
> branch )
> > > >> we
> > > >> will not add any major changes and it is in testing stage. This will
> > > >> give
> > > >> us time to discuss and finalize requirements for the next release ,
> it
> > > >> can
> > > >> be either 0.16 or 1.0.
> > > >> > >
> > > >> > >
> > > >> > > As per the feedback from our user community, they need more
> > > >> transparent view of what Airavata does when they submit an
> experiment
> > to
> > > >> run a job on remote computer resource. Airavata users are science
> > > >> gateway
> > > >> developers, they are not only interested in Experiment level and
> > remote
> > > >> Job
> > > >> level status changes. They would like to know some degree of
> > > >> transparency
> > > >> about pre-processing and post-processing tasks performed by airavata
> > > >> framework, before and after Job submission. For example they would
> > like
> > > >> to
> > > >> see which task is being executed at particular time, does scp file
> > > >> transferring succeed or not. With current Hander architecture, it is
> > not
> > > >> possible to Airavata framework to know which handler does what. User
> > can
> > > >> write and integrate different kind of handlers and integrate it with
> > the
> > > >> execution chain. If Airavata Job submission failed while
> transferring
> > > >> input
> > > >> file to the compute resource. Gateway developer should be able to
> find
> > > >> the
> > > >> reason without any trouble. Current Airavata save the failure reason
> > > >> with
> > > >> stracktrace but that is too low level for a gateway developer.
> > > >> > >
> > > >> > > Here we are thinking of replace this static handler architecture
> > > >> with
> > > >> dynamic task mechanism. Here framework has different type of tasks,
> > lets
> > > >> say for input staging we have SCP , GRIDFTP and HTTP tasks. each
> task
> > > >> clearly know what it need to do and how. When Airavata get an
> > experiment
> > > >> with three inputs, one is simple string and other two are SCP and
> HTTP
> > > >> type
> > > >> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP
> > tasks
> > > >> to
> > > >> the dynamic task chain. Then add another Job submission task, let's
> > say
> > > >> job
> > > >> need to submit using ssh keys then Airavata add SSH job submission
> > task.
> > > >> as
> > > >> same add required task for the outputs. Each task has three states
> > > >> Processing, Completed, Failed. In case of failure, framework know
> > which
> > > >> type of works it was doing or which task failed, is it SCP file
> > transfer
> > > >> task or GRIDFTP file transfering task. Then Airavata can
> provide(show)
> > > >> this
> > > >> details to Users by messaging. Please see following diagram to get
> an
> > > >> idea
> > > >> about different level of state transitions.
> > > >> > >
> > > >> > > Yours feedback are highly appreciate. ​
> > > >> > >
> > > >> > > ​
> > > >> > >
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Shameera.
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Best Regards,
> > > >> > Shameera Rathnayaka.
> > > >> >
> > > >> > email: shameera AT apache.org , shameerainfo AT gmail.com
> > > >> > Blog : http://shameerarathnayaka.blogspot.com/
> > > >>
> > > >>
> > > >
> > > >
> > > > --
> > > > Best Regards,
> > > > Shameera Rathnayaka.
> > > >
> > > > email: shameera AT apache.org , shameerainfo AT gmail.com
> > > > Blog : http://shameerarathnayaka.blogspot.com/
> > > >
> > >
> > >
> > > --
> > > Borries Demeler, Ph.D.
> > > Associate Professor
> > > The University of Texas Health Science Center at San Antonio
> > > Dept. of Biochemistry
> > > Email: demeler@biochem.uthscsa.edu
> > >
> >
> >
> >
> > --
> > Best Regards,
> > Shameera Rathnayaka.
> >
> > email: shameera AT apache.org , shameerainfo AT gmail.com
> > Blog : http://shameerarathnayaka.blogspot.com/
> >
>
>
>
> --
> Thanks,
> Sachith Withana
>

Re: Airavata 0.16 Design Refactoring Suggestion?

[Sachith Withana <sw...@gmail.com>]

Hi Shameera,

Wouldn't a job specific distributed logging mechanism solve the problem
that you mentioned? Allow the user/GWDeveloper to specify which
logs/messages
they want to see ( apart from that log everything anyways).



On Sun, May 31, 2015 at 8:57 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:

> Hi Gary & Borries,
>
> It seems architecture mailing list doesn't show(drop) attachments, here is
> the link
> https://creately.com/diagram/iaadwt6t1/yWLDNUPgfz60jl4k8WVH0Up3mk%3D
>
> @Suresh, Could you please have a look ?
>
> Thanks,
> Shameera.
>
> On Sun, May 31, 2015 at 5:58 AM, Borries Demeler <
> demeler@biochem.uthscsa.edu> wrote:
>
> > I think your attachments are stripped somewhere, you would be better off
> > posting a URL.
> >
> > Thanks, -b.
> >
> > > Hi Gary,
> > >
> > > I converted the image to pdf and attached here. in both mails i can see
> > > the
> > > image, wonder how you all not getting that image. Please let me know if
> > > you
> > > still can't see it.
> > >
> > > Thanks,
> > > Shameera.
> > >
> > > On Sat, May 30, 2015 at 9:43 PM, Gary E. Gorbet <ge...@gmail.com>
> > > wrote:
> > >
> > >>
> > >> > On May 30, 2015, at 5:32 PM, Shameera Rathnayaka
> > >> <sh...@gmail.com>
> > >> wrote:
> > >> >
> > >> > Hi Gary,
> > >> >
> > >> > I inserted the diagram as image, let me attache it as attachment.
> > >>
> > >> I still see nothing.
> > >> - Gary
> > >> >
> > >> > Thanks,
> > >> > Shameera.
> > >> >
> > >> > On Sat, May 30, 2015 at 5:35 PM, Gary E. Gorbet <gegorbet@gmail.com
> >
> > >> wrote:
> > >> > Shameera,
> > >> >
> > >> > On my copy of this email there was no attachment, no following
> > >> diagram.
> > >> Would you please send that attachment or a URL that points to it.
> > >> >
> > >> > Thanks,
> > >> > Gary
> > >> >
> > >> > > On May 30, 2015, at 3:55 PM, Shameera Rathnayaka
> > >> <sh...@apache.org>
> > >> wrote:
> > >> > >
> > >> > > Hi Devs,
> > >> > >
> > >> > > As we are about to release Airavata 0.15( already cut the branch )
> > >> we
> > >> will not add any major changes and it is in testing stage. This will
> > >> give
> > >> us time to discuss and finalize requirements for the next release , it
> > >> can
> > >> be either 0.16 or 1.0.
> > >> > >
> > >> > >
> > >> > > As per the feedback from our user community, they need more
> > >> transparent view of what Airavata does when they submit an experiment
> to
> > >> run a job on remote computer resource. Airavata users are science
> > >> gateway
> > >> developers, they are not only interested in Experiment level and
> remote
> > >> Job
> > >> level status changes. They would like to know some degree of
> > >> transparency
> > >> about pre-processing and post-processing tasks performed by airavata
> > >> framework, before and after Job submission. For example they would
> like
> > >> to
> > >> see which task is being executed at particular time, does scp file
> > >> transferring succeed or not. With current Hander architecture, it is
> not
> > >> possible to Airavata framework to know which handler does what. User
> can
> > >> write and integrate different kind of handlers and integrate it with
> the
> > >> execution chain. If Airavata Job submission failed while transferring
> > >> input
> > >> file to the compute resource. Gateway developer should be able to find
> > >> the
> > >> reason without any trouble. Current Airavata save the failure reason
> > >> with
> > >> stracktrace but that is too low level for a gateway developer.
> > >> > >
> > >> > > Here we are thinking of replace this static handler architecture
> > >> with
> > >> dynamic task mechanism. Here framework has different type of tasks,
> lets
> > >> say for input staging we have SCP , GRIDFTP and HTTP tasks. each task
> > >> clearly know what it need to do and how. When Airavata get an
> experiment
> > >> with three inputs, one is simple string and other two are SCP and HTTP
> > >> type
> > >> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP
> tasks
> > >> to
> > >> the dynamic task chain. Then add another Job submission task, let's
> say
> > >> job
> > >> need to submit using ssh keys then Airavata add SSH job submission
> task.
> > >> as
> > >> same add required task for the outputs. Each task has three states
> > >> Processing, Completed, Failed. In case of failure, framework know
> which
> > >> type of works it was doing or which task failed, is it SCP file
> transfer
> > >> task or GRIDFTP file transfering task. Then Airavata can provide(show)
> > >> this
> > >> details to Users by messaging. Please see following diagram to get an
> > >> idea
> > >> about different level of state transitions.
> > >> > >
> > >> > > Yours feedback are highly appreciate. ​
> > >> > >
> > >> > > ​
> > >> > >
> > >> > >
> > >> > > Thanks,
> > >> > > Shameera.
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Best Regards,
> > >> > Shameera Rathnayaka.
> > >> >
> > >> > email: shameera AT apache.org , shameerainfo AT gmail.com
> > >> > Blog : http://shameerarathnayaka.blogspot.com/
> > >>
> > >>
> > >
> > >
> > > --
> > > Best Regards,
> > > Shameera Rathnayaka.
> > >
> > > email: shameera AT apache.org , shameerainfo AT gmail.com
> > > Blog : http://shameerarathnayaka.blogspot.com/
> > >
> >
> >
> > --
> > Borries Demeler, Ph.D.
> > Associate Professor
> > The University of Texas Health Science Center at San Antonio
> > Dept. of Biochemistry
> > Email: demeler@biochem.uthscsa.edu
> >
>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>



-- 
Thanks,
Sachith Withana

Re: Airavata 0.16 Design Refactoring Suggestion?

[Shameera Rathnayaka <sh...@gmail.com>]

Hi Gary & Borries,

It seems architecture mailing list doesn't show(drop) attachments, here is
the link
https://creately.com/diagram/iaadwt6t1/yWLDNUPgfz60jl4k8WVH0Up3mk%3D

@Suresh, Could you please have a look ?

Thanks,
Shameera.

On Sun, May 31, 2015 at 5:58 AM, Borries Demeler <
demeler@biochem.uthscsa.edu> wrote:

> I think your attachments are stripped somewhere, you would be better off
> posting a URL.
>
> Thanks, -b.
>
> > Hi Gary,
> >
> > I converted the image to pdf and attached here. in both mails i can see
> > the
> > image, wonder how you all not getting that image. Please let me know if
> > you
> > still can't see it.
> >
> > Thanks,
> > Shameera.
> >
> > On Sat, May 30, 2015 at 9:43 PM, Gary E. Gorbet <ge...@gmail.com>
> > wrote:
> >
> >>
> >> > On May 30, 2015, at 5:32 PM, Shameera Rathnayaka
> >> <sh...@gmail.com>
> >> wrote:
> >> >
> >> > Hi Gary,
> >> >
> >> > I inserted the diagram as image, let me attache it as attachment.
> >>
> >> I still see nothing.
> >> - Gary
> >> >
> >> > Thanks,
> >> > Shameera.
> >> >
> >> > On Sat, May 30, 2015 at 5:35 PM, Gary E. Gorbet <ge...@gmail.com>
> >> wrote:
> >> > Shameera,
> >> >
> >> > On my copy of this email there was no attachment, no following
> >> diagram.
> >> Would you please send that attachment or a URL that points to it.
> >> >
> >> > Thanks,
> >> > Gary
> >> >
> >> > > On May 30, 2015, at 3:55 PM, Shameera Rathnayaka
> >> <sh...@apache.org>
> >> wrote:
> >> > >
> >> > > Hi Devs,
> >> > >
> >> > > As we are about to release Airavata 0.15( already cut the branch )
> >> we
> >> will not add any major changes and it is in testing stage. This will
> >> give
> >> us time to discuss and finalize requirements for the next release , it
> >> can
> >> be either 0.16 or 1.0.
> >> > >
> >> > >
> >> > > As per the feedback from our user community, they need more
> >> transparent view of what Airavata does when they submit an experiment to
> >> run a job on remote computer resource. Airavata users are science
> >> gateway
> >> developers, they are not only interested in Experiment level and remote
> >> Job
> >> level status changes. They would like to know some degree of
> >> transparency
> >> about pre-processing and post-processing tasks performed by airavata
> >> framework, before and after Job submission. For example they would like
> >> to
> >> see which task is being executed at particular time, does scp file
> >> transferring succeed or not. With current Hander architecture, it is not
> >> possible to Airavata framework to know which handler does what. User can
> >> write and integrate different kind of handlers and integrate it with the
> >> execution chain. If Airavata Job submission failed while transferring
> >> input
> >> file to the compute resource. Gateway developer should be able to find
> >> the
> >> reason without any trouble. Current Airavata save the failure reason
> >> with
> >> stracktrace but that is too low level for a gateway developer.
> >> > >
> >> > > Here we are thinking of replace this static handler architecture
> >> with
> >> dynamic task mechanism. Here framework has different type of tasks, lets
> >> say for input staging we have SCP , GRIDFTP and HTTP tasks. each task
> >> clearly know what it need to do and how. When Airavata get an experiment
> >> with three inputs, one is simple string and other two are SCP and HTTP
> >> type
> >> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks
> >> to
> >> the dynamic task chain. Then add another Job submission task, let's say
> >> job
> >> need to submit using ssh keys then Airavata add SSH job submission task.
> >> as
> >> same add required task for the outputs. Each task has three states
> >> Processing, Completed, Failed. In case of failure, framework know which
> >> type of works it was doing or which task failed, is it SCP file transfer
> >> task or GRIDFTP file transfering task. Then Airavata can provide(show)
> >> this
> >> details to Users by messaging. Please see following diagram to get an
> >> idea
> >> about different level of state transitions.
> >> > >
> >> > > Yours feedback are highly appreciate. ​
> >> > >
> >> > > ​
> >> > >
> >> > >
> >> > > Thanks,
> >> > > Shameera.
> >> > >
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Best Regards,
> >> > Shameera Rathnayaka.
> >> >
> >> > email: shameera AT apache.org , shameerainfo AT gmail.com
> >> > Blog : http://shameerarathnayaka.blogspot.com/
> >>
> >>
> >
> >
> > --
> > Best Regards,
> > Shameera Rathnayaka.
> >
> > email: shameera AT apache.org , shameerainfo AT gmail.com
> > Blog : http://shameerarathnayaka.blogspot.com/
> >
>
>
> --
> Borries Demeler, Ph.D.
> Associate Professor
> The University of Texas Health Science Center at San Antonio
> Dept. of Biochemistry
> Email: demeler@biochem.uthscsa.edu
>



-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Airavata 0.16 Design Refactoring Suggestion?

[Borries Demeler <de...@biochem.uthscsa.edu>]

I think your attachments are stripped somewhere, you would be better off
posting a URL.

Thanks, -b.

> Hi Gary,
>
> I converted the image to pdf and attached here. in both mails i can see
> the
> image, wonder how you all not getting that image. Please let me know if
> you
> still can't see it.
>
> Thanks,
> Shameera.
>
> On Sat, May 30, 2015 at 9:43 PM, Gary E. Gorbet <ge...@gmail.com>
> wrote:
>
>>
>> > On May 30, 2015, at 5:32 PM, Shameera Rathnayaka
>> <sh...@gmail.com>
>> wrote:
>> >
>> > Hi Gary,
>> >
>> > I inserted the diagram as image, let me attache it as attachment.
>>
>> I still see nothing.
>> - Gary
>> >
>> > Thanks,
>> > Shameera.
>> >
>> > On Sat, May 30, 2015 at 5:35 PM, Gary E. Gorbet <ge...@gmail.com>
>> wrote:
>> > Shameera,
>> >
>> > On my copy of this email there was no attachment, no following
>> diagram.
>> Would you please send that attachment or a URL that points to it.
>> >
>> > Thanks,
>> > Gary
>> >
>> > > On May 30, 2015, at 3:55 PM, Shameera Rathnayaka
>> <sh...@apache.org>
>> wrote:
>> > >
>> > > Hi Devs,
>> > >
>> > > As we are about to release Airavata 0.15( already cut the branch )
>> we
>> will not add any major changes and it is in testing stage. This will
>> give
>> us time to discuss and finalize requirements for the next release , it
>> can
>> be either 0.16 or 1.0.
>> > >
>> > >
>> > > As per the feedback from our user community, they need more
>> transparent view of what Airavata does when they submit an experiment to
>> run a job on remote computer resource. Airavata users are science
>> gateway
>> developers, they are not only interested in Experiment level and remote
>> Job
>> level status changes. They would like to know some degree of
>> transparency
>> about pre-processing and post-processing tasks performed by airavata
>> framework, before and after Job submission. For example they would like
>> to
>> see which task is being executed at particular time, does scp file
>> transferring succeed or not. With current Hander architecture, it is not
>> possible to Airavata framework to know which handler does what. User can
>> write and integrate different kind of handlers and integrate it with the
>> execution chain. If Airavata Job submission failed while transferring
>> input
>> file to the compute resource. Gateway developer should be able to find
>> the
>> reason without any trouble. Current Airavata save the failure reason
>> with
>> stracktrace but that is too low level for a gateway developer.
>> > >
>> > > Here we are thinking of replace this static handler architecture
>> with
>> dynamic task mechanism. Here framework has different type of tasks, lets
>> say for input staging we have SCP , GRIDFTP and HTTP tasks. each task
>> clearly know what it need to do and how. When Airavata get an experiment
>> with three inputs, one is simple string and other two are SCP and HTTP
>> type
>> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks
>> to
>> the dynamic task chain. Then add another Job submission task, let's say
>> job
>> need to submit using ssh keys then Airavata add SSH job submission task.
>> as
>> same add required task for the outputs. Each task has three states
>> Processing, Completed, Failed. In case of failure, framework know which
>> type of works it was doing or which task failed, is it SCP file transfer
>> task or GRIDFTP file transfering task. Then Airavata can provide(show)
>> this
>> details to Users by messaging. Please see following diagram to get an
>> idea
>> about different level of state transitions.
>> > >
>> > > Yours feedback are highly appreciate. ​
>> > >
>> > > ​
>> > >
>> > >
>> > > Thanks,
>> > > Shameera.
>> > >
>> >
>> >
>> >
>> >
>> > --
>> > Best Regards,
>> > Shameera Rathnayaka.
>> >
>> > email: shameera AT apache.org , shameerainfo AT gmail.com
>> > Blog : http://shameerarathnayaka.blogspot.com/
>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>


-- 
Borries Demeler, Ph.D.
Associate Professor
The University of Texas Health Science Center at San Antonio
Dept. of Biochemistry
Email: demeler@biochem.uthscsa.edu

Re: Airavata 0.16 Design Refactoring Suggestion?

[Shameera Rathnayaka <sh...@gmail.com>]

Hi Gary,

I converted the image to pdf and attached here. in both mails i can see the
image, wonder how you all not getting that image. Please let me know if you
still can't see it.

Thanks,
Shameera.

On Sat, May 30, 2015 at 9:43 PM, Gary E. Gorbet <ge...@gmail.com> wrote:

>
> > On May 30, 2015, at 5:32 PM, Shameera Rathnayaka <sh...@gmail.com>
> wrote:
> >
> > Hi Gary,
> >
> > I inserted the diagram as image, let me attache it as attachment.
>
> I still see nothing.
> - Gary
> >
> > Thanks,
> > Shameera.
> >
> > On Sat, May 30, 2015 at 5:35 PM, Gary E. Gorbet <ge...@gmail.com>
> wrote:
> > Shameera,
> >
> > On my copy of this email there was no attachment, no following diagram.
> Would you please send that attachment or a URL that points to it.
> >
> > Thanks,
> > Gary
> >
> > > On May 30, 2015, at 3:55 PM, Shameera Rathnayaka <sh...@apache.org>
> wrote:
> > >
> > > Hi Devs,
> > >
> > > As we are about to release Airavata 0.15( already cut the branch ) we
> will not add any major changes and it is in testing stage. This will give
> us time to discuss and finalize requirements for the next release , it can
> be either 0.16 or 1.0.
> > >
> > >
> > > As per the feedback from our user community, they need more
> transparent view of what Airavata does when they submit an experiment to
> run a job on remote computer resource. Airavata users are science gateway
> developers, they are not only interested in Experiment level and remote Job
> level status changes. They would like to know some degree of transparency
> about pre-processing and post-processing tasks performed by airavata
> framework, before and after Job submission. For example they would like to
> see which task is being executed at particular time, does scp file
> transferring succeed or not. With current Hander architecture, it is not
> possible to Airavata framework to know which handler does what. User can
> write and integrate different kind of handlers and integrate it with the
> execution chain. If Airavata Job submission failed while transferring input
> file to the compute resource. Gateway developer should be able to find the
> reason without any trouble. Current Airavata save the failure reason with
> stracktrace but that is too low level for a gateway developer.
> > >
> > > Here we are thinking of replace this static handler architecture with
> dynamic task mechanism. Here framework has different type of tasks, lets
> say for input staging we have SCP , GRIDFTP and HTTP tasks. each task
> clearly know what it need to do and how. When Airavata get an experiment
> with three inputs, one is simple string and other two are SCP and HTTP type
> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to
> the dynamic task chain. Then add another Job submission task, let's say job
> need to submit using ssh keys then Airavata add SSH job submission task. as
> same add required task for the outputs. Each task has three states
> Processing, Completed, Failed. In case of failure, framework know which
> type of works it was doing or which task failed, is it SCP file transfer
> task or GRIDFTP file transfering task. Then Airavata can provide(show) this
> details to Users by messaging. Please see following diagram to get an idea
> about different level of state transitions.
> > >
> > > Yours feedback are highly appreciate. ​
> > >
> > > ​
> > >
> > >
> > > Thanks,
> > > Shameera.
> > >
> >
> >
> >
> >
> > --
> > Best Regards,
> > Shameera Rathnayaka.
> >
> > email: shameera AT apache.org , shameerainfo AT gmail.com
> > Blog : http://shameerarathnayaka.blogspot.com/
>
>


-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Airavata 0.16 Design Refactoring Suggestion?

["Gary E. Gorbet" <ge...@gmail.com>]

> On May 30, 2015, at 5:32 PM, Shameera Rathnayaka <sh...@gmail.com> wrote:
> 
> Hi Gary,
> 
> I inserted the diagram as image, let me attache it as attachment.

I still see nothing.
- Gary
> 
> Thanks,
> Shameera.
> 
> On Sat, May 30, 2015 at 5:35 PM, Gary E. Gorbet <ge...@gmail.com> wrote:
> Shameera,
> 
> On my copy of this email there was no attachment, no following diagram. Would you please send that attachment or a URL that points to it.
> 
> Thanks,
> Gary
> 
> > On May 30, 2015, at 3:55 PM, Shameera Rathnayaka <sh...@apache.org> wrote:
> >
> > Hi Devs,
> >
> > As we are about to release Airavata 0.15( already cut the branch ) we will not add any major changes and it is in testing stage. This will give us time to discuss and finalize requirements for the next release , it can be either 0.16 or 1.0.
> >
> >
> > As per the feedback from our user community, they need more transparent view of what Airavata does when they submit an experiment to run a job on remote computer resource. Airavata users are science gateway developers, they are not only interested in Experiment level and remote Job level status changes. They would like to know some degree of transparency about pre-processing and post-processing tasks performed by airavata framework, before and after Job submission. For example they would like to see which task is being executed at particular time, does scp file transferring succeed or not. With current Hander architecture, it is not possible to Airavata framework to know which handler does what. User can write and integrate different kind of handlers and integrate it with the execution chain. If Airavata Job submission failed while transferring input file to the compute resource. Gateway developer should be able to find the reason without any trouble. Current Airavata save the failure reason with stracktrace but that is too low level for a gateway developer.
> >
> > Here we are thinking of replace this static handler architecture with dynamic task mechanism. Here framework has different type of tasks, lets say for input staging we have SCP , GRIDFTP and HTTP tasks. each task clearly know what it need to do and how. When Airavata get an experiment with three inputs, one is simple string and other two are SCP and HTTP type file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to the dynamic task chain. Then add another Job submission task, let's say job need to submit using ssh keys then Airavata add SSH job submission task. as same add required task for the outputs. Each task has three states Processing, Completed, Failed. In case of failure, framework know which type of works it was doing or which task failed, is it SCP file transfer task or GRIDFTP file transfering task. Then Airavata can provide(show) this details to Users by messaging. Please see following diagram to get an idea about different level of state transitions.
> >
> > Yours feedback are highly appreciate. ​
> >
> > ​
> >
> >
> > Thanks,
> > Shameera.
> >
> 
> 
> 
> 
> -- 
> Best Regards,
> Shameera Rathnayaka.
> 
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/

Re: Airavata 0.16 Design Refactoring Suggestion?

[Borries Demeler <de...@biochem.uthscsa.edu>]

There is nothing attached to my e-mail. You can post a URL.

-b.
> Hi Gary,
>
> I inserted the diagram as image, let me attache it as attachment.
>
> Thanks,
> Shameera.
>
> On Sat, May 30, 2015 at 5:35 PM, Gary E. Gorbet <ge...@gmail.com>
> wrote:
>
>> Shameera,
>>
>> On my copy of this email there was no attachment, no following diagram.
>> Would you please send that attachment or a URL that points to it.
>>
>> Thanks,
>> Gary
>>
>> > On May 30, 2015, at 3:55 PM, Shameera Rathnayaka <sh...@apache.org>
>> wrote:
>> >
>> > Hi Devs,
>> >
>> > As we are about to release Airavata 0.15( already cut the branch ) we
>> will not add any major changes and it is in testing stage. This will
>> give
>> us time to discuss and finalize requirements for the next release , it
>> can
>> be either 0.16 or 1.0.
>> >
>> >
>> > As per the feedback from our user community, they need more
>> transparent
>> view of what Airavata does when they submit an experiment to run a job
>> on
>> remote computer resource. Airavata users are science gateway developers,
>> they are not only interested in Experiment level and remote Job level
>> status changes. They would like to know some degree of transparency
>> about
>> pre-processing and post-processing tasks performed by airavata
>> framework,
>> before and after Job submission. For example they would like to see
>> which
>> task is being executed at particular time, does scp file transferring
>> succeed or not. With current Hander architecture, it is not possible to
>> Airavata framework to know which handler does what. User can write and
>> integrate different kind of handlers and integrate it with the execution
>> chain. If Airavata Job submission failed while transferring input file
>> to
>> the compute resource. Gateway developer should be able to find the
>> reason
>> without any trouble. Current Airavata save the failure reason with
>> stracktrace but that is too low level for a gateway developer.
>> >
>> > Here we are thinking of replace this static handler architecture with
>> dynamic task mechanism. Here framework has different type of tasks, lets
>> say for input staging we have SCP , GRIDFTP and HTTP tasks. each task
>> clearly know what it need to do and how. When Airavata get an experiment
>> with three inputs, one is simple string and other two are SCP and HTTP
>> type
>> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks
>> to
>> the dynamic task chain. Then add another Job submission task, let's say
>> job
>> need to submit using ssh keys then Airavata add SSH job submission task.
>> as
>> same add required task for the outputs. Each task has three states
>> Processing, Completed, Failed. In case of failure, framework know which
>> type of works it was doing or which task failed, is it SCP file transfer
>> task or GRIDFTP file transfering task. Then Airavata can provide(show)
>> this
>> details to Users by messaging. Please see following diagram to get an
>> idea
>> about different level of state transitions.
>> >
>> > Yours feedback are highly appreciate. ​
>> >
>> > ​
>> >
>> >
>> > Thanks,
>> > Shameera.
>> >
>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>


-- 
Borries Demeler, Ph.D.
Associate Professor
The University of Texas Health Science Center at San Antonio
Dept. of Biochemistry
Email: demeler@biochem.uthscsa.edu

Re: Airavata 0.16 Design Refactoring Suggestion?

[Shameera Rathnayaka <sh...@gmail.com>]

Hi Gary,

I inserted the diagram as image, let me attache it as attachment.

Thanks,
Shameera.

On Sat, May 30, 2015 at 5:35 PM, Gary E. Gorbet <ge...@gmail.com> wrote:

> Shameera,
>
> On my copy of this email there was no attachment, no following diagram.
> Would you please send that attachment or a URL that points to it.
>
> Thanks,
> Gary
>
> > On May 30, 2015, at 3:55 PM, Shameera Rathnayaka <sh...@apache.org>
> wrote:
> >
> > Hi Devs,
> >
> > As we are about to release Airavata 0.15( already cut the branch ) we
> will not add any major changes and it is in testing stage. This will give
> us time to discuss and finalize requirements for the next release , it can
> be either 0.16 or 1.0.
> >
> >
> > As per the feedback from our user community, they need more transparent
> view of what Airavata does when they submit an experiment to run a job on
> remote computer resource. Airavata users are science gateway developers,
> they are not only interested in Experiment level and remote Job level
> status changes. They would like to know some degree of transparency about
> pre-processing and post-processing tasks performed by airavata framework,
> before and after Job submission. For example they would like to see which
> task is being executed at particular time, does scp file transferring
> succeed or not. With current Hander architecture, it is not possible to
> Airavata framework to know which handler does what. User can write and
> integrate different kind of handlers and integrate it with the execution
> chain. If Airavata Job submission failed while transferring input file to
> the compute resource. Gateway developer should be able to find the reason
> without any trouble. Current Airavata save the failure reason with
> stracktrace but that is too low level for a gateway developer.
> >
> > Here we are thinking of replace this static handler architecture with
> dynamic task mechanism. Here framework has different type of tasks, lets
> say for input staging we have SCP , GRIDFTP and HTTP tasks. each task
> clearly know what it need to do and how. When Airavata get an experiment
> with three inputs, one is simple string and other two are SCP and HTTP type
> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to
> the dynamic task chain. Then add another Job submission task, let's say job
> need to submit using ssh keys then Airavata add SSH job submission task. as
> same add required task for the outputs. Each task has three states
> Processing, Completed, Failed. In case of failure, framework know which
> type of works it was doing or which task failed, is it SCP file transfer
> task or GRIDFTP file transfering task. Then Airavata can provide(show) this
> details to Users by messaging. Please see following diagram to get an idea
> about different level of state transitions.
> >
> > Yours feedback are highly appreciate. ​
> >
> > ​
> >
> >
> > Thanks,
> > Shameera.
> >
>
>


-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Airavata 0.16 Design Refactoring Suggestion?

[Shameera Rathnayaka <sh...@gmail.com>]

Hi Gary,

I inserted the diagram as image, let me attache it as attachment.

Thanks,
Shameera.

On Sat, May 30, 2015 at 5:35 PM, Gary E. Gorbet <ge...@gmail.com> wrote:

> Shameera,
>
> On my copy of this email there was no attachment, no following diagram.
> Would you please send that attachment or a URL that points to it.
>
> Thanks,
> Gary
>
> > On May 30, 2015, at 3:55 PM, Shameera Rathnayaka <sh...@apache.org>
> wrote:
> >
> > Hi Devs,
> >
> > As we are about to release Airavata 0.15( already cut the branch ) we
> will not add any major changes and it is in testing stage. This will give
> us time to discuss and finalize requirements for the next release , it can
> be either 0.16 or 1.0.
> >
> >
> > As per the feedback from our user community, they need more transparent
> view of what Airavata does when they submit an experiment to run a job on
> remote computer resource. Airavata users are science gateway developers,
> they are not only interested in Experiment level and remote Job level
> status changes. They would like to know some degree of transparency about
> pre-processing and post-processing tasks performed by airavata framework,
> before and after Job submission. For example they would like to see which
> task is being executed at particular time, does scp file transferring
> succeed or not. With current Hander architecture, it is not possible to
> Airavata framework to know which handler does what. User can write and
> integrate different kind of handlers and integrate it with the execution
> chain. If Airavata Job submission failed while transferring input file to
> the compute resource. Gateway developer should be able to find the reason
> without any trouble. Current Airavata save the failure reason with
> stracktrace but that is too low level for a gateway developer.
> >
> > Here we are thinking of replace this static handler architecture with
> dynamic task mechanism. Here framework has different type of tasks, lets
> say for input staging we have SCP , GRIDFTP and HTTP tasks. each task
> clearly know what it need to do and how. When Airavata get an experiment
> with three inputs, one is simple string and other two are SCP and HTTP type
> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to
> the dynamic task chain. Then add another Job submission task, let's say job
> need to submit using ssh keys then Airavata add SSH job submission task. as
> same add required task for the outputs. Each task has three states
> Processing, Completed, Failed. In case of failure, framework know which
> type of works it was doing or which task failed, is it SCP file transfer
> task or GRIDFTP file transfering task. Then Airavata can provide(show) this
> details to Users by messaging. Please see following diagram to get an idea
> about different level of state transitions.
> >
> > Yours feedback are highly appreciate. ​
> >
> > ​
> >
> >
> > Thanks,
> > Shameera.
> >
>
>


-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Airavata 0.16 Design Refactoring Suggestion?

["Gary E. Gorbet" <ge...@gmail.com>]

Shameera,

On my copy of this email there was no attachment, no following diagram. Would you please send that attachment or a URL that points to it.

Thanks,
Gary

> On May 30, 2015, at 3:55 PM, Shameera Rathnayaka <sh...@apache.org> wrote:
> 
> Hi Devs, 
> 
> As we are about to release Airavata 0.15( already cut the branch ) we will not add any major changes and it is in testing stage. This will give us time to discuss and finalize requirements for the next release , it can be either 0.16 or 1.0. 
> 
> 
> As per the feedback from our user community, they need more transparent view of what Airavata does when they submit an experiment to run a job on remote computer resource. Airavata users are science gateway developers, they are not only interested in Experiment level and remote Job level status changes. They would like to know some degree of transparency about pre-processing and post-processing tasks performed by airavata framework, before and after Job submission. For example they would like to see which task is being executed at particular time, does scp file transferring succeed or not. With current Hander architecture, it is not possible to Airavata framework to know which handler does what. User can write and integrate different kind of handlers and integrate it with the execution chain. If Airavata Job submission failed while transferring input file to the compute resource. Gateway developer should be able to find the reason without any trouble. Current Airavata save the failure reason with stracktrace but that is too low level for a gateway developer. 
> 
> Here we are thinking of replace this static handler architecture with dynamic task mechanism. Here framework has different type of tasks, lets say for input staging we have SCP , GRIDFTP and HTTP tasks. each task clearly know what it need to do and how. When Airavata get an experiment with three inputs, one is simple string and other two are SCP and HTTP type file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to the dynamic task chain. Then add another Job submission task, let's say job need to submit using ssh keys then Airavata add SSH job submission task. as same add required task for the outputs. Each task has three states Processing, Completed, Failed. In case of failure, framework know which type of works it was doing or which task failed, is it SCP file transfer task or GRIDFTP file transfering task. Then Airavata can provide(show) this details to Users by messaging. Please see following diagram to get an idea about different level of state transitions.
> 
> Yours feedback are highly appreciate. ​
> 
> ​
> 
> 
> Thanks,
> Shameera.
> 

Re: Airavata 0.16 Design Refactoring Suggestion?

["Gary E. Gorbet" <ge...@gmail.com>]

Shameera,

On my copy of this email there was no attachment, no following diagram. Would you please send that attachment or a URL that points to it.

Thanks,
Gary

> On May 30, 2015, at 3:55 PM, Shameera Rathnayaka <sh...@apache.org> wrote:
> 
> Hi Devs, 
> 
> As we are about to release Airavata 0.15( already cut the branch ) we will not add any major changes and it is in testing stage. This will give us time to discuss and finalize requirements for the next release , it can be either 0.16 or 1.0. 
> 
> 
> As per the feedback from our user community, they need more transparent view of what Airavata does when they submit an experiment to run a job on remote computer resource. Airavata users are science gateway developers, they are not only interested in Experiment level and remote Job level status changes. They would like to know some degree of transparency about pre-processing and post-processing tasks performed by airavata framework, before and after Job submission. For example they would like to see which task is being executed at particular time, does scp file transferring succeed or not. With current Hander architecture, it is not possible to Airavata framework to know which handler does what. User can write and integrate different kind of handlers and integrate it with the execution chain. If Airavata Job submission failed while transferring input file to the compute resource. Gateway developer should be able to find the reason without any trouble. Current Airavata save the failure reason with stracktrace but that is too low level for a gateway developer. 
> 
> Here we are thinking of replace this static handler architecture with dynamic task mechanism. Here framework has different type of tasks, lets say for input staging we have SCP , GRIDFTP and HTTP tasks. each task clearly know what it need to do and how. When Airavata get an experiment with three inputs, one is simple string and other two are SCP and HTTP type file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to the dynamic task chain. Then add another Job submission task, let's say job need to submit using ssh keys then Airavata add SSH job submission task. as same add required task for the outputs. Each task has three states Processing, Completed, Failed. In case of failure, framework know which type of works it was doing or which task failed, is it SCP file transfer task or GRIDFTP file transfering task. Then Airavata can provide(show) this details to Users by messaging. Please see following diagram to get an idea about different level of state transitions.
> 
> Yours feedback are highly appreciate. ​
> 
> ​
> 
> 
> Thanks,
> Shameera.
> 

Re: Airavata 0.16 Design Refactoring Suggestion?

[Suresh Marru <sm...@apache.org>]

Thanks Shameera for clearly illustrating the refactoring goals. More below:

> On May 30, 2015, at 4:55 PM, Shameera Rathnayaka <sh...@apache.org> wrote:
> 
> Hi Devs, 
> 
> As we are about to release Airavata 0.15( already cut the branch ) we will not add any major changes and it is in testing stage. This will give us time to discuss and finalize requirements for the next release , it can be either 0.16 or 1.0. 

We probably should make a 0.16 and see where we stand. If we can really convince ourselves the API for specific set of advertised functionality can remain stable and the code quality and testing is production ready, we should move ahead to make it 1.0 (after we properly document the design and API).

> As per the feedback from our user community, they need more transparent view of what Airavata does when they submit an experiment to run a job on remote computer resource. Airavata users are science gateway developers, they are not only interested in Experiment level and remote Job level status changes. They would like to know some degree of transparency about pre-processing and post-processing tasks performed by airavata framework, before and after Job submission. For example they would like to see which task is being executed at particular time, does scp file transferring succeed or not. With current Hander architecture, it is not possible to Airavata framework to know which handler does what. User can write and integrate different kind of handlers and integrate it with the execution chain. If Airavata Job submission failed while transferring input file to the compute resource. Gateway developer should be able to find the reason without any trouble. Current Airavata save the failure reason with stracktrace but that is too low level for a gateway developer. 

This is good summary. Just to add to it, gateway administrator would care to have a transparent view of “what happened”. The stack trace approach shows “how it happened” which is not so interested and more over require knowledge of Airavata inner workings. If we can clearly illustrate the “whats” through data models then the system can be used more seamlessly. Ofcource there will be lot of “hows” in the airavata logs which can be analyzed. But we should focus on one having one state machine. Currently there are internal vs external state machines which are troubling. 

> Here we are thinking of replace this static handler architecture with dynamic task mechanism. Here framework has different type of tasks, lets say for input staging we have SCP , GRIDFTP and HTTP tasks. each task clearly know what it need to do and how. When Airavata get an experiment with three inputs, one is simple string and other two are SCP and HTTP type file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to the dynamic task chain. Then add another Job submission task, let's say job need to submit using ssh keys then Airavata add SSH job submission task. as same add required task for the outputs. Each task has three states Processing, Completed, Failed. In case of failure, framework know which type of works it was doing or which task failed, is it SCP file transfer task or GRIDFTP file transfering task. Then Airavata can provide(show) this details to Users by messaging. Please see following diagram to get an idea about different level of state transitions.

> Yours feedback are highly appreciate. ​

Would like to really listen to critical feedback on the proposed changes. I am strongly opinionated on Airavata providing a transparent view to end users and gateway administrators but have no biases on how it can be accomplished. Will appreciate every one’s time on guidance on the design changes Shameera is proposing. 

Suresh
 

Re: Airavata 0.16 Design Refactoring Suggestion?

[Shameera Rathnayaka <sh...@gmail.com>]

Hi Amila,

>
> By looking at the diagram I can ask lot of questions but I guess your
> complete design is not depicted in the picture.
>

​Yes sequence diagram is not descriptive enough to explain the whole
design. But this will help to get some idea about experiment execution
path.


> Few quick questions are as follows;
>
> 1. What is API server ? (sorry I dont know what this is) and what is the
> purpose of that ?
>

API server is already a part of existing design, which will allow different
clients to interact with Airavata components. All client requests go
through this.
​


> 2. Is experiment id generated by the PHP gateway (as per diagram that is
> the idea i got) ? (i.e. by client side not Airavata server) If so I would
> like to know how it works.
>

​This diagram only explain submit request path ( I already have mentioned
that ), experiment creation is not explain by this sequence diagram. Yes
experiment Id will be generated by server side.
​

> 3. When you first mention about tasks I was under the impression that GFac
> is going to spawn different tasks for a process. i.e. Scp task, Qsub task
> etc. but diagram doesnt exactly depict that behaviour.
>

​Dynamic task chain will be generated with process creation time at
orchestration level. Gfac will invoke those tasks. ​I agree, diagram
doesn't show those details.

4. Is monitoring also a task ?
>

​Nope, monitoring is not a task. It is a common service for all tasks which
will handle monitoring.


> 5. Seems some tasks need to share information among other components (or
> other tasks). E.g:- job id from job submission task. How are you planning
> to share those data ?
>

​There will be process context, which acts as container for that process.
each task will get this process context as a parameter.


> 6. Also diagram doesnt show how this design resolves your original
> question ("transparency"); i.e. sending feedback about each task to amqp
>

​As I have explained above, we are trying to solve two problems, and
transparency is one of them. Yes, amqp messages will be used to publish
status changes.


> 7. Also did you consider FT in the design ? It seems tasks have a pretty
> important role to acheive FT. E.g:- The job submission task
>

​Plan is to use zookeeper + work queue FT mechanism which we currently
have. But would like to talk about other alternatives too.


>
> I remember we discussing a design on FT sometime back and I believe
> Saminda came up with few nice diagrams that handle most of the failure
> cases. I suggest you looking at them if you are planning to consider FT in
> your design.
>

​Sure will have a look.


Thanks,
Shameera.
​


>
> Thanks
> -Thejaka
>
>
> On Tue, Jun 2, 2015 at 6:03 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi All,
>>
>> Here is how the sequence diagram looks like for experiment submit
>> request, with the proposed changes.
>>
>>
>>
>> ​
>>
>> Thanks,
>> Shameera.
>>
>> On Sat, May 30, 2015 at 4:55 PM, Shameera Rathnayaka <shameera@apache.org
>> > wrote:
>>
>>> Hi Devs,
>>>
>>> As we are about to release Airavata 0.15( already cut the branch ) we
>>> will not add any major changes and it is in testing stage. This will give
>>> us time to discuss and finalize requirements for the next release , it can
>>> be either 0.16 or 1.0.
>>>
>>>
>>> As per the feedback from our user community, they need more transparent
>>> view of what Airavata does when they submit an experiment to run a job on
>>> remote computer resource. Airavata users are science gateway developers,
>>> they are not only interested in Experiment level and remote Job level
>>> status changes. They would like to know some degree of transparency about
>>> pre-processing and post-processing tasks performed by airavata framework,
>>> before and after Job submission. For example they would like to see which
>>> task is being executed at particular time, does scp file transferring
>>> succeed or not. With current Hander architecture, it is not possible to
>>> Airavata framework to know which handler does what. User can write and
>>> integrate different kind of handlers and integrate it with the execution
>>> chain. If Airavata Job submission failed while transferring input file to
>>> the compute resource. Gateway developer should be able to find the reason
>>> without any trouble. Current Airavata save the failure reason with
>>> stracktrace but that is too low level for a gateway developer.
>>>
>>> Here we are thinking of replace this static handler architecture with
>>> dynamic task mechanism. Here framework has different type of tasks, lets
>>> say for input staging we have SCP , GRIDFTP and HTTP tasks. each task
>>> clearly know what it need to do and how. When Airavata get an experiment
>>> with three inputs, one is simple string and other two are SCP and HTTP type
>>> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to
>>> the dynamic task chain. Then add another Job submission task, let's say job
>>> need to submit using ssh keys then Airavata add SSH job submission task. as
>>> same add required task for the outputs. Each task has three states
>>> Processing, Completed, Failed. In case of failure, framework know which
>>> type of works it was doing or which task failed, is it SCP file transfer
>>> task or GRIDFTP file transfering task. Then Airavata can provide(show) this
>>> details to Users by messaging. Please see following diagram to get an idea
>>> about different level of state transitions.
>>>
>>> Yours feedback are highly appreciate. ​
>>>
>>> ​
>>>
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>


-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Airavata 0.16 Design Refactoring Suggestion?

[Amila Jayasekara <th...@gmail.com>]

Hi Shameera,

By looking at the diagram I can ask lot of questions but I guess your
complete design is not depicted in the picture.

Few quick questions are as follows;

1. What is API server ? (sorry I dont know what this is) and what is the
purpose of that ?
2. Is experiment id generated by the PHP gateway (as per diagram that is
the idea i got) ? (i.e. by client side not Airavata server) If so I would
like to know how it works.
3. When you first mention about tasks I was under the impression that GFac
is going to spawn different tasks for a process. i.e. Scp task, Qsub task
etc. but diagram doesnt exactly depict that behaviour.
4. Is monitoring also a task ?
5. Seems some tasks need to share information among other components (or
other tasks). E.g:- job id from job submission task. How are you planning
to share those data ?
6. Also diagram doesnt show how this design resolves your original question
("transparency"); i.e. sending feedback about each task to amqp
7. Also did you consider FT in the design ? It seems tasks have a pretty
important role to acheive FT. E.g:- The job submission task

I remember we discussing a design on FT sometime back and I believe Saminda
came up with few nice diagrams that handle most of the failure cases. I
suggest you looking at them if you are planning to consider FT in your
design.

Thanks
-Thejaka


On Tue, Jun 2, 2015 at 6:03 PM, Shameera Rathnayaka <sh...@gmail.com>
wrote:

> Hi All,
>
> Here is how the sequence diagram looks like for experiment submit request,
> with the proposed changes.
>
>
>
> ​
>
> Thanks,
> Shameera.
>
> On Sat, May 30, 2015 at 4:55 PM, Shameera Rathnayaka <sh...@apache.org>
> wrote:
>
>> Hi Devs,
>>
>> As we are about to release Airavata 0.15( already cut the branch ) we
>> will not add any major changes and it is in testing stage. This will give
>> us time to discuss and finalize requirements for the next release , it can
>> be either 0.16 or 1.0.
>>
>>
>> As per the feedback from our user community, they need more transparent
>> view of what Airavata does when they submit an experiment to run a job on
>> remote computer resource. Airavata users are science gateway developers,
>> they are not only interested in Experiment level and remote Job level
>> status changes. They would like to know some degree of transparency about
>> pre-processing and post-processing tasks performed by airavata framework,
>> before and after Job submission. For example they would like to see which
>> task is being executed at particular time, does scp file transferring
>> succeed or not. With current Hander architecture, it is not possible to
>> Airavata framework to know which handler does what. User can write and
>> integrate different kind of handlers and integrate it with the execution
>> chain. If Airavata Job submission failed while transferring input file to
>> the compute resource. Gateway developer should be able to find the reason
>> without any trouble. Current Airavata save the failure reason with
>> stracktrace but that is too low level for a gateway developer.
>>
>> Here we are thinking of replace this static handler architecture with
>> dynamic task mechanism. Here framework has different type of tasks, lets
>> say for input staging we have SCP , GRIDFTP and HTTP tasks. each task
>> clearly know what it need to do and how. When Airavata get an experiment
>> with three inputs, one is simple string and other two are SCP and HTTP type
>> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to
>> the dynamic task chain. Then add another Job submission task, let's say job
>> need to submit using ssh keys then Airavata add SSH job submission task. as
>> same add required task for the outputs. Each task has three states
>> Processing, Completed, Failed. In case of failure, framework know which
>> type of works it was doing or which task failed, is it SCP file transfer
>> task or GRIDFTP file transfering task. Then Airavata can provide(show) this
>> details to Users by messaging. Please see following diagram to get an idea
>> about different level of state transitions.
>>
>> Yours feedback are highly appreciate. ​
>>
>> ​
>>
>>
>> Thanks,
>> Shameera.
>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>

Re: Airavata 0.16 Design Refactoring Suggestion?

[Suresh Marru <sm...@apache.org>]

Hi Shameera,

This looks good. I have a cosmetic suggestion not related to the design. 

While we re-organize the code, we probably also want to rename the components so they are self-explanatory. 

The monitor in your diagram should be made more concrete like “HPC Job Monitor”. 

I think the name GFac has lived its life. This component by itself is no longer a Generic Application Factory. We should consider a new name which explains what it is doing. One though is Task Executor, other suggestions? 

Suresh

> On Jun 2, 2015, at 6:03 PM, Shameera Rathnayaka <sh...@gmail.com> wrote:
> 
> Hi All, 
> 
> Here is how the sequence diagram looks like for experiment submit request, with the proposed changes. 
> 
> 
> <Airavata 0.16 Design - Sequence Diagram.jpg>
> ​
> 
> Thanks,
> Shameera.
> 
> On Sat, May 30, 2015 at 4:55 PM, Shameera Rathnayaka <shameera@apache.org <ma...@apache.org>> wrote:
> Hi Devs, 
> 
> As we are about to release Airavata 0.15( already cut the branch ) we will not add any major changes and it is in testing stage. This will give us time to discuss and finalize requirements for the next release , it can be either 0.16 or 1.0. 
> 
> 
> As per the feedback from our user community, they need more transparent view of what Airavata does when they submit an experiment to run a job on remote computer resource. Airavata users are science gateway developers, they are not only interested in Experiment level and remote Job level status changes. They would like to know some degree of transparency about pre-processing and post-processing tasks performed by airavata framework, before and after Job submission. For example they would like to see which task is being executed at particular time, does scp file transferring succeed or not. With current Hander architecture, it is not possible to Airavata framework to know which handler does what. User can write and integrate different kind of handlers and integrate it with the execution chain. If Airavata Job submission failed while transferring input file to the compute resource. Gateway developer should be able to find the reason without any trouble. Current Airavata save the failure reason with stracktrace but that is too low level for a gateway developer. 
> 
> Here we are thinking of replace this static handler architecture with dynamic task mechanism. Here framework has different type of tasks, lets say for input staging we have SCP , GRIDFTP and HTTP tasks. each task clearly know what it need to do and how. When Airavata get an experiment with three inputs, one is simple string and other two are SCP and HTTP type file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to the dynamic task chain. Then add another Job submission task, let's say job need to submit using ssh keys then Airavata add SSH job submission task. as same add required task for the outputs. Each task has three states Processing, Completed, Failed. In case of failure, framework know which type of works it was doing or which task failed, is it SCP file transfer task or GRIDFTP file transfering task. Then Airavata can provide(show) this details to Users by messaging. Please see following diagram to get an idea about different level of state transitions.
> 
> Yours feedback are highly appreciate. ​
> <Airavata 0.16 Design.png>
> ​
> 
> 
> Thanks,
> Shameera.
> 
> 
> 
> 
> -- 
> Best Regards,
> Shameera Rathnayaka.
> 
> email: shameera AT apache.org <http://apache.org/> , shameerainfo AT gmail.com <http://gmail.com/>
> Blog : http://shameerarathnayaka.blogspot.com/ <http://shameerarathnayaka.blogspot.com/>

Re: Airavata 0.16 Design Refactoring Suggestion?

[Shameera Rathnayaka <sh...@gmail.com>]

Hi All,

Here is how the sequence diagram looks like for experiment submit request,
with the proposed changes.



​

Thanks,
Shameera.

On Sat, May 30, 2015 at 4:55 PM, Shameera Rathnayaka <sh...@apache.org>
wrote:

> Hi Devs,
>
> As we are about to release Airavata 0.15( already cut the branch ) we will
> not add any major changes and it is in testing stage. This will give us
> time to discuss and finalize requirements for the next release , it can be
> either 0.16 or 1.0.
>
>
> As per the feedback from our user community, they need more transparent
> view of what Airavata does when they submit an experiment to run a job on
> remote computer resource. Airavata users are science gateway developers,
> they are not only interested in Experiment level and remote Job level
> status changes. They would like to know some degree of transparency about
> pre-processing and post-processing tasks performed by airavata framework,
> before and after Job submission. For example they would like to see which
> task is being executed at particular time, does scp file transferring
> succeed or not. With current Hander architecture, it is not possible to
> Airavata framework to know which handler does what. User can write and
> integrate different kind of handlers and integrate it with the execution
> chain. If Airavata Job submission failed while transferring input file to
> the compute resource. Gateway developer should be able to find the reason
> without any trouble. Current Airavata save the failure reason with
> stracktrace but that is too low level for a gateway developer.
>
> Here we are thinking of replace this static handler architecture with
> dynamic task mechanism. Here framework has different type of tasks, lets
> say for input staging we have SCP , GRIDFTP and HTTP tasks. each task
> clearly know what it need to do and how. When Airavata get an experiment
> with three inputs, one is simple string and other two are SCP and HTTP type
> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to
> the dynamic task chain. Then add another Job submission task, let's say job
> need to submit using ssh keys then Airavata add SSH job submission task. as
> same add required task for the outputs. Each task has three states
> Processing, Completed, Failed. In case of failure, framework know which
> type of works it was doing or which task failed, is it SCP file transfer
> task or GRIDFTP file transfering task. Then Airavata can provide(show) this
> details to Users by messaging. Please see following diagram to get an idea
> about different level of state transitions.
>
> Yours feedback are highly appreciate. ​
>
> ​
>
>
> Thanks,
> Shameera.
>
>


-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Airavata 0.16 Design Refactoring Suggestion?

[Suresh Marru <sm...@apache.org>]

Thanks Shameera for clearly illustrating the refactoring goals. More below:

> On May 30, 2015, at 4:55 PM, Shameera Rathnayaka <sh...@apache.org> wrote:
> 
> Hi Devs, 
> 
> As we are about to release Airavata 0.15( already cut the branch ) we will not add any major changes and it is in testing stage. This will give us time to discuss and finalize requirements for the next release , it can be either 0.16 or 1.0. 

We probably should make a 0.16 and see where we stand. If we can really convince ourselves the API for specific set of advertised functionality can remain stable and the code quality and testing is production ready, we should move ahead to make it 1.0 (after we properly document the design and API).

> As per the feedback from our user community, they need more transparent view of what Airavata does when they submit an experiment to run a job on remote computer resource. Airavata users are science gateway developers, they are not only interested in Experiment level and remote Job level status changes. They would like to know some degree of transparency about pre-processing and post-processing tasks performed by airavata framework, before and after Job submission. For example they would like to see which task is being executed at particular time, does scp file transferring succeed or not. With current Hander architecture, it is not possible to Airavata framework to know which handler does what. User can write and integrate different kind of handlers and integrate it with the execution chain. If Airavata Job submission failed while transferring input file to the compute resource. Gateway developer should be able to find the reason without any trouble. Current Airavata save the failure reason with stracktrace but that is too low level for a gateway developer. 

This is good summary. Just to add to it, gateway administrator would care to have a transparent view of “what happened”. The stack trace approach shows “how it happened” which is not so interested and more over require knowledge of Airavata inner workings. If we can clearly illustrate the “whats” through data models then the system can be used more seamlessly. Ofcource there will be lot of “hows” in the airavata logs which can be analyzed. But we should focus on one having one state machine. Currently there are internal vs external state machines which are troubling. 

> Here we are thinking of replace this static handler architecture with dynamic task mechanism. Here framework has different type of tasks, lets say for input staging we have SCP , GRIDFTP and HTTP tasks. each task clearly know what it need to do and how. When Airavata get an experiment with three inputs, one is simple string and other two are SCP and HTTP type file transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to the dynamic task chain. Then add another Job submission task, let's say job need to submit using ssh keys then Airavata add SSH job submission task. as same add required task for the outputs. Each task has three states Processing, Completed, Failed. In case of failure, framework know which type of works it was doing or which task failed, is it SCP file transfer task or GRIDFTP file transfering task. Then Airavata can provide(show) this details to Users by messaging. Please see following diagram to get an idea about different level of state transitions.

> Yours feedback are highly appreciate. ​

Would like to really listen to critical feedback on the proposed changes. I am strongly opinionated on Airavata providing a transparent view to end users and gateway administrators but have no biases on how it can be accomplished. Will appreciate every one’s time on guidance on the design changes Shameera is proposing. 

Suresh