You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov> on 2014/09/18 19:15:22 UTC
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Hi Guys,
I've been interested in this too - we don't per have a specific
OODT workflow language, but we specific workflows using XML, and
other configuration (we are also thinking of moving to JSON for
this).
In the past I've also looked at YAWL and BPEL - both seem complex
to me.
I wonder at the end of the day if we should adopt something more
modern like PIG or some other data flow type of language (PIG
is really neat).
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Shameera Rathnayaka <sh...@gmail.com>
Reply-To: "architecture@airavata.apache.org"
<ar...@airavata.apache.org>
Date: Thursday, September 18, 2014 8:26 AM
To: "architecture@airavata.apache.org" <ar...@airavata.apache.org>,
dev <de...@airavata.apache.org>
Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>Hi All,
>
>As we all know Airavata has its own workflow language call XWF. When XWF
>was introduced, main focus points are interoperability and convertibility.
>But with years of experience it is convinced that above requirements are
>not really useful when we come to real world use cases. And XWF is XML
>based bulky language where we attache WSDLs and Workflow image it self.
>But
>with the recent changes WSDL part is being removed from XWF.
>
>It is worth to evaluate handy Scientific workflow languages in industry
>and
>find out pros and cons, at the end of this evaluation we need to come up
>with idea how we should improve Airavata workflow language, either we can
>improve existing XWF language, totally change to a new language available
>in industry or write a new light weight language. Basic requirements that
>we expect from new improvement are, high usability, flexible, light weight
>and real time monitoring support. As you can see above requirements are
>not
>direct comes with workflow languages but we need workflow language which
>help to support above requirements.
>
>After reading few papers and googling, initially i have come up with
>following three existing languages,
>1. YAWL <http://www.yawlfoundation.org/>
>2. WS-BPEL
>3. SIDL
><http://computation.llnl.gov/casc/components/index.html#page=home>
>
>In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
>also uses slightly modified version of SIDL. Other than above three
>languages we can come up with simple workflow language base on json(or
>yaml) which support all our requirements for some extends.
>
>It would be grate if I can get more input regarding the $Subject form the
>airavata community. You all are more than welcome to provide any type of
>suggestions.
>
>Thanks,
>Shameera.
>
>
>
>--
>Best Regards,
>Shameera Rathnayaka.
>
>email: shameera AT apache.org , shameerainfo AT gmail.com
>Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Great Bruce - these are amazing insights and I hope this
in the Airavata and OODT community check this out.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Bruce Barkstrom <br...@gmail.com>
Reply-To: "architecture@airavata.apache.org"
<ar...@airavata.apache.org>
Date: Friday, September 19, 2014 9:29 AM
To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Cc: "architecture@airavata.apache.org" <ar...@airavata.apache.org>,
dev <de...@airavata.apache.org>
Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
>One factor that should be included in the group's deliberations
>on adding a workflow language to the other things in OODT
>is the impact on long-term maintenance. While there's a lot
>of enthusiasm in the developer community right now, we need
>to think about what happens when development turns into
>maintenance. The account that follows is based on my experience
>with trying to resurrect a W3C-related project to visualize RDF
>graphs.
>
>The project is called IsaViz. It's even got a W3 web site:
>http://www.w3.org/2001/11/IsaViz/
>IsaViz identifies itself as a visual authoring tool for RDF.
>Right up near the top are two dates that should serve as
>a cautionary note for people who want to pick up this tool:
>Current Stable Version: October 2007 and Current Development Version:
>May 2007. It also looks like the site was maintained by a single
>developer who did the development as a postdoc project.
>The overview page was last modified on Oct. 21, 2007.
>The installation instructions were last modified on Oct. 18, 2004.
>
>IsaViz uses a number of tools. The Installation Instructions
>identify the following:
>
> - A Java Virtual Machine version 1.3.0 or later (1.4 strongly
> recommended - see Known problems
> <http://www.w3.org/2001/11/IsaViz/overview.html#bugs>)
> - A distribution of IsaViz, which contains the following Java JAR
>files:
> - IsaViz itself *(isaviz.jar)*
> - Zoomable Visual Transformation Machine *(zvtm.jar)*
> - Jena 2.1 for IsaViz 2.1, Jena 1.6.1 for IsaViz 1.2
> - Xerces-J version 2 *(xercesImpl.jar,xmlParserAPIs.jar)* for IsaViz
> 2, Xerces 1.4.4 for IsaViz 1.2
> - GraphViz from AT&T version 1.8.9 or later (version 1.7.x is no longer
> supported in IsaViz 2, and has actually only been tested with
>version 1.9). *Note:
> some instances of version 1.10.0 had a bug that produced incomplete SVG
> files, but it has been corrected in subsequent releases *(newer
>versions
> can be obtained on the graphviz.org site
> <http://www.graphviz.org/pub/graphviz/CURRENT>).
>
>So, what complications ensue:
>1. Java has moved way beyond version 1.3 or 1.4. Since Java can
>deprecate
>code and
>since there's Oracle and OpenJDK, there may be some unpleasantries that
>might need
>fixes. I haven't seen comments from the community on whether or not these
>might be
>significant. The IsaViz documentation refers to the ancient time when Sun
>controlled
>the language. Apparently, the IsaViz code was only tested with Sun's
>j2se/1.3 or 1.4.
>2. I suppose the jar files from IsaViz version 2 would be the place to
>start in reconstructing
>this piece of software. However, one might be careful about this when you
>get into the
>installation scripts from the Installation Instructions.
>3. The Zoomable Visual Transformation Machine project is on Sourceforge.
>It's apparently
>done in Java. However, the IsaViz code used version 0.9.0, while the
>current Sourceforge
>project (at http://zvtm.sourceforge.net/) is now up to 0.11.1 for the
>stable version (Aug. 2013)
>with a more recent development version (0.11.2 - snapshot; June 2014). No
>idea if
>there would be any serious ramifications from this change.
>4. The Installation Instructions have a link to the HP Jena site. If you
>link to it, the
>page says "Oops! ..." Jena was moved from HP to apache. So if you want
>to
>do
>Jena, you now need to consult <https://jena.apache.org/>. I'm not sure
>exactly how
>the apache Jena source code or binary installations compare with what
>IsaViz is expecting.
>As a note, Jena is a BIG chunk of software. I think the tutorials on RDF
>(including OWL
>and related reasoners) are going to take a novice user (including many
>scientists) a month
>or two of dedicated time to work through. I don't know how easy IsaViz
>would be to install
>without at least a basic understanding of RDF and of the related triple
>store database.
>5. Xerces-J is the XML Java parser (see <Xerces.apache.org>), which is
>now
>up to
>version 2.11.0. Again, it isn't clear what kinds of difficulties one
>would
>encounter to use
>this library.
>6. GraphViz (at <http://www.graphviz.org/>) is now at version 2.38.0-1.
>AT&T seems
>to be maintaining a lot of installation options. I was interested in
>Ubuntu - and then
>there are different versions of that.
>
>As an additional note, Linux has developed a bunch of variants. A
>particularly active
>area of development is the creation of automated package managers - often
>with centralized
>control over installation procedures and source code libraries. The
>packages have dependencies
>on the libraries -- and there's no guarantee that an RPM package has the
>same dependencies
>as a Debian package. This is a bit like the DOI guarantee of providing a
>unique location to
>obtain original items - although publishers have been known to substitute
>new versions of
>the unique object for the "true" original.
>
>At the same time, software packages with complex networks of dependencies
>are
>not exactly easy to maintain with Linux (or Unix) scripts. Exploring the
>integrity of
>the whole package requires a fair amount of work by experienced system
>administrators.
>
>If the intent is to produce data archives (or data production facilities)
>that have long-term
>maintainability, they need to handle replication [see Barkstrom and
>Mattman, 2010, ESI]
>of objects, as well as transparency. The key attributes of such systems
>need to be
>simplicity, provenance integrity, and reliability. They aren't easy
>attributes to maintain
>over the long haul. The article on "being digital" in the current CACM
>has
>a useful
>perspective on how our enthusiasm for "rupture talk" plays out:
>Haigh, T., 2014: We Have Never Been Digital, CACM, 57,No. 09,24-28
>Peter Denning's article that follows immediately in the print version
>[Denning, P. J., 2014: The Profession of IT: Learning for the New Digital
>Age,
>CACM, 57, No. 09, 29-31] offers some additional perspective that's
>probably
>relevant to the issue of the learning curve for new technologies. That
>curve
>is usually underestimated. While everyone wants "user friendly" tools, it
>isn't
>easy for developers to get an accurate idea for how many person-hours of
>work
>it will require to make a user proficient enough to use new tools,
>particularly in the
>presence of "version churn" like we can see in the IsaViz example.
>
>Bruce B.
>
>
>On Thu, Sep 18, 2014 at 2:54 PM, Lavanya Ramakrishnan
><lramakrishnan@lbl.gov
>> wrote:
>
>> Here is my 2c -
>>
>> I think it is important to try and understand what your users are going
>>to
>> do with workflow and what kind of language they are used to
>> (domain-specific, functional, etc). They are processes called
>>user-centered
>> design processes you can use to do this or do at a minimum an informal
>> study.
>>
>> A couple of years ago, we did an introspection on why all the existing
>> workflow tools didn't have the uptake we had assumed it would. I have
>>been
>> part of a half dozen different tools over my career. We have since
>>launched
>> a project called Tigres - http://tigres.lbl.gov/ where we have learned a
>> lot due to using a user-centered design approach. We have an IEEE
>>eScience
>> paper on our initial work - which you might find interesting. I am also
>> happy to share more details on Tigres and/or the process.
>>
>> Lavanya
>>
>>
>>
>>
>>
>> On Thu, Sep 18, 2014 at 10:53 AM, BW <bw...@mysoftcloud.com> wrote:
>>
>> > Is there a list of graphical BEL workflow tools?
>> >
>> > On Thursday, September 18, 2014, Mattmann, Chris A (3980) <
>> > chris.a.mattmann@jpl.nasa.gov> wrote:
>> >
>> > > Hi Guys,
>> > >
>> > > I've been interested in this too - we don't per have a specific
>> > > OODT workflow language, but we specific workflows using XML, and
>> > > other configuration (we are also thinking of moving to JSON for
>> > > this).
>> > >
>> > > In the past I've also looked at YAWL and BPEL - both seem complex
>> > > to me.
>> > >
>> > > I wonder at the end of the day if we should adopt something more
>> > > modern like PIG or some other data flow type of language (PIG
>> > > is really neat).
>> > >
>> > > Cheers,
>> > > Chris
>> > >
>> > >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > Chris Mattmann, Ph.D.
>> > > Chief Architect
>> > > Instrument Software and Science Data Systems Section (398)
>> > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > > Office: 168-519, Mailstop: 168-527
>> > > Email: chris.a.mattmann@nasa.gov <javascript:;>
>> > > WWW: http://sunset.usc.edu/~mattmann/
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > Adjunct Associate Professor, Computer Science Department
>> > > University of Southern California, Los Angeles, CA 90089 USA
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: Shameera Rathnayaka <shameerainfo@gmail.com <javascript:;>>
>> > > Reply-To: "architecture@airavata.apache.org <javascript:;>"
>> > > <architecture@airavata.apache.org <javascript:;>>
>> > > Date: Thursday, September 18, 2014 8:26 AM
>> > > To: "architecture@airavata.apache.org <javascript:;>" <
>> > > architecture@airavata.apache.org <javascript:;>>,
>> > > dev <dev@airavata.apache.org <javascript:;>>
>> > > Subject: Evaluate Suitable Scientific Workflow Language for
>>Airavata.
>> > >
>> > > >Hi All,
>> > > >
>> > > >As we all know Airavata has its own workflow language call XWF.
>>When
>> XWF
>> > > >was introduced, main focus points are interoperability and
>> > convertibility.
>> > > >But with years of experience it is convinced that above
>>requirements
>> are
>> > > >not really useful when we come to real world use cases. And XWF is
>>XML
>> > > >based bulky language where we attache WSDLs and Workflow image it
>> self.
>> > > >But
>> > > >with the recent changes WSDL part is being removed from XWF.
>> > > >
>> > > >It is worth to evaluate handy Scientific workflow languages in
>> industry
>> > > >and
>> > > >find out pros and cons, at the end of this evaluation we need to
>>come
>> up
>> > > >with idea how we should improve Airavata workflow language, either
>>we
>> > can
>> > > >improve existing XWF language, totally change to a new language
>> > available
>> > > >in industry or write a new light weight language. Basic
>>requirements
>> > that
>> > > >we expect from new improvement are, high usability, flexible, light
>> > weight
>> > > >and real time monitoring support. As you can see above requirements
>> are
>> > > >not
>> > > >direct comes with workflow languages but we need workflow language
>> which
>> > > >help to support above requirements.
>> > > >
>> > > >After reading few papers and googling, initially i have come up
>>with
>> > > >following three existing languages,
>> > > >1. YAWL <http://www.yawlfoundation.org/>
>> > > >2. WS-BPEL
>> > > >3. SIDL
>> > > ><http://computation.llnl.gov/casc/components/index.html#page=home>
>> > > >
>> > > >In my opinion SIDL is more familiar with scientific domain,
>> Radical-SAGA
>> > > >also uses slightly modified version of SIDL. Other than above three
>> > > >languages we can come up with simple workflow language base on
>>json(or
>> > > >yaml) which support all our requirements for some extends.
>> > > >
>> > > >It would be grate if I can get more input regarding the $Subject
>>form
>> > the
>> > > >airavata community. You all are more than welcome to provide any
>>type
>> of
>> > > >suggestions.
>> > > >
>> > > >Thanks,
>> > > >Shameera.
>> > > >
>> > > >
>> > > >
>> > > >--
>> > > >Best Regards,
>> > > >Shameera Rathnayaka.
>> > > >
>> > > >email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > >Blog : http://shameerarathnayaka.blogspot.com/
>> > >
>> > >
>> >
>>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Great Bruce - these are amazing insights and I hope this
in the Airavata and OODT community check this out.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Bruce Barkstrom <br...@gmail.com>
Reply-To: "architecture@airavata.apache.org"
<ar...@airavata.apache.org>
Date: Friday, September 19, 2014 9:29 AM
To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Cc: "architecture@airavata.apache.org" <ar...@airavata.apache.org>,
dev <de...@airavata.apache.org>
Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
>One factor that should be included in the group's deliberations
>on adding a workflow language to the other things in OODT
>is the impact on long-term maintenance. While there's a lot
>of enthusiasm in the developer community right now, we need
>to think about what happens when development turns into
>maintenance. The account that follows is based on my experience
>with trying to resurrect a W3C-related project to visualize RDF
>graphs.
>
>The project is called IsaViz. It's even got a W3 web site:
>http://www.w3.org/2001/11/IsaViz/
>IsaViz identifies itself as a visual authoring tool for RDF.
>Right up near the top are two dates that should serve as
>a cautionary note for people who want to pick up this tool:
>Current Stable Version: October 2007 and Current Development Version:
>May 2007. It also looks like the site was maintained by a single
>developer who did the development as a postdoc project.
>The overview page was last modified on Oct. 21, 2007.
>The installation instructions were last modified on Oct. 18, 2004.
>
>IsaViz uses a number of tools. The Installation Instructions
>identify the following:
>
> - A Java Virtual Machine version 1.3.0 or later (1.4 strongly
> recommended - see Known problems
> <http://www.w3.org/2001/11/IsaViz/overview.html#bugs>)
> - A distribution of IsaViz, which contains the following Java JAR
>files:
> - IsaViz itself *(isaviz.jar)*
> - Zoomable Visual Transformation Machine *(zvtm.jar)*
> - Jena 2.1 for IsaViz 2.1, Jena 1.6.1 for IsaViz 1.2
> - Xerces-J version 2 *(xercesImpl.jar,xmlParserAPIs.jar)* for IsaViz
> 2, Xerces 1.4.4 for IsaViz 1.2
> - GraphViz from AT&T version 1.8.9 or later (version 1.7.x is no longer
> supported in IsaViz 2, and has actually only been tested with
>version 1.9). *Note:
> some instances of version 1.10.0 had a bug that produced incomplete SVG
> files, but it has been corrected in subsequent releases *(newer
>versions
> can be obtained on the graphviz.org site
> <http://www.graphviz.org/pub/graphviz/CURRENT>).
>
>So, what complications ensue:
>1. Java has moved way beyond version 1.3 or 1.4. Since Java can
>deprecate
>code and
>since there's Oracle and OpenJDK, there may be some unpleasantries that
>might need
>fixes. I haven't seen comments from the community on whether or not these
>might be
>significant. The IsaViz documentation refers to the ancient time when Sun
>controlled
>the language. Apparently, the IsaViz code was only tested with Sun's
>j2se/1.3 or 1.4.
>2. I suppose the jar files from IsaViz version 2 would be the place to
>start in reconstructing
>this piece of software. However, one might be careful about this when you
>get into the
>installation scripts from the Installation Instructions.
>3. The Zoomable Visual Transformation Machine project is on Sourceforge.
>It's apparently
>done in Java. However, the IsaViz code used version 0.9.0, while the
>current Sourceforge
>project (at http://zvtm.sourceforge.net/) is now up to 0.11.1 for the
>stable version (Aug. 2013)
>with a more recent development version (0.11.2 - snapshot; June 2014). No
>idea if
>there would be any serious ramifications from this change.
>4. The Installation Instructions have a link to the HP Jena site. If you
>link to it, the
>page says "Oops! ..." Jena was moved from HP to apache. So if you want
>to
>do
>Jena, you now need to consult <https://jena.apache.org/>. I'm not sure
>exactly how
>the apache Jena source code or binary installations compare with what
>IsaViz is expecting.
>As a note, Jena is a BIG chunk of software. I think the tutorials on RDF
>(including OWL
>and related reasoners) are going to take a novice user (including many
>scientists) a month
>or two of dedicated time to work through. I don't know how easy IsaViz
>would be to install
>without at least a basic understanding of RDF and of the related triple
>store database.
>5. Xerces-J is the XML Java parser (see <Xerces.apache.org>), which is
>now
>up to
>version 2.11.0. Again, it isn't clear what kinds of difficulties one
>would
>encounter to use
>this library.
>6. GraphViz (at <http://www.graphviz.org/>) is now at version 2.38.0-1.
>AT&T seems
>to be maintaining a lot of installation options. I was interested in
>Ubuntu - and then
>there are different versions of that.
>
>As an additional note, Linux has developed a bunch of variants. A
>particularly active
>area of development is the creation of automated package managers - often
>with centralized
>control over installation procedures and source code libraries. The
>packages have dependencies
>on the libraries -- and there's no guarantee that an RPM package has the
>same dependencies
>as a Debian package. This is a bit like the DOI guarantee of providing a
>unique location to
>obtain original items - although publishers have been known to substitute
>new versions of
>the unique object for the "true" original.
>
>At the same time, software packages with complex networks of dependencies
>are
>not exactly easy to maintain with Linux (or Unix) scripts. Exploring the
>integrity of
>the whole package requires a fair amount of work by experienced system
>administrators.
>
>If the intent is to produce data archives (or data production facilities)
>that have long-term
>maintainability, they need to handle replication [see Barkstrom and
>Mattman, 2010, ESI]
>of objects, as well as transparency. The key attributes of such systems
>need to be
>simplicity, provenance integrity, and reliability. They aren't easy
>attributes to maintain
>over the long haul. The article on "being digital" in the current CACM
>has
>a useful
>perspective on how our enthusiasm for "rupture talk" plays out:
>Haigh, T., 2014: We Have Never Been Digital, CACM, 57,No. 09,24-28
>Peter Denning's article that follows immediately in the print version
>[Denning, P. J., 2014: The Profession of IT: Learning for the New Digital
>Age,
>CACM, 57, No. 09, 29-31] offers some additional perspective that's
>probably
>relevant to the issue of the learning curve for new technologies. That
>curve
>is usually underestimated. While everyone wants "user friendly" tools, it
>isn't
>easy for developers to get an accurate idea for how many person-hours of
>work
>it will require to make a user proficient enough to use new tools,
>particularly in the
>presence of "version churn" like we can see in the IsaViz example.
>
>Bruce B.
>
>
>On Thu, Sep 18, 2014 at 2:54 PM, Lavanya Ramakrishnan
><lramakrishnan@lbl.gov
>> wrote:
>
>> Here is my 2c -
>>
>> I think it is important to try and understand what your users are going
>>to
>> do with workflow and what kind of language they are used to
>> (domain-specific, functional, etc). They are processes called
>>user-centered
>> design processes you can use to do this or do at a minimum an informal
>> study.
>>
>> A couple of years ago, we did an introspection on why all the existing
>> workflow tools didn't have the uptake we had assumed it would. I have
>>been
>> part of a half dozen different tools over my career. We have since
>>launched
>> a project called Tigres - http://tigres.lbl.gov/ where we have learned a
>> lot due to using a user-centered design approach. We have an IEEE
>>eScience
>> paper on our initial work - which you might find interesting. I am also
>> happy to share more details on Tigres and/or the process.
>>
>> Lavanya
>>
>>
>>
>>
>>
>> On Thu, Sep 18, 2014 at 10:53 AM, BW <bw...@mysoftcloud.com> wrote:
>>
>> > Is there a list of graphical BEL workflow tools?
>> >
>> > On Thursday, September 18, 2014, Mattmann, Chris A (3980) <
>> > chris.a.mattmann@jpl.nasa.gov> wrote:
>> >
>> > > Hi Guys,
>> > >
>> > > I've been interested in this too - we don't per have a specific
>> > > OODT workflow language, but we specific workflows using XML, and
>> > > other configuration (we are also thinking of moving to JSON for
>> > > this).
>> > >
>> > > In the past I've also looked at YAWL and BPEL - both seem complex
>> > > to me.
>> > >
>> > > I wonder at the end of the day if we should adopt something more
>> > > modern like PIG or some other data flow type of language (PIG
>> > > is really neat).
>> > >
>> > > Cheers,
>> > > Chris
>> > >
>> > >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > Chris Mattmann, Ph.D.
>> > > Chief Architect
>> > > Instrument Software and Science Data Systems Section (398)
>> > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > > Office: 168-519, Mailstop: 168-527
>> > > Email: chris.a.mattmann@nasa.gov <javascript:;>
>> > > WWW: http://sunset.usc.edu/~mattmann/
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > Adjunct Associate Professor, Computer Science Department
>> > > University of Southern California, Los Angeles, CA 90089 USA
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: Shameera Rathnayaka <shameerainfo@gmail.com <javascript:;>>
>> > > Reply-To: "architecture@airavata.apache.org <javascript:;>"
>> > > <architecture@airavata.apache.org <javascript:;>>
>> > > Date: Thursday, September 18, 2014 8:26 AM
>> > > To: "architecture@airavata.apache.org <javascript:;>" <
>> > > architecture@airavata.apache.org <javascript:;>>,
>> > > dev <dev@airavata.apache.org <javascript:;>>
>> > > Subject: Evaluate Suitable Scientific Workflow Language for
>>Airavata.
>> > >
>> > > >Hi All,
>> > > >
>> > > >As we all know Airavata has its own workflow language call XWF.
>>When
>> XWF
>> > > >was introduced, main focus points are interoperability and
>> > convertibility.
>> > > >But with years of experience it is convinced that above
>>requirements
>> are
>> > > >not really useful when we come to real world use cases. And XWF is
>>XML
>> > > >based bulky language where we attache WSDLs and Workflow image it
>> self.
>> > > >But
>> > > >with the recent changes WSDL part is being removed from XWF.
>> > > >
>> > > >It is worth to evaluate handy Scientific workflow languages in
>> industry
>> > > >and
>> > > >find out pros and cons, at the end of this evaluation we need to
>>come
>> up
>> > > >with idea how we should improve Airavata workflow language, either
>>we
>> > can
>> > > >improve existing XWF language, totally change to a new language
>> > available
>> > > >in industry or write a new light weight language. Basic
>>requirements
>> > that
>> > > >we expect from new improvement are, high usability, flexible, light
>> > weight
>> > > >and real time monitoring support. As you can see above requirements
>> are
>> > > >not
>> > > >direct comes with workflow languages but we need workflow language
>> which
>> > > >help to support above requirements.
>> > > >
>> > > >After reading few papers and googling, initially i have come up
>>with
>> > > >following three existing languages,
>> > > >1. YAWL <http://www.yawlfoundation.org/>
>> > > >2. WS-BPEL
>> > > >3. SIDL
>> > > ><http://computation.llnl.gov/casc/components/index.html#page=home>
>> > > >
>> > > >In my opinion SIDL is more familiar with scientific domain,
>> Radical-SAGA
>> > > >also uses slightly modified version of SIDL. Other than above three
>> > > >languages we can come up with simple workflow language base on
>>json(or
>> > > >yaml) which support all our requirements for some extends.
>> > > >
>> > > >It would be grate if I can get more input regarding the $Subject
>>form
>> > the
>> > > >airavata community. You all are more than welcome to provide any
>>type
>> of
>> > > >suggestions.
>> > > >
>> > > >Thanks,
>> > > >Shameera.
>> > > >
>> > > >
>> > > >
>> > > >--
>> > > >Best Regards,
>> > > >Shameera Rathnayaka.
>> > > >
>> > > >email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > >Blog : http://shameerarathnayaka.blogspot.com/
>> > >
>> > >
>> >
>>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Great Bruce - these are amazing insights and I hope this
in the Airavata and OODT community check this out.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Bruce Barkstrom <br...@gmail.com>
Reply-To: "architecture@airavata.apache.org"
<ar...@airavata.apache.org>
Date: Friday, September 19, 2014 9:29 AM
To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Cc: "architecture@airavata.apache.org" <ar...@airavata.apache.org>,
dev <de...@airavata.apache.org>
Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
>One factor that should be included in the group's deliberations
>on adding a workflow language to the other things in OODT
>is the impact on long-term maintenance. While there's a lot
>of enthusiasm in the developer community right now, we need
>to think about what happens when development turns into
>maintenance. The account that follows is based on my experience
>with trying to resurrect a W3C-related project to visualize RDF
>graphs.
>
>The project is called IsaViz. It's even got a W3 web site:
>http://www.w3.org/2001/11/IsaViz/
>IsaViz identifies itself as a visual authoring tool for RDF.
>Right up near the top are two dates that should serve as
>a cautionary note for people who want to pick up this tool:
>Current Stable Version: October 2007 and Current Development Version:
>May 2007. It also looks like the site was maintained by a single
>developer who did the development as a postdoc project.
>The overview page was last modified on Oct. 21, 2007.
>The installation instructions were last modified on Oct. 18, 2004.
>
>IsaViz uses a number of tools. The Installation Instructions
>identify the following:
>
> - A Java Virtual Machine version 1.3.0 or later (1.4 strongly
> recommended - see Known problems
> <http://www.w3.org/2001/11/IsaViz/overview.html#bugs>)
> - A distribution of IsaViz, which contains the following Java JAR
>files:
> - IsaViz itself *(isaviz.jar)*
> - Zoomable Visual Transformation Machine *(zvtm.jar)*
> - Jena 2.1 for IsaViz 2.1, Jena 1.6.1 for IsaViz 1.2
> - Xerces-J version 2 *(xercesImpl.jar,xmlParserAPIs.jar)* for IsaViz
> 2, Xerces 1.4.4 for IsaViz 1.2
> - GraphViz from AT&T version 1.8.9 or later (version 1.7.x is no longer
> supported in IsaViz 2, and has actually only been tested with
>version 1.9). *Note:
> some instances of version 1.10.0 had a bug that produced incomplete SVG
> files, but it has been corrected in subsequent releases *(newer
>versions
> can be obtained on the graphviz.org site
> <http://www.graphviz.org/pub/graphviz/CURRENT>).
>
>So, what complications ensue:
>1. Java has moved way beyond version 1.3 or 1.4. Since Java can
>deprecate
>code and
>since there's Oracle and OpenJDK, there may be some unpleasantries that
>might need
>fixes. I haven't seen comments from the community on whether or not these
>might be
>significant. The IsaViz documentation refers to the ancient time when Sun
>controlled
>the language. Apparently, the IsaViz code was only tested with Sun's
>j2se/1.3 or 1.4.
>2. I suppose the jar files from IsaViz version 2 would be the place to
>start in reconstructing
>this piece of software. However, one might be careful about this when you
>get into the
>installation scripts from the Installation Instructions.
>3. The Zoomable Visual Transformation Machine project is on Sourceforge.
>It's apparently
>done in Java. However, the IsaViz code used version 0.9.0, while the
>current Sourceforge
>project (at http://zvtm.sourceforge.net/) is now up to 0.11.1 for the
>stable version (Aug. 2013)
>with a more recent development version (0.11.2 - snapshot; June 2014). No
>idea if
>there would be any serious ramifications from this change.
>4. The Installation Instructions have a link to the HP Jena site. If you
>link to it, the
>page says "Oops! ..." Jena was moved from HP to apache. So if you want
>to
>do
>Jena, you now need to consult <https://jena.apache.org/>. I'm not sure
>exactly how
>the apache Jena source code or binary installations compare with what
>IsaViz is expecting.
>As a note, Jena is a BIG chunk of software. I think the tutorials on RDF
>(including OWL
>and related reasoners) are going to take a novice user (including many
>scientists) a month
>or two of dedicated time to work through. I don't know how easy IsaViz
>would be to install
>without at least a basic understanding of RDF and of the related triple
>store database.
>5. Xerces-J is the XML Java parser (see <Xerces.apache.org>), which is
>now
>up to
>version 2.11.0. Again, it isn't clear what kinds of difficulties one
>would
>encounter to use
>this library.
>6. GraphViz (at <http://www.graphviz.org/>) is now at version 2.38.0-1.
>AT&T seems
>to be maintaining a lot of installation options. I was interested in
>Ubuntu - and then
>there are different versions of that.
>
>As an additional note, Linux has developed a bunch of variants. A
>particularly active
>area of development is the creation of automated package managers - often
>with centralized
>control over installation procedures and source code libraries. The
>packages have dependencies
>on the libraries -- and there's no guarantee that an RPM package has the
>same dependencies
>as a Debian package. This is a bit like the DOI guarantee of providing a
>unique location to
>obtain original items - although publishers have been known to substitute
>new versions of
>the unique object for the "true" original.
>
>At the same time, software packages with complex networks of dependencies
>are
>not exactly easy to maintain with Linux (or Unix) scripts. Exploring the
>integrity of
>the whole package requires a fair amount of work by experienced system
>administrators.
>
>If the intent is to produce data archives (or data production facilities)
>that have long-term
>maintainability, they need to handle replication [see Barkstrom and
>Mattman, 2010, ESI]
>of objects, as well as transparency. The key attributes of such systems
>need to be
>simplicity, provenance integrity, and reliability. They aren't easy
>attributes to maintain
>over the long haul. The article on "being digital" in the current CACM
>has
>a useful
>perspective on how our enthusiasm for "rupture talk" plays out:
>Haigh, T., 2014: We Have Never Been Digital, CACM, 57,No. 09,24-28
>Peter Denning's article that follows immediately in the print version
>[Denning, P. J., 2014: The Profession of IT: Learning for the New Digital
>Age,
>CACM, 57, No. 09, 29-31] offers some additional perspective that's
>probably
>relevant to the issue of the learning curve for new technologies. That
>curve
>is usually underestimated. While everyone wants "user friendly" tools, it
>isn't
>easy for developers to get an accurate idea for how many person-hours of
>work
>it will require to make a user proficient enough to use new tools,
>particularly in the
>presence of "version churn" like we can see in the IsaViz example.
>
>Bruce B.
>
>
>On Thu, Sep 18, 2014 at 2:54 PM, Lavanya Ramakrishnan
><lramakrishnan@lbl.gov
>> wrote:
>
>> Here is my 2c -
>>
>> I think it is important to try and understand what your users are going
>>to
>> do with workflow and what kind of language they are used to
>> (domain-specific, functional, etc). They are processes called
>>user-centered
>> design processes you can use to do this or do at a minimum an informal
>> study.
>>
>> A couple of years ago, we did an introspection on why all the existing
>> workflow tools didn't have the uptake we had assumed it would. I have
>>been
>> part of a half dozen different tools over my career. We have since
>>launched
>> a project called Tigres - http://tigres.lbl.gov/ where we have learned a
>> lot due to using a user-centered design approach. We have an IEEE
>>eScience
>> paper on our initial work - which you might find interesting. I am also
>> happy to share more details on Tigres and/or the process.
>>
>> Lavanya
>>
>>
>>
>>
>>
>> On Thu, Sep 18, 2014 at 10:53 AM, BW <bw...@mysoftcloud.com> wrote:
>>
>> > Is there a list of graphical BEL workflow tools?
>> >
>> > On Thursday, September 18, 2014, Mattmann, Chris A (3980) <
>> > chris.a.mattmann@jpl.nasa.gov> wrote:
>> >
>> > > Hi Guys,
>> > >
>> > > I've been interested in this too - we don't per have a specific
>> > > OODT workflow language, but we specific workflows using XML, and
>> > > other configuration (we are also thinking of moving to JSON for
>> > > this).
>> > >
>> > > In the past I've also looked at YAWL and BPEL - both seem complex
>> > > to me.
>> > >
>> > > I wonder at the end of the day if we should adopt something more
>> > > modern like PIG or some other data flow type of language (PIG
>> > > is really neat).
>> > >
>> > > Cheers,
>> > > Chris
>> > >
>> > >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > Chris Mattmann, Ph.D.
>> > > Chief Architect
>> > > Instrument Software and Science Data Systems Section (398)
>> > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > > Office: 168-519, Mailstop: 168-527
>> > > Email: chris.a.mattmann@nasa.gov <javascript:;>
>> > > WWW: http://sunset.usc.edu/~mattmann/
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > Adjunct Associate Professor, Computer Science Department
>> > > University of Southern California, Los Angeles, CA 90089 USA
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: Shameera Rathnayaka <shameerainfo@gmail.com <javascript:;>>
>> > > Reply-To: "architecture@airavata.apache.org <javascript:;>"
>> > > <architecture@airavata.apache.org <javascript:;>>
>> > > Date: Thursday, September 18, 2014 8:26 AM
>> > > To: "architecture@airavata.apache.org <javascript:;>" <
>> > > architecture@airavata.apache.org <javascript:;>>,
>> > > dev <dev@airavata.apache.org <javascript:;>>
>> > > Subject: Evaluate Suitable Scientific Workflow Language for
>>Airavata.
>> > >
>> > > >Hi All,
>> > > >
>> > > >As we all know Airavata has its own workflow language call XWF.
>>When
>> XWF
>> > > >was introduced, main focus points are interoperability and
>> > convertibility.
>> > > >But with years of experience it is convinced that above
>>requirements
>> are
>> > > >not really useful when we come to real world use cases. And XWF is
>>XML
>> > > >based bulky language where we attache WSDLs and Workflow image it
>> self.
>> > > >But
>> > > >with the recent changes WSDL part is being removed from XWF.
>> > > >
>> > > >It is worth to evaluate handy Scientific workflow languages in
>> industry
>> > > >and
>> > > >find out pros and cons, at the end of this evaluation we need to
>>come
>> up
>> > > >with idea how we should improve Airavata workflow language, either
>>we
>> > can
>> > > >improve existing XWF language, totally change to a new language
>> > available
>> > > >in industry or write a new light weight language. Basic
>>requirements
>> > that
>> > > >we expect from new improvement are, high usability, flexible, light
>> > weight
>> > > >and real time monitoring support. As you can see above requirements
>> are
>> > > >not
>> > > >direct comes with workflow languages but we need workflow language
>> which
>> > > >help to support above requirements.
>> > > >
>> > > >After reading few papers and googling, initially i have come up
>>with
>> > > >following three existing languages,
>> > > >1. YAWL <http://www.yawlfoundation.org/>
>> > > >2. WS-BPEL
>> > > >3. SIDL
>> > > ><http://computation.llnl.gov/casc/components/index.html#page=home>
>> > > >
>> > > >In my opinion SIDL is more familiar with scientific domain,
>> Radical-SAGA
>> > > >also uses slightly modified version of SIDL. Other than above three
>> > > >languages we can come up with simple workflow language base on
>>json(or
>> > > >yaml) which support all our requirements for some extends.
>> > > >
>> > > >It would be grate if I can get more input regarding the $Subject
>>form
>> > the
>> > > >airavata community. You all are more than welcome to provide any
>>type
>> of
>> > > >suggestions.
>> > > >
>> > > >Thanks,
>> > > >Shameera.
>> > > >
>> > > >
>> > > >
>> > > >--
>> > > >Best Regards,
>> > > >Shameera Rathnayaka.
>> > > >
>> > > >email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > >Blog : http://shameerarathnayaka.blogspot.com/
>> > >
>> > >
>> >
>>
RE: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by "Miller, Mark" <mm...@sdsc.edu>.
Very interesting thoughts Bruce.
The issue of sustainability is certainly a key one, and building dependencies on another platform always brings risk.
Going back to Lavanya’s comment, I think this thought is critical.
What workflow tools in use really have big uptake by users?
At CIPRES we have tended to focus on being driven by what users are actually asking for
(of course we are always behind schedule, but in adding new features this is not the worst thing).
What do we know about the requirements for workflow within Airavata from the user POV?
It seems like the correct place to start.
Mark
From: Bruce Barkstrom [mailto:brbarkstrom@gmail.com]
Sent: Friday, September 19, 2014 9:29 AM
To: dev@oodt.apache.org
Cc: architecture@airavata.apache.org; dev
Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
One factor that should be included in the group's deliberations
on adding a workflow language to the other things in OODT
is the impact on long-term maintenance. While there's a lot
of enthusiasm in the developer community right now, we need
to think about what happens when development turns into
maintenance. The account that follows is based on my experience
with trying to resurrect a W3C-related project to visualize RDF
graphs.
The project is called IsaViz. It's even got a W3 web site:
http://www.w3.org/2001/11/IsaViz/
IsaViz identifies itself as a visual authoring tool for RDF.
Right up near the top are two dates that should serve as
a cautionary note for people who want to pick up this tool:
Current Stable Version: October 2007 and Current Development Version:
May 2007. It also looks like the site was maintained by a single
developer who did the development as a postdoc project.
The overview page was last modified on Oct. 21, 2007.
The installation instructions were last modified on Oct. 18, 2004.
IsaViz uses a number of tools. The Installation Instructions
identify the following:
* A Java Virtual Machine version 1.3.0 or later (1.4 strongly recommended - see Known problems<http://www.w3.org/2001/11/IsaViz/overview.html#bugs>)
* A distribution of IsaViz, which contains the following Java JAR files:
* IsaViz itself (isaviz.jar)
* Zoomable Visual Transformation Machine (zvtm.jar)
* Jena 2.1 for IsaViz 2.1, Jena 1.6.1 for IsaViz 1.2
* Xerces-J version 2 (xercesImpl.jar,xmlParserAPIs.jar) for IsaViz 2, Xerces 1.4.4 for IsaViz 1.2
* GraphViz from AT&T version 1.8.9 or later (version 1.7.x is no longer supported in IsaViz 2, and has actually only been tested with version 1.9). Note: some instances of version 1.10.0 had a bug that produced incomplete SVG files, but it has been corrected in subsequent releases (newer versions can be obtained on the graphviz.org site<http://www.graphviz.org/pub/graphviz/CURRENT>).
So, what complications ensue:
1. Java has moved way beyond version 1.3 or 1.4. Since Java can deprecate code and
since there's Oracle and OpenJDK, there may be some unpleasantries that might need
fixes. I haven't seen comments from the community on whether or not these might be
significant. The IsaViz documentation refers to the ancient time when Sun controlled
the language. Apparently, the IsaViz code was only tested with Sun's j2se/1.3 or 1.4.
2. I suppose the jar files from IsaViz version 2 would be the place to start in reconstructing
this piece of software. However, one might be careful about this when you get into the
installation scripts from the Installation Instructions.
3. The Zoomable Visual Transformation Machine project is on Sourceforge. It's apparently
done in Java. However, the IsaViz code used version 0.9.0, while the current Sourceforge
project (at http://zvtm.sourceforge.net/) is now up to 0.11.1 for the stable version (Aug. 2013)
with a more recent development version (0.11.2 - snapshot; June 2014). No idea if
there would be any serious ramifications from this change.
4. The Installation Instructions have a link to the HP Jena site. If you link to it, the
page says "Oops! ..." Jena was moved from HP to apache. So if you want to do
Jena, you now need to consult <https://jena.apache.org/>. I'm not sure exactly how
the apache Jena source code or binary installations compare with what IsaViz is expecting.
As a note, Jena is a BIG chunk of software. I think the tutorials on RDF (including OWL
and related reasoners) are going to take a novice user (including many scientists) a month
or two of dedicated time to work through. I don't know how easy IsaViz would be to install
without at least a basic understanding of RDF and of the related triple store database.
5. Xerces-J is the XML Java parser (see <Xerces.apache.org<http://Xerces.apache.org>>), which is now up to
version 2.11.0. Again, it isn't clear what kinds of difficulties one would encounter to use
this library.
6. GraphViz (at <http://www.graphviz.org/>) is now at version 2.38.0-1. AT&T seems
to be maintaining a lot of installation options. I was interested in Ubuntu - and then
there are different versions of that.
As an additional note, Linux has developed a bunch of variants. A particularly active
area of development is the creation of automated package managers - often with centralized
control over installation procedures and source code libraries. The packages have dependencies
on the libraries -- and there's no guarantee that an RPM package has the same dependencies
as a Debian package. This is a bit like the DOI guarantee of providing a unique location to
obtain original items - although publishers have been known to substitute new versions of
the unique object for the "true" original.
At the same time, software packages with complex networks of dependencies are
not exactly easy to maintain with Linux (or Unix) scripts. Exploring the integrity of
the whole package requires a fair amount of work by experienced system administrators.
If the intent is to produce data archives (or data production facilities) that have long-term
maintainability, they need to handle replication [see Barkstrom and Mattman, 2010, ESI]
of objects, as well as transparency. The key attributes of such systems need to be
simplicity, provenance integrity, and reliability. They aren't easy attributes to maintain
over the long haul. The article on "being digital" in the current CACM has a useful
perspective on how our enthusiasm for "rupture talk" plays out:
Haigh, T., 2014: We Have Never Been Digital, CACM, 57,No. 09,24-28
Peter Denning's article that follows immediately in the print version
[Denning, P. J., 2014: The Profession of IT: Learning for the New Digital Age,
CACM, 57, No. 09, 29-31] offers some additional perspective that's probably
relevant to the issue of the learning curve for new technologies. That curve
is usually underestimated. While everyone wants "user friendly" tools, it isn't
easy for developers to get an accurate idea for how many person-hours of work
it will require to make a user proficient enough to use new tools, particularly in the
presence of "version churn" like we can see in the IsaViz example.
Bruce B.
On Thu, Sep 18, 2014 at 2:54 PM, Lavanya Ramakrishnan <lr...@lbl.gov>> wrote:
Here is my 2c -
I think it is important to try and understand what your users are going to
do with workflow and what kind of language they are used to
(domain-specific, functional, etc). They are processes called user-centered
design processes you can use to do this or do at a minimum an informal
study.
A couple of years ago, we did an introspection on why all the existing
workflow tools didn't have the uptake we had assumed it would. I have been
part of a half dozen different tools over my career. We have since launched
a project called Tigres - http://tigres.lbl.gov/ where we have learned a
lot due to using a user-centered design approach. We have an IEEE eScience
paper on our initial work - which you might find interesting. I am also
happy to share more details on Tigres and/or the process.
Lavanya
On Thu, Sep 18, 2014 at 10:53 AM, BW <bw...@mysoftcloud.com>> wrote:
> Is there a list of graphical BEL workflow tools?
>
> On Thursday, September 18, 2014, Mattmann, Chris A (3980) <
> chris.a.mattmann@jpl.nasa.gov<ma...@jpl.nasa.gov>> wrote:
>
> > Hi Guys,
> >
> > I've been interested in this too - we don't per have a specific
> > OODT workflow language, but we specific workflows using XML, and
> > other configuration (we are also thinking of moving to JSON for
> > this).
> >
> > In the past I've also looked at YAWL and BPEL - both seem complex
> > to me.
> >
> > I wonder at the end of the day if we should adopt something more
> > modern like PIG or some other data flow type of language (PIG
> > is really neat).
> >
> > Cheers,
> > Chris
> >
> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Chris Mattmann, Ph.D.
> > Chief Architect
> > Instrument Software and Science Data Systems Section (398)
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 168-519, Mailstop: 168-527
> > Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov> <javascript:;>
> > WWW: http://sunset.usc.edu/~mattmann/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Adjunct Associate Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Shameera Rathnayaka <sh...@gmail.com> <javascript:;>>
> > Reply-To: "architecture@airavata.apache.org<ma...@airavata.apache.org> <javascript:;>"
> > <ar...@airavata.apache.org> <javascript:;>>
> > Date: Thursday, September 18, 2014 8:26 AM
> > To: "architecture@airavata.apache.org<ma...@airavata.apache.org> <javascript:;>" <
> > architecture@airavata.apache.org<ma...@airavata.apache.org> <javascript:;>>,
> > dev <de...@airavata.apache.org> <javascript:;>>
> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> >
> > >Hi All,
> > >
> > >As we all know Airavata has its own workflow language call XWF. When XWF
> > >was introduced, main focus points are interoperability and
> convertibility.
> > >But with years of experience it is convinced that above requirements are
> > >not really useful when we come to real world use cases. And XWF is XML
> > >based bulky language where we attache WSDLs and Workflow image it self.
> > >But
> > >with the recent changes WSDL part is being removed from XWF.
> > >
> > >It is worth to evaluate handy Scientific workflow languages in industry
> > >and
> > >find out pros and cons, at the end of this evaluation we need to come up
> > >with idea how we should improve Airavata workflow language, either we
> can
> > >improve existing XWF language, totally change to a new language
> available
> > >in industry or write a new light weight language. Basic requirements
> that
> > >we expect from new improvement are, high usability, flexible, light
> weight
> > >and real time monitoring support. As you can see above requirements are
> > >not
> > >direct comes with workflow languages but we need workflow language which
> > >help to support above requirements.
> > >
> > >After reading few papers and googling, initially i have come up with
> > >following three existing languages,
> > >1. YAWL <http://www.yawlfoundation.org/>
> > >2. WS-BPEL
> > >3. SIDL
> > ><http://computation.llnl.gov/casc/components/index.html#page=home>
> > >
> > >In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
> > >also uses slightly modified version of SIDL. Other than above three
> > >languages we can come up with simple workflow language base on json(or
> > >yaml) which support all our requirements for some extends.
> > >
> > >It would be grate if I can get more input regarding the $Subject form
> the
> > >airavata community. You all are more than welcome to provide any type of
> > >suggestions.
> > >
> > >Thanks,
> > >Shameera.
> > >
> > >
> > >
> > >--
> > >Best Regards,
> > >Shameera Rathnayaka.
> > >
> > >email: shameera AT apache.org<http://apache.org> , shameerainfo AT gmail.com<http://gmail.com>
> > >Blog : http://shameerarathnayaka.blogspot.com/
> >
> >
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Bruce Barkstrom <br...@gmail.com>.
One factor that should be included in the group's deliberations
on adding a workflow language to the other things in OODT
is the impact on long-term maintenance. While there's a lot
of enthusiasm in the developer community right now, we need
to think about what happens when development turns into
maintenance. The account that follows is based on my experience
with trying to resurrect a W3C-related project to visualize RDF
graphs.
The project is called IsaViz. It's even got a W3 web site:
http://www.w3.org/2001/11/IsaViz/
IsaViz identifies itself as a visual authoring tool for RDF.
Right up near the top are two dates that should serve as
a cautionary note for people who want to pick up this tool:
Current Stable Version: October 2007 and Current Development Version:
May 2007. It also looks like the site was maintained by a single
developer who did the development as a postdoc project.
The overview page was last modified on Oct. 21, 2007.
The installation instructions were last modified on Oct. 18, 2004.
IsaViz uses a number of tools. The Installation Instructions
identify the following:
- A Java Virtual Machine version 1.3.0 or later (1.4 strongly
recommended - see Known problems
<http://www.w3.org/2001/11/IsaViz/overview.html#bugs>)
- A distribution of IsaViz, which contains the following Java JAR files:
- IsaViz itself *(isaviz.jar)*
- Zoomable Visual Transformation Machine *(zvtm.jar)*
- Jena 2.1 for IsaViz 2.1, Jena 1.6.1 for IsaViz 1.2
- Xerces-J version 2 *(xercesImpl.jar,xmlParserAPIs.jar)* for IsaViz
2, Xerces 1.4.4 for IsaViz 1.2
- GraphViz from AT&T version 1.8.9 or later (version 1.7.x is no longer
supported in IsaViz 2, and has actually only been tested with
version 1.9). *Note:
some instances of version 1.10.0 had a bug that produced incomplete SVG
files, but it has been corrected in subsequent releases *(newer versions
can be obtained on the graphviz.org site
<http://www.graphviz.org/pub/graphviz/CURRENT>).
So, what complications ensue:
1. Java has moved way beyond version 1.3 or 1.4. Since Java can deprecate
code and
since there's Oracle and OpenJDK, there may be some unpleasantries that
might need
fixes. I haven't seen comments from the community on whether or not these
might be
significant. The IsaViz documentation refers to the ancient time when Sun
controlled
the language. Apparently, the IsaViz code was only tested with Sun's
j2se/1.3 or 1.4.
2. I suppose the jar files from IsaViz version 2 would be the place to
start in reconstructing
this piece of software. However, one might be careful about this when you
get into the
installation scripts from the Installation Instructions.
3. The Zoomable Visual Transformation Machine project is on Sourceforge.
It's apparently
done in Java. However, the IsaViz code used version 0.9.0, while the
current Sourceforge
project (at http://zvtm.sourceforge.net/) is now up to 0.11.1 for the
stable version (Aug. 2013)
with a more recent development version (0.11.2 - snapshot; June 2014). No
idea if
there would be any serious ramifications from this change.
4. The Installation Instructions have a link to the HP Jena site. If you
link to it, the
page says "Oops! ..." Jena was moved from HP to apache. So if you want to
do
Jena, you now need to consult <https://jena.apache.org/>. I'm not sure
exactly how
the apache Jena source code or binary installations compare with what
IsaViz is expecting.
As a note, Jena is a BIG chunk of software. I think the tutorials on RDF
(including OWL
and related reasoners) are going to take a novice user (including many
scientists) a month
or two of dedicated time to work through. I don't know how easy IsaViz
would be to install
without at least a basic understanding of RDF and of the related triple
store database.
5. Xerces-J is the XML Java parser (see <Xerces.apache.org>), which is now
up to
version 2.11.0. Again, it isn't clear what kinds of difficulties one would
encounter to use
this library.
6. GraphViz (at <http://www.graphviz.org/>) is now at version 2.38.0-1.
AT&T seems
to be maintaining a lot of installation options. I was interested in
Ubuntu - and then
there are different versions of that.
As an additional note, Linux has developed a bunch of variants. A
particularly active
area of development is the creation of automated package managers - often
with centralized
control over installation procedures and source code libraries. The
packages have dependencies
on the libraries -- and there's no guarantee that an RPM package has the
same dependencies
as a Debian package. This is a bit like the DOI guarantee of providing a
unique location to
obtain original items - although publishers have been known to substitute
new versions of
the unique object for the "true" original.
At the same time, software packages with complex networks of dependencies
are
not exactly easy to maintain with Linux (or Unix) scripts. Exploring the
integrity of
the whole package requires a fair amount of work by experienced system
administrators.
If the intent is to produce data archives (or data production facilities)
that have long-term
maintainability, they need to handle replication [see Barkstrom and
Mattman, 2010, ESI]
of objects, as well as transparency. The key attributes of such systems
need to be
simplicity, provenance integrity, and reliability. They aren't easy
attributes to maintain
over the long haul. The article on "being digital" in the current CACM has
a useful
perspective on how our enthusiasm for "rupture talk" plays out:
Haigh, T., 2014: We Have Never Been Digital, CACM, 57,No. 09,24-28
Peter Denning's article that follows immediately in the print version
[Denning, P. J., 2014: The Profession of IT: Learning for the New Digital
Age,
CACM, 57, No. 09, 29-31] offers some additional perspective that's probably
relevant to the issue of the learning curve for new technologies. That
curve
is usually underestimated. While everyone wants "user friendly" tools, it
isn't
easy for developers to get an accurate idea for how many person-hours of
work
it will require to make a user proficient enough to use new tools,
particularly in the
presence of "version churn" like we can see in the IsaViz example.
Bruce B.
On Thu, Sep 18, 2014 at 2:54 PM, Lavanya Ramakrishnan <lramakrishnan@lbl.gov
> wrote:
> Here is my 2c -
>
> I think it is important to try and understand what your users are going to
> do with workflow and what kind of language they are used to
> (domain-specific, functional, etc). They are processes called user-centered
> design processes you can use to do this or do at a minimum an informal
> study.
>
> A couple of years ago, we did an introspection on why all the existing
> workflow tools didn't have the uptake we had assumed it would. I have been
> part of a half dozen different tools over my career. We have since launched
> a project called Tigres - http://tigres.lbl.gov/ where we have learned a
> lot due to using a user-centered design approach. We have an IEEE eScience
> paper on our initial work - which you might find interesting. I am also
> happy to share more details on Tigres and/or the process.
>
> Lavanya
>
>
>
>
>
> On Thu, Sep 18, 2014 at 10:53 AM, BW <bw...@mysoftcloud.com> wrote:
>
> > Is there a list of graphical BEL workflow tools?
> >
> > On Thursday, September 18, 2014, Mattmann, Chris A (3980) <
> > chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> > > Hi Guys,
> > >
> > > I've been interested in this too - we don't per have a specific
> > > OODT workflow language, but we specific workflows using XML, and
> > > other configuration (we are also thinking of moving to JSON for
> > > this).
> > >
> > > In the past I've also looked at YAWL and BPEL - both seem complex
> > > to me.
> > >
> > > I wonder at the end of the day if we should adopt something more
> > > modern like PIG or some other data flow type of language (PIG
> > > is really neat).
> > >
> > > Cheers,
> > > Chris
> > >
> > >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > Chris Mattmann, Ph.D.
> > > Chief Architect
> > > Instrument Software and Science Data Systems Section (398)
> > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > Office: 168-519, Mailstop: 168-527
> > > Email: chris.a.mattmann@nasa.gov <javascript:;>
> > > WWW: http://sunset.usc.edu/~mattmann/
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > Adjunct Associate Professor, Computer Science Department
> > > University of Southern California, Los Angeles, CA 90089 USA
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Shameera Rathnayaka <shameerainfo@gmail.com <javascript:;>>
> > > Reply-To: "architecture@airavata.apache.org <javascript:;>"
> > > <architecture@airavata.apache.org <javascript:;>>
> > > Date: Thursday, September 18, 2014 8:26 AM
> > > To: "architecture@airavata.apache.org <javascript:;>" <
> > > architecture@airavata.apache.org <javascript:;>>,
> > > dev <dev@airavata.apache.org <javascript:;>>
> > > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> > >
> > > >Hi All,
> > > >
> > > >As we all know Airavata has its own workflow language call XWF. When
> XWF
> > > >was introduced, main focus points are interoperability and
> > convertibility.
> > > >But with years of experience it is convinced that above requirements
> are
> > > >not really useful when we come to real world use cases. And XWF is XML
> > > >based bulky language where we attache WSDLs and Workflow image it
> self.
> > > >But
> > > >with the recent changes WSDL part is being removed from XWF.
> > > >
> > > >It is worth to evaluate handy Scientific workflow languages in
> industry
> > > >and
> > > >find out pros and cons, at the end of this evaluation we need to come
> up
> > > >with idea how we should improve Airavata workflow language, either we
> > can
> > > >improve existing XWF language, totally change to a new language
> > available
> > > >in industry or write a new light weight language. Basic requirements
> > that
> > > >we expect from new improvement are, high usability, flexible, light
> > weight
> > > >and real time monitoring support. As you can see above requirements
> are
> > > >not
> > > >direct comes with workflow languages but we need workflow language
> which
> > > >help to support above requirements.
> > > >
> > > >After reading few papers and googling, initially i have come up with
> > > >following three existing languages,
> > > >1. YAWL <http://www.yawlfoundation.org/>
> > > >2. WS-BPEL
> > > >3. SIDL
> > > ><http://computation.llnl.gov/casc/components/index.html#page=home>
> > > >
> > > >In my opinion SIDL is more familiar with scientific domain,
> Radical-SAGA
> > > >also uses slightly modified version of SIDL. Other than above three
> > > >languages we can come up with simple workflow language base on json(or
> > > >yaml) which support all our requirements for some extends.
> > > >
> > > >It would be grate if I can get more input regarding the $Subject form
> > the
> > > >airavata community. You all are more than welcome to provide any type
> of
> > > >suggestions.
> > > >
> > > >Thanks,
> > > >Shameera.
> > > >
> > > >
> > > >
> > > >--
> > > >Best Regards,
> > > >Shameera Rathnayaka.
> > > >
> > > >email: shameera AT apache.org , shameerainfo AT gmail.com
> > > >Blog : http://shameerarathnayaka.blogspot.com/
> > >
> > >
> >
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Bruce Barkstrom <br...@gmail.com>.
Can you make a copy of your IEEE article available
without the paywall?
Thanks.
Bruce B.
On Thu, Sep 18, 2014 at 2:54 PM, Lavanya Ramakrishnan <lramakrishnan@lbl.gov
> wrote:
> Here is my 2c -
>
> I think it is important to try and understand what your users are going to
> do with workflow and what kind of language they are used to
> (domain-specific, functional, etc). They are processes called user-centered
> design processes you can use to do this or do at a minimum an informal
> study.
>
> A couple of years ago, we did an introspection on why all the existing
> workflow tools didn't have the uptake we had assumed it would. I have been
> part of a half dozen different tools over my career. We have since launched
> a project called Tigres - http://tigres.lbl.gov/ where we have learned a
> lot due to using a user-centered design approach. We have an IEEE eScience
> paper on our initial work - which you might find interesting. I am also
> happy to share more details on Tigres and/or the process.
>
> Lavanya
>
>
>
>
>
> On Thu, Sep 18, 2014 at 10:53 AM, BW <bw...@mysoftcloud.com> wrote:
>
> > Is there a list of graphical BEL workflow tools?
> >
> > On Thursday, September 18, 2014, Mattmann, Chris A (3980) <
> > chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> > > Hi Guys,
> > >
> > > I've been interested in this too - we don't per have a specific
> > > OODT workflow language, but we specific workflows using XML, and
> > > other configuration (we are also thinking of moving to JSON for
> > > this).
> > >
> > > In the past I've also looked at YAWL and BPEL - both seem complex
> > > to me.
> > >
> > > I wonder at the end of the day if we should adopt something more
> > > modern like PIG or some other data flow type of language (PIG
> > > is really neat).
> > >
> > > Cheers,
> > > Chris
> > >
> > >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > Chris Mattmann, Ph.D.
> > > Chief Architect
> > > Instrument Software and Science Data Systems Section (398)
> > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > Office: 168-519, Mailstop: 168-527
> > > Email: chris.a.mattmann@nasa.gov <javascript:;>
> > > WWW: http://sunset.usc.edu/~mattmann/
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > Adjunct Associate Professor, Computer Science Department
> > > University of Southern California, Los Angeles, CA 90089 USA
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Shameera Rathnayaka <shameerainfo@gmail.com <javascript:;>>
> > > Reply-To: "architecture@airavata.apache.org <javascript:;>"
> > > <architecture@airavata.apache.org <javascript:;>>
> > > Date: Thursday, September 18, 2014 8:26 AM
> > > To: "architecture@airavata.apache.org <javascript:;>" <
> > > architecture@airavata.apache.org <javascript:;>>,
> > > dev <dev@airavata.apache.org <javascript:;>>
> > > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> > >
> > > >Hi All,
> > > >
> > > >As we all know Airavata has its own workflow language call XWF. When
> XWF
> > > >was introduced, main focus points are interoperability and
> > convertibility.
> > > >But with years of experience it is convinced that above requirements
> are
> > > >not really useful when we come to real world use cases. And XWF is XML
> > > >based bulky language where we attache WSDLs and Workflow image it
> self.
> > > >But
> > > >with the recent changes WSDL part is being removed from XWF.
> > > >
> > > >It is worth to evaluate handy Scientific workflow languages in
> industry
> > > >and
> > > >find out pros and cons, at the end of this evaluation we need to come
> up
> > > >with idea how we should improve Airavata workflow language, either we
> > can
> > > >improve existing XWF language, totally change to a new language
> > available
> > > >in industry or write a new light weight language. Basic requirements
> > that
> > > >we expect from new improvement are, high usability, flexible, light
> > weight
> > > >and real time monitoring support. As you can see above requirements
> are
> > > >not
> > > >direct comes with workflow languages but we need workflow language
> which
> > > >help to support above requirements.
> > > >
> > > >After reading few papers and googling, initially i have come up with
> > > >following three existing languages,
> > > >1. YAWL <http://www.yawlfoundation.org/>
> > > >2. WS-BPEL
> > > >3. SIDL
> > > ><http://computation.llnl.gov/casc/components/index.html#page=home>
> > > >
> > > >In my opinion SIDL is more familiar with scientific domain,
> Radical-SAGA
> > > >also uses slightly modified version of SIDL. Other than above three
> > > >languages we can come up with simple workflow language base on json(or
> > > >yaml) which support all our requirements for some extends.
> > > >
> > > >It would be grate if I can get more input regarding the $Subject form
> > the
> > > >airavata community. You all are more than welcome to provide any type
> of
> > > >suggestions.
> > > >
> > > >Thanks,
> > > >Shameera.
> > > >
> > > >
> > > >
> > > >--
> > > >Best Regards,
> > > >Shameera Rathnayaka.
> > > >
> > > >email: shameera AT apache.org , shameerainfo AT gmail.com
> > > >Blog : http://shameerarathnayaka.blogspot.com/
> > >
> > >
> >
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Bruce Barkstrom <br...@gmail.com>.
One factor that should be included in the group's deliberations
on adding a workflow language to the other things in OODT
is the impact on long-term maintenance. While there's a lot
of enthusiasm in the developer community right now, we need
to think about what happens when development turns into
maintenance. The account that follows is based on my experience
with trying to resurrect a W3C-related project to visualize RDF
graphs.
The project is called IsaViz. It's even got a W3 web site:
http://www.w3.org/2001/11/IsaViz/
IsaViz identifies itself as a visual authoring tool for RDF.
Right up near the top are two dates that should serve as
a cautionary note for people who want to pick up this tool:
Current Stable Version: October 2007 and Current Development Version:
May 2007. It also looks like the site was maintained by a single
developer who did the development as a postdoc project.
The overview page was last modified on Oct. 21, 2007.
The installation instructions were last modified on Oct. 18, 2004.
IsaViz uses a number of tools. The Installation Instructions
identify the following:
- A Java Virtual Machine version 1.3.0 or later (1.4 strongly
recommended - see Known problems
<http://www.w3.org/2001/11/IsaViz/overview.html#bugs>)
- A distribution of IsaViz, which contains the following Java JAR files:
- IsaViz itself *(isaviz.jar)*
- Zoomable Visual Transformation Machine *(zvtm.jar)*
- Jena 2.1 for IsaViz 2.1, Jena 1.6.1 for IsaViz 1.2
- Xerces-J version 2 *(xercesImpl.jar,xmlParserAPIs.jar)* for IsaViz
2, Xerces 1.4.4 for IsaViz 1.2
- GraphViz from AT&T version 1.8.9 or later (version 1.7.x is no longer
supported in IsaViz 2, and has actually only been tested with
version 1.9). *Note:
some instances of version 1.10.0 had a bug that produced incomplete SVG
files, but it has been corrected in subsequent releases *(newer versions
can be obtained on the graphviz.org site
<http://www.graphviz.org/pub/graphviz/CURRENT>).
So, what complications ensue:
1. Java has moved way beyond version 1.3 or 1.4. Since Java can deprecate
code and
since there's Oracle and OpenJDK, there may be some unpleasantries that
might need
fixes. I haven't seen comments from the community on whether or not these
might be
significant. The IsaViz documentation refers to the ancient time when Sun
controlled
the language. Apparently, the IsaViz code was only tested with Sun's
j2se/1.3 or 1.4.
2. I suppose the jar files from IsaViz version 2 would be the place to
start in reconstructing
this piece of software. However, one might be careful about this when you
get into the
installation scripts from the Installation Instructions.
3. The Zoomable Visual Transformation Machine project is on Sourceforge.
It's apparently
done in Java. However, the IsaViz code used version 0.9.0, while the
current Sourceforge
project (at http://zvtm.sourceforge.net/) is now up to 0.11.1 for the
stable version (Aug. 2013)
with a more recent development version (0.11.2 - snapshot; June 2014). No
idea if
there would be any serious ramifications from this change.
4. The Installation Instructions have a link to the HP Jena site. If you
link to it, the
page says "Oops! ..." Jena was moved from HP to apache. So if you want to
do
Jena, you now need to consult <https://jena.apache.org/>. I'm not sure
exactly how
the apache Jena source code or binary installations compare with what
IsaViz is expecting.
As a note, Jena is a BIG chunk of software. I think the tutorials on RDF
(including OWL
and related reasoners) are going to take a novice user (including many
scientists) a month
or two of dedicated time to work through. I don't know how easy IsaViz
would be to install
without at least a basic understanding of RDF and of the related triple
store database.
5. Xerces-J is the XML Java parser (see <Xerces.apache.org>), which is now
up to
version 2.11.0. Again, it isn't clear what kinds of difficulties one would
encounter to use
this library.
6. GraphViz (at <http://www.graphviz.org/>) is now at version 2.38.0-1.
AT&T seems
to be maintaining a lot of installation options. I was interested in
Ubuntu - and then
there are different versions of that.
As an additional note, Linux has developed a bunch of variants. A
particularly active
area of development is the creation of automated package managers - often
with centralized
control over installation procedures and source code libraries. The
packages have dependencies
on the libraries -- and there's no guarantee that an RPM package has the
same dependencies
as a Debian package. This is a bit like the DOI guarantee of providing a
unique location to
obtain original items - although publishers have been known to substitute
new versions of
the unique object for the "true" original.
At the same time, software packages with complex networks of dependencies
are
not exactly easy to maintain with Linux (or Unix) scripts. Exploring the
integrity of
the whole package requires a fair amount of work by experienced system
administrators.
If the intent is to produce data archives (or data production facilities)
that have long-term
maintainability, they need to handle replication [see Barkstrom and
Mattman, 2010, ESI]
of objects, as well as transparency. The key attributes of such systems
need to be
simplicity, provenance integrity, and reliability. They aren't easy
attributes to maintain
over the long haul. The article on "being digital" in the current CACM has
a useful
perspective on how our enthusiasm for "rupture talk" plays out:
Haigh, T., 2014: We Have Never Been Digital, CACM, 57,No. 09,24-28
Peter Denning's article that follows immediately in the print version
[Denning, P. J., 2014: The Profession of IT: Learning for the New Digital
Age,
CACM, 57, No. 09, 29-31] offers some additional perspective that's probably
relevant to the issue of the learning curve for new technologies. That
curve
is usually underestimated. While everyone wants "user friendly" tools, it
isn't
easy for developers to get an accurate idea for how many person-hours of
work
it will require to make a user proficient enough to use new tools,
particularly in the
presence of "version churn" like we can see in the IsaViz example.
Bruce B.
On Thu, Sep 18, 2014 at 2:54 PM, Lavanya Ramakrishnan <lramakrishnan@lbl.gov
> wrote:
> Here is my 2c -
>
> I think it is important to try and understand what your users are going to
> do with workflow and what kind of language they are used to
> (domain-specific, functional, etc). They are processes called user-centered
> design processes you can use to do this or do at a minimum an informal
> study.
>
> A couple of years ago, we did an introspection on why all the existing
> workflow tools didn't have the uptake we had assumed it would. I have been
> part of a half dozen different tools over my career. We have since launched
> a project called Tigres - http://tigres.lbl.gov/ where we have learned a
> lot due to using a user-centered design approach. We have an IEEE eScience
> paper on our initial work - which you might find interesting. I am also
> happy to share more details on Tigres and/or the process.
>
> Lavanya
>
>
>
>
>
> On Thu, Sep 18, 2014 at 10:53 AM, BW <bw...@mysoftcloud.com> wrote:
>
> > Is there a list of graphical BEL workflow tools?
> >
> > On Thursday, September 18, 2014, Mattmann, Chris A (3980) <
> > chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> > > Hi Guys,
> > >
> > > I've been interested in this too - we don't per have a specific
> > > OODT workflow language, but we specific workflows using XML, and
> > > other configuration (we are also thinking of moving to JSON for
> > > this).
> > >
> > > In the past I've also looked at YAWL and BPEL - both seem complex
> > > to me.
> > >
> > > I wonder at the end of the day if we should adopt something more
> > > modern like PIG or some other data flow type of language (PIG
> > > is really neat).
> > >
> > > Cheers,
> > > Chris
> > >
> > >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > Chris Mattmann, Ph.D.
> > > Chief Architect
> > > Instrument Software and Science Data Systems Section (398)
> > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > Office: 168-519, Mailstop: 168-527
> > > Email: chris.a.mattmann@nasa.gov <javascript:;>
> > > WWW: http://sunset.usc.edu/~mattmann/
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > Adjunct Associate Professor, Computer Science Department
> > > University of Southern California, Los Angeles, CA 90089 USA
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Shameera Rathnayaka <shameerainfo@gmail.com <javascript:;>>
> > > Reply-To: "architecture@airavata.apache.org <javascript:;>"
> > > <architecture@airavata.apache.org <javascript:;>>
> > > Date: Thursday, September 18, 2014 8:26 AM
> > > To: "architecture@airavata.apache.org <javascript:;>" <
> > > architecture@airavata.apache.org <javascript:;>>,
> > > dev <dev@airavata.apache.org <javascript:;>>
> > > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> > >
> > > >Hi All,
> > > >
> > > >As we all know Airavata has its own workflow language call XWF. When
> XWF
> > > >was introduced, main focus points are interoperability and
> > convertibility.
> > > >But with years of experience it is convinced that above requirements
> are
> > > >not really useful when we come to real world use cases. And XWF is XML
> > > >based bulky language where we attache WSDLs and Workflow image it
> self.
> > > >But
> > > >with the recent changes WSDL part is being removed from XWF.
> > > >
> > > >It is worth to evaluate handy Scientific workflow languages in
> industry
> > > >and
> > > >find out pros and cons, at the end of this evaluation we need to come
> up
> > > >with idea how we should improve Airavata workflow language, either we
> > can
> > > >improve existing XWF language, totally change to a new language
> > available
> > > >in industry or write a new light weight language. Basic requirements
> > that
> > > >we expect from new improvement are, high usability, flexible, light
> > weight
> > > >and real time monitoring support. As you can see above requirements
> are
> > > >not
> > > >direct comes with workflow languages but we need workflow language
> which
> > > >help to support above requirements.
> > > >
> > > >After reading few papers and googling, initially i have come up with
> > > >following three existing languages,
> > > >1. YAWL <http://www.yawlfoundation.org/>
> > > >2. WS-BPEL
> > > >3. SIDL
> > > ><http://computation.llnl.gov/casc/components/index.html#page=home>
> > > >
> > > >In my opinion SIDL is more familiar with scientific domain,
> Radical-SAGA
> > > >also uses slightly modified version of SIDL. Other than above three
> > > >languages we can come up with simple workflow language base on json(or
> > > >yaml) which support all our requirements for some extends.
> > > >
> > > >It would be grate if I can get more input regarding the $Subject form
> > the
> > > >airavata community. You all are more than welcome to provide any type
> of
> > > >suggestions.
> > > >
> > > >Thanks,
> > > >Shameera.
> > > >
> > > >
> > > >
> > > >--
> > > >Best Regards,
> > > >Shameera Rathnayaka.
> > > >
> > > >email: shameera AT apache.org , shameerainfo AT gmail.com
> > > >Blog : http://shameerarathnayaka.blogspot.com/
> > >
> > >
> >
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Bruce Barkstrom <br...@gmail.com>.
One factor that should be included in the group's deliberations
on adding a workflow language to the other things in OODT
is the impact on long-term maintenance. While there's a lot
of enthusiasm in the developer community right now, we need
to think about what happens when development turns into
maintenance. The account that follows is based on my experience
with trying to resurrect a W3C-related project to visualize RDF
graphs.
The project is called IsaViz. It's even got a W3 web site:
http://www.w3.org/2001/11/IsaViz/
IsaViz identifies itself as a visual authoring tool for RDF.
Right up near the top are two dates that should serve as
a cautionary note for people who want to pick up this tool:
Current Stable Version: October 2007 and Current Development Version:
May 2007. It also looks like the site was maintained by a single
developer who did the development as a postdoc project.
The overview page was last modified on Oct. 21, 2007.
The installation instructions were last modified on Oct. 18, 2004.
IsaViz uses a number of tools. The Installation Instructions
identify the following:
- A Java Virtual Machine version 1.3.0 or later (1.4 strongly
recommended - see Known problems
<http://www.w3.org/2001/11/IsaViz/overview.html#bugs>)
- A distribution of IsaViz, which contains the following Java JAR files:
- IsaViz itself *(isaviz.jar)*
- Zoomable Visual Transformation Machine *(zvtm.jar)*
- Jena 2.1 for IsaViz 2.1, Jena 1.6.1 for IsaViz 1.2
- Xerces-J version 2 *(xercesImpl.jar,xmlParserAPIs.jar)* for IsaViz
2, Xerces 1.4.4 for IsaViz 1.2
- GraphViz from AT&T version 1.8.9 or later (version 1.7.x is no longer
supported in IsaViz 2, and has actually only been tested with
version 1.9). *Note:
some instances of version 1.10.0 had a bug that produced incomplete SVG
files, but it has been corrected in subsequent releases *(newer versions
can be obtained on the graphviz.org site
<http://www.graphviz.org/pub/graphviz/CURRENT>).
So, what complications ensue:
1. Java has moved way beyond version 1.3 or 1.4. Since Java can deprecate
code and
since there's Oracle and OpenJDK, there may be some unpleasantries that
might need
fixes. I haven't seen comments from the community on whether or not these
might be
significant. The IsaViz documentation refers to the ancient time when Sun
controlled
the language. Apparently, the IsaViz code was only tested with Sun's
j2se/1.3 or 1.4.
2. I suppose the jar files from IsaViz version 2 would be the place to
start in reconstructing
this piece of software. However, one might be careful about this when you
get into the
installation scripts from the Installation Instructions.
3. The Zoomable Visual Transformation Machine project is on Sourceforge.
It's apparently
done in Java. However, the IsaViz code used version 0.9.0, while the
current Sourceforge
project (at http://zvtm.sourceforge.net/) is now up to 0.11.1 for the
stable version (Aug. 2013)
with a more recent development version (0.11.2 - snapshot; June 2014). No
idea if
there would be any serious ramifications from this change.
4. The Installation Instructions have a link to the HP Jena site. If you
link to it, the
page says "Oops! ..." Jena was moved from HP to apache. So if you want to
do
Jena, you now need to consult <https://jena.apache.org/>. I'm not sure
exactly how
the apache Jena source code or binary installations compare with what
IsaViz is expecting.
As a note, Jena is a BIG chunk of software. I think the tutorials on RDF
(including OWL
and related reasoners) are going to take a novice user (including many
scientists) a month
or two of dedicated time to work through. I don't know how easy IsaViz
would be to install
without at least a basic understanding of RDF and of the related triple
store database.
5. Xerces-J is the XML Java parser (see <Xerces.apache.org>), which is now
up to
version 2.11.0. Again, it isn't clear what kinds of difficulties one would
encounter to use
this library.
6. GraphViz (at <http://www.graphviz.org/>) is now at version 2.38.0-1.
AT&T seems
to be maintaining a lot of installation options. I was interested in
Ubuntu - and then
there are different versions of that.
As an additional note, Linux has developed a bunch of variants. A
particularly active
area of development is the creation of automated package managers - often
with centralized
control over installation procedures and source code libraries. The
packages have dependencies
on the libraries -- and there's no guarantee that an RPM package has the
same dependencies
as a Debian package. This is a bit like the DOI guarantee of providing a
unique location to
obtain original items - although publishers have been known to substitute
new versions of
the unique object for the "true" original.
At the same time, software packages with complex networks of dependencies
are
not exactly easy to maintain with Linux (or Unix) scripts. Exploring the
integrity of
the whole package requires a fair amount of work by experienced system
administrators.
If the intent is to produce data archives (or data production facilities)
that have long-term
maintainability, they need to handle replication [see Barkstrom and
Mattman, 2010, ESI]
of objects, as well as transparency. The key attributes of such systems
need to be
simplicity, provenance integrity, and reliability. They aren't easy
attributes to maintain
over the long haul. The article on "being digital" in the current CACM has
a useful
perspective on how our enthusiasm for "rupture talk" plays out:
Haigh, T., 2014: We Have Never Been Digital, CACM, 57,No. 09,24-28
Peter Denning's article that follows immediately in the print version
[Denning, P. J., 2014: The Profession of IT: Learning for the New Digital
Age,
CACM, 57, No. 09, 29-31] offers some additional perspective that's probably
relevant to the issue of the learning curve for new technologies. That
curve
is usually underestimated. While everyone wants "user friendly" tools, it
isn't
easy for developers to get an accurate idea for how many person-hours of
work
it will require to make a user proficient enough to use new tools,
particularly in the
presence of "version churn" like we can see in the IsaViz example.
Bruce B.
On Thu, Sep 18, 2014 at 2:54 PM, Lavanya Ramakrishnan <lramakrishnan@lbl.gov
> wrote:
> Here is my 2c -
>
> I think it is important to try and understand what your users are going to
> do with workflow and what kind of language they are used to
> (domain-specific, functional, etc). They are processes called user-centered
> design processes you can use to do this or do at a minimum an informal
> study.
>
> A couple of years ago, we did an introspection on why all the existing
> workflow tools didn't have the uptake we had assumed it would. I have been
> part of a half dozen different tools over my career. We have since launched
> a project called Tigres - http://tigres.lbl.gov/ where we have learned a
> lot due to using a user-centered design approach. We have an IEEE eScience
> paper on our initial work - which you might find interesting. I am also
> happy to share more details on Tigres and/or the process.
>
> Lavanya
>
>
>
>
>
> On Thu, Sep 18, 2014 at 10:53 AM, BW <bw...@mysoftcloud.com> wrote:
>
> > Is there a list of graphical BEL workflow tools?
> >
> > On Thursday, September 18, 2014, Mattmann, Chris A (3980) <
> > chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> > > Hi Guys,
> > >
> > > I've been interested in this too - we don't per have a specific
> > > OODT workflow language, but we specific workflows using XML, and
> > > other configuration (we are also thinking of moving to JSON for
> > > this).
> > >
> > > In the past I've also looked at YAWL and BPEL - both seem complex
> > > to me.
> > >
> > > I wonder at the end of the day if we should adopt something more
> > > modern like PIG or some other data flow type of language (PIG
> > > is really neat).
> > >
> > > Cheers,
> > > Chris
> > >
> > >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > Chris Mattmann, Ph.D.
> > > Chief Architect
> > > Instrument Software and Science Data Systems Section (398)
> > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > Office: 168-519, Mailstop: 168-527
> > > Email: chris.a.mattmann@nasa.gov <javascript:;>
> > > WWW: http://sunset.usc.edu/~mattmann/
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > Adjunct Associate Professor, Computer Science Department
> > > University of Southern California, Los Angeles, CA 90089 USA
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Shameera Rathnayaka <shameerainfo@gmail.com <javascript:;>>
> > > Reply-To: "architecture@airavata.apache.org <javascript:;>"
> > > <architecture@airavata.apache.org <javascript:;>>
> > > Date: Thursday, September 18, 2014 8:26 AM
> > > To: "architecture@airavata.apache.org <javascript:;>" <
> > > architecture@airavata.apache.org <javascript:;>>,
> > > dev <dev@airavata.apache.org <javascript:;>>
> > > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> > >
> > > >Hi All,
> > > >
> > > >As we all know Airavata has its own workflow language call XWF. When
> XWF
> > > >was introduced, main focus points are interoperability and
> > convertibility.
> > > >But with years of experience it is convinced that above requirements
> are
> > > >not really useful when we come to real world use cases. And XWF is XML
> > > >based bulky language where we attache WSDLs and Workflow image it
> self.
> > > >But
> > > >with the recent changes WSDL part is being removed from XWF.
> > > >
> > > >It is worth to evaluate handy Scientific workflow languages in
> industry
> > > >and
> > > >find out pros and cons, at the end of this evaluation we need to come
> up
> > > >with idea how we should improve Airavata workflow language, either we
> > can
> > > >improve existing XWF language, totally change to a new language
> > available
> > > >in industry or write a new light weight language. Basic requirements
> > that
> > > >we expect from new improvement are, high usability, flexible, light
> > weight
> > > >and real time monitoring support. As you can see above requirements
> are
> > > >not
> > > >direct comes with workflow languages but we need workflow language
> which
> > > >help to support above requirements.
> > > >
> > > >After reading few papers and googling, initially i have come up with
> > > >following three existing languages,
> > > >1. YAWL <http://www.yawlfoundation.org/>
> > > >2. WS-BPEL
> > > >3. SIDL
> > > ><http://computation.llnl.gov/casc/components/index.html#page=home>
> > > >
> > > >In my opinion SIDL is more familiar with scientific domain,
> Radical-SAGA
> > > >also uses slightly modified version of SIDL. Other than above three
> > > >languages we can come up with simple workflow language base on json(or
> > > >yaml) which support all our requirements for some extends.
> > > >
> > > >It would be grate if I can get more input regarding the $Subject form
> > the
> > > >airavata community. You all are more than welcome to provide any type
> of
> > > >suggestions.
> > > >
> > > >Thanks,
> > > >Shameera.
> > > >
> > > >
> > > >
> > > >--
> > > >Best Regards,
> > > >Shameera Rathnayaka.
> > > >
> > > >email: shameera AT apache.org , shameerainfo AT gmail.com
> > > >Blog : http://shameerarathnayaka.blogspot.com/
> > >
> > >
> >
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Lavanya Ramakrishnan <lr...@lbl.gov>.
Here is my 2c -
I think it is important to try and understand what your users are going to
do with workflow and what kind of language they are used to
(domain-specific, functional, etc). They are processes called user-centered
design processes you can use to do this or do at a minimum an informal
study.
A couple of years ago, we did an introspection on why all the existing
workflow tools didn't have the uptake we had assumed it would. I have been
part of a half dozen different tools over my career. We have since launched
a project called Tigres - http://tigres.lbl.gov/ where we have learned a
lot due to using a user-centered design approach. We have an IEEE eScience
paper on our initial work - which you might find interesting. I am also
happy to share more details on Tigres and/or the process.
Lavanya
On Thu, Sep 18, 2014 at 10:53 AM, BW <bw...@mysoftcloud.com> wrote:
> Is there a list of graphical BEL workflow tools?
>
> On Thursday, September 18, 2014, Mattmann, Chris A (3980) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
> > Hi Guys,
> >
> > I've been interested in this too - we don't per have a specific
> > OODT workflow language, but we specific workflows using XML, and
> > other configuration (we are also thinking of moving to JSON for
> > this).
> >
> > In the past I've also looked at YAWL and BPEL - both seem complex
> > to me.
> >
> > I wonder at the end of the day if we should adopt something more
> > modern like PIG or some other data flow type of language (PIG
> > is really neat).
> >
> > Cheers,
> > Chris
> >
> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Chris Mattmann, Ph.D.
> > Chief Architect
> > Instrument Software and Science Data Systems Section (398)
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 168-519, Mailstop: 168-527
> > Email: chris.a.mattmann@nasa.gov <javascript:;>
> > WWW: http://sunset.usc.edu/~mattmann/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Adjunct Associate Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Shameera Rathnayaka <shameerainfo@gmail.com <javascript:;>>
> > Reply-To: "architecture@airavata.apache.org <javascript:;>"
> > <architecture@airavata.apache.org <javascript:;>>
> > Date: Thursday, September 18, 2014 8:26 AM
> > To: "architecture@airavata.apache.org <javascript:;>" <
> > architecture@airavata.apache.org <javascript:;>>,
> > dev <dev@airavata.apache.org <javascript:;>>
> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> >
> > >Hi All,
> > >
> > >As we all know Airavata has its own workflow language call XWF. When XWF
> > >was introduced, main focus points are interoperability and
> convertibility.
> > >But with years of experience it is convinced that above requirements are
> > >not really useful when we come to real world use cases. And XWF is XML
> > >based bulky language where we attache WSDLs and Workflow image it self.
> > >But
> > >with the recent changes WSDL part is being removed from XWF.
> > >
> > >It is worth to evaluate handy Scientific workflow languages in industry
> > >and
> > >find out pros and cons, at the end of this evaluation we need to come up
> > >with idea how we should improve Airavata workflow language, either we
> can
> > >improve existing XWF language, totally change to a new language
> available
> > >in industry or write a new light weight language. Basic requirements
> that
> > >we expect from new improvement are, high usability, flexible, light
> weight
> > >and real time monitoring support. As you can see above requirements are
> > >not
> > >direct comes with workflow languages but we need workflow language which
> > >help to support above requirements.
> > >
> > >After reading few papers and googling, initially i have come up with
> > >following three existing languages,
> > >1. YAWL <http://www.yawlfoundation.org/>
> > >2. WS-BPEL
> > >3. SIDL
> > ><http://computation.llnl.gov/casc/components/index.html#page=home>
> > >
> > >In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
> > >also uses slightly modified version of SIDL. Other than above three
> > >languages we can come up with simple workflow language base on json(or
> > >yaml) which support all our requirements for some extends.
> > >
> > >It would be grate if I can get more input regarding the $Subject form
> the
> > >airavata community. You all are more than welcome to provide any type of
> > >suggestions.
> > >
> > >Thanks,
> > >Shameera.
> > >
> > >
> > >
> > >--
> > >Best Regards,
> > >Shameera Rathnayaka.
> > >
> > >email: shameera AT apache.org , shameerainfo AT gmail.com
> > >Blog : http://shameerarathnayaka.blogspot.com/
> >
> >
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Lavanya Ramakrishnan <lr...@lbl.gov>.
Here is my 2c -
I think it is important to try and understand what your users are going to
do with workflow and what kind of language they are used to
(domain-specific, functional, etc). They are processes called user-centered
design processes you can use to do this or do at a minimum an informal
study.
A couple of years ago, we did an introspection on why all the existing
workflow tools didn't have the uptake we had assumed it would. I have been
part of a half dozen different tools over my career. We have since launched
a project called Tigres - http://tigres.lbl.gov/ where we have learned a
lot due to using a user-centered design approach. We have an IEEE eScience
paper on our initial work - which you might find interesting. I am also
happy to share more details on Tigres and/or the process.
Lavanya
On Thu, Sep 18, 2014 at 10:53 AM, BW <bw...@mysoftcloud.com> wrote:
> Is there a list of graphical BEL workflow tools?
>
> On Thursday, September 18, 2014, Mattmann, Chris A (3980) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
> > Hi Guys,
> >
> > I've been interested in this too - we don't per have a specific
> > OODT workflow language, but we specific workflows using XML, and
> > other configuration (we are also thinking of moving to JSON for
> > this).
> >
> > In the past I've also looked at YAWL and BPEL - both seem complex
> > to me.
> >
> > I wonder at the end of the day if we should adopt something more
> > modern like PIG or some other data flow type of language (PIG
> > is really neat).
> >
> > Cheers,
> > Chris
> >
> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Chris Mattmann, Ph.D.
> > Chief Architect
> > Instrument Software and Science Data Systems Section (398)
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 168-519, Mailstop: 168-527
> > Email: chris.a.mattmann@nasa.gov <javascript:;>
> > WWW: http://sunset.usc.edu/~mattmann/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Adjunct Associate Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Shameera Rathnayaka <shameerainfo@gmail.com <javascript:;>>
> > Reply-To: "architecture@airavata.apache.org <javascript:;>"
> > <architecture@airavata.apache.org <javascript:;>>
> > Date: Thursday, September 18, 2014 8:26 AM
> > To: "architecture@airavata.apache.org <javascript:;>" <
> > architecture@airavata.apache.org <javascript:;>>,
> > dev <dev@airavata.apache.org <javascript:;>>
> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> >
> > >Hi All,
> > >
> > >As we all know Airavata has its own workflow language call XWF. When XWF
> > >was introduced, main focus points are interoperability and
> convertibility.
> > >But with years of experience it is convinced that above requirements are
> > >not really useful when we come to real world use cases. And XWF is XML
> > >based bulky language where we attache WSDLs and Workflow image it self.
> > >But
> > >with the recent changes WSDL part is being removed from XWF.
> > >
> > >It is worth to evaluate handy Scientific workflow languages in industry
> > >and
> > >find out pros and cons, at the end of this evaluation we need to come up
> > >with idea how we should improve Airavata workflow language, either we
> can
> > >improve existing XWF language, totally change to a new language
> available
> > >in industry or write a new light weight language. Basic requirements
> that
> > >we expect from new improvement are, high usability, flexible, light
> weight
> > >and real time monitoring support. As you can see above requirements are
> > >not
> > >direct comes with workflow languages but we need workflow language which
> > >help to support above requirements.
> > >
> > >After reading few papers and googling, initially i have come up with
> > >following three existing languages,
> > >1. YAWL <http://www.yawlfoundation.org/>
> > >2. WS-BPEL
> > >3. SIDL
> > ><http://computation.llnl.gov/casc/components/index.html#page=home>
> > >
> > >In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
> > >also uses slightly modified version of SIDL. Other than above three
> > >languages we can come up with simple workflow language base on json(or
> > >yaml) which support all our requirements for some extends.
> > >
> > >It would be grate if I can get more input regarding the $Subject form
> the
> > >airavata community. You all are more than welcome to provide any type of
> > >suggestions.
> > >
> > >Thanks,
> > >Shameera.
> > >
> > >
> > >
> > >--
> > >Best Regards,
> > >Shameera Rathnayaka.
> > >
> > >email: shameera AT apache.org , shameerainfo AT gmail.com
> > >Blog : http://shameerarathnayaka.blogspot.com/
> >
> >
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Lavanya Ramakrishnan <lr...@lbl.gov>.
Here is my 2c -
I think it is important to try and understand what your users are going to
do with workflow and what kind of language they are used to
(domain-specific, functional, etc). They are processes called user-centered
design processes you can use to do this or do at a minimum an informal
study.
A couple of years ago, we did an introspection on why all the existing
workflow tools didn't have the uptake we had assumed it would. I have been
part of a half dozen different tools over my career. We have since launched
a project called Tigres - http://tigres.lbl.gov/ where we have learned a
lot due to using a user-centered design approach. We have an IEEE eScience
paper on our initial work - which you might find interesting. I am also
happy to share more details on Tigres and/or the process.
Lavanya
On Thu, Sep 18, 2014 at 10:53 AM, BW <bw...@mysoftcloud.com> wrote:
> Is there a list of graphical BEL workflow tools?
>
> On Thursday, September 18, 2014, Mattmann, Chris A (3980) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
> > Hi Guys,
> >
> > I've been interested in this too - we don't per have a specific
> > OODT workflow language, but we specific workflows using XML, and
> > other configuration (we are also thinking of moving to JSON for
> > this).
> >
> > In the past I've also looked at YAWL and BPEL - both seem complex
> > to me.
> >
> > I wonder at the end of the day if we should adopt something more
> > modern like PIG or some other data flow type of language (PIG
> > is really neat).
> >
> > Cheers,
> > Chris
> >
> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Chris Mattmann, Ph.D.
> > Chief Architect
> > Instrument Software and Science Data Systems Section (398)
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 168-519, Mailstop: 168-527
> > Email: chris.a.mattmann@nasa.gov <javascript:;>
> > WWW: http://sunset.usc.edu/~mattmann/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Adjunct Associate Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Shameera Rathnayaka <shameerainfo@gmail.com <javascript:;>>
> > Reply-To: "architecture@airavata.apache.org <javascript:;>"
> > <architecture@airavata.apache.org <javascript:;>>
> > Date: Thursday, September 18, 2014 8:26 AM
> > To: "architecture@airavata.apache.org <javascript:;>" <
> > architecture@airavata.apache.org <javascript:;>>,
> > dev <dev@airavata.apache.org <javascript:;>>
> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> >
> > >Hi All,
> > >
> > >As we all know Airavata has its own workflow language call XWF. When XWF
> > >was introduced, main focus points are interoperability and
> convertibility.
> > >But with years of experience it is convinced that above requirements are
> > >not really useful when we come to real world use cases. And XWF is XML
> > >based bulky language where we attache WSDLs and Workflow image it self.
> > >But
> > >with the recent changes WSDL part is being removed from XWF.
> > >
> > >It is worth to evaluate handy Scientific workflow languages in industry
> > >and
> > >find out pros and cons, at the end of this evaluation we need to come up
> > >with idea how we should improve Airavata workflow language, either we
> can
> > >improve existing XWF language, totally change to a new language
> available
> > >in industry or write a new light weight language. Basic requirements
> that
> > >we expect from new improvement are, high usability, flexible, light
> weight
> > >and real time monitoring support. As you can see above requirements are
> > >not
> > >direct comes with workflow languages but we need workflow language which
> > >help to support above requirements.
> > >
> > >After reading few papers and googling, initially i have come up with
> > >following three existing languages,
> > >1. YAWL <http://www.yawlfoundation.org/>
> > >2. WS-BPEL
> > >3. SIDL
> > ><http://computation.llnl.gov/casc/components/index.html#page=home>
> > >
> > >In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
> > >also uses slightly modified version of SIDL. Other than above three
> > >languages we can come up with simple workflow language base on json(or
> > >yaml) which support all our requirements for some extends.
> > >
> > >It would be grate if I can get more input regarding the $Subject form
> the
> > >airavata community. You all are more than welcome to provide any type of
> > >suggestions.
> > >
> > >Thanks,
> > >Shameera.
> > >
> > >
> > >
> > >--
> > >Best Regards,
> > >Shameera Rathnayaka.
> > >
> > >email: shameera AT apache.org , shameerainfo AT gmail.com
> > >Blog : http://shameerarathnayaka.blogspot.com/
> >
> >
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by BW <bw...@mysoftcloud.com>.
Is there a list of graphical BEL workflow tools?
On Thursday, September 18, 2014, Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hi Guys,
>
> I've been interested in this too - we don't per have a specific
> OODT workflow language, but we specific workflows using XML, and
> other configuration (we are also thinking of moving to JSON for
> this).
>
> In the past I've also looked at YAWL and BPEL - both seem complex
> to me.
>
> I wonder at the end of the day if we should adopt something more
> modern like PIG or some other data flow type of language (PIG
> is really neat).
>
> Cheers,
> Chris
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov <javascript:;>
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Shameera Rathnayaka <shameerainfo@gmail.com <javascript:;>>
> Reply-To: "architecture@airavata.apache.org <javascript:;>"
> <architecture@airavata.apache.org <javascript:;>>
> Date: Thursday, September 18, 2014 8:26 AM
> To: "architecture@airavata.apache.org <javascript:;>" <
> architecture@airavata.apache.org <javascript:;>>,
> dev <dev@airavata.apache.org <javascript:;>>
> Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>
> >Hi All,
> >
> >As we all know Airavata has its own workflow language call XWF. When XWF
> >was introduced, main focus points are interoperability and convertibility.
> >But with years of experience it is convinced that above requirements are
> >not really useful when we come to real world use cases. And XWF is XML
> >based bulky language where we attache WSDLs and Workflow image it self.
> >But
> >with the recent changes WSDL part is being removed from XWF.
> >
> >It is worth to evaluate handy Scientific workflow languages in industry
> >and
> >find out pros and cons, at the end of this evaluation we need to come up
> >with idea how we should improve Airavata workflow language, either we can
> >improve existing XWF language, totally change to a new language available
> >in industry or write a new light weight language. Basic requirements that
> >we expect from new improvement are, high usability, flexible, light weight
> >and real time monitoring support. As you can see above requirements are
> >not
> >direct comes with workflow languages but we need workflow language which
> >help to support above requirements.
> >
> >After reading few papers and googling, initially i have come up with
> >following three existing languages,
> >1. YAWL <http://www.yawlfoundation.org/>
> >2. WS-BPEL
> >3. SIDL
> ><http://computation.llnl.gov/casc/components/index.html#page=home>
> >
> >In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
> >also uses slightly modified version of SIDL. Other than above three
> >languages we can come up with simple workflow language base on json(or
> >yaml) which support all our requirements for some extends.
> >
> >It would be grate if I can get more input regarding the $Subject form the
> >airavata community. You all are more than welcome to provide any type of
> >suggestions.
> >
> >Thanks,
> >Shameera.
> >
> >
> >
> >--
> >Best Regards,
> >Shameera Rathnayaka.
> >
> >email: shameera AT apache.org , shameerainfo AT gmail.com
> >Blog : http://shameerarathnayaka.blogspot.com/
>
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by BW <bw...@mysoftcloud.com>.
Is there a list of graphical BEL workflow tools?
On Thursday, September 18, 2014, Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hi Guys,
>
> I've been interested in this too - we don't per have a specific
> OODT workflow language, but we specific workflows using XML, and
> other configuration (we are also thinking of moving to JSON for
> this).
>
> In the past I've also looked at YAWL and BPEL - both seem complex
> to me.
>
> I wonder at the end of the day if we should adopt something more
> modern like PIG or some other data flow type of language (PIG
> is really neat).
>
> Cheers,
> Chris
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov <javascript:;>
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Shameera Rathnayaka <shameerainfo@gmail.com <javascript:;>>
> Reply-To: "architecture@airavata.apache.org <javascript:;>"
> <architecture@airavata.apache.org <javascript:;>>
> Date: Thursday, September 18, 2014 8:26 AM
> To: "architecture@airavata.apache.org <javascript:;>" <
> architecture@airavata.apache.org <javascript:;>>,
> dev <dev@airavata.apache.org <javascript:;>>
> Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>
> >Hi All,
> >
> >As we all know Airavata has its own workflow language call XWF. When XWF
> >was introduced, main focus points are interoperability and convertibility.
> >But with years of experience it is convinced that above requirements are
> >not really useful when we come to real world use cases. And XWF is XML
> >based bulky language where we attache WSDLs and Workflow image it self.
> >But
> >with the recent changes WSDL part is being removed from XWF.
> >
> >It is worth to evaluate handy Scientific workflow languages in industry
> >and
> >find out pros and cons, at the end of this evaluation we need to come up
> >with idea how we should improve Airavata workflow language, either we can
> >improve existing XWF language, totally change to a new language available
> >in industry or write a new light weight language. Basic requirements that
> >we expect from new improvement are, high usability, flexible, light weight
> >and real time monitoring support. As you can see above requirements are
> >not
> >direct comes with workflow languages but we need workflow language which
> >help to support above requirements.
> >
> >After reading few papers and googling, initially i have come up with
> >following three existing languages,
> >1. YAWL <http://www.yawlfoundation.org/>
> >2. WS-BPEL
> >3. SIDL
> ><http://computation.llnl.gov/casc/components/index.html#page=home>
> >
> >In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
> >also uses slightly modified version of SIDL. Other than above three
> >languages we can come up with simple workflow language base on json(or
> >yaml) which support all our requirements for some extends.
> >
> >It would be grate if I can get more input regarding the $Subject form the
> >airavata community. You all are more than welcome to provide any type of
> >suggestions.
> >
> >Thanks,
> >Shameera.
> >
> >
> >
> >--
> >Best Regards,
> >Shameera Rathnayaka.
> >
> >email: shameera AT apache.org , shameerainfo AT gmail.com
> >Blog : http://shameerarathnayaka.blogspot.com/
>
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Amila Jayasekara <th...@gmail.com>.
Hi Shameera,
Thanks for the reply.
I wouldnt call XWF a "language". It is not a language in any sense but an
intermediate representation of a workflow graph.
To my understanding what you need is not a language but an efficient
portable intermediate representation. Seems you also have realized it. JSON
might be a good candidate provided most of the frontends are web based.
Thanks
-Thejaka
On Mon, Oct 20, 2014 at 10:35 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:
> Hi Amila,
>
> Please see my comments inline.
>
> On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara <thejaka.amila@gmail.com
> >
> wrote:
>
> > Hello,
> >
> > I am sorry, I am bit late on this thread. But when reading through this
> > thread I simply got lost, what this thread is discussing. I have few
> > questions.
> >
> > 1. @Shameera : Is XWF actually a language to define workflow ? To my
> > understanding it was an intermediate representation to convert workflow
> > defined in UI to java object model. Was XWF ever used by any airavata
> user
> > to define a workflow graph ?
> >
>
> Yes, XWF is the language defined and used by Airavata to explain the
> workflows but it is not well documented. Both server and client sides read
> this description language to generate runtime java representation. so
> when you used XBaya to create a workflow with multiple applications, under
> the hood XBaya generate xwf which describe that workflow to server.
>
>
>
> >
> > From initial description what I understood is we are looking for a
> > improved "intermediate representation", not a language which describes
> > workflows.
> >
> >
> >
>
> > 2. So what is the exact purpose of this proposed language ?
> > - Is it to hide parallelism in workflows ?
> > - Is it to replace the XBAYA functionalities ? (i hope not)
> >
>
>
> Actually initial idea was to identify good, well-defined and scientific
> domain friendly language. Whole purpose of this effort is reduce the entry
> barrier of the end user. But later it is understood that introducing a new
> language won't fix our issue.
>
>
> >
> > 3. What are we trying to achieve by this proposed language which we
> cannot
> > achieved through workflow UI tool ?
> >
> > 4. Who is going to use this language ?
> >
>
> As I explained, our direction has been changed.By introducing a new
> language we are get nothing but nice description file. No functional
> improvements etc ... The current language should use all airavata
> client(Currently we have only XBaya) side applications to explain the
> workflow to the server side.
>
>
>
> >
> > 5. Why would a user prefer (assuming intended audience of proposed
> > language is end users) a language over a Work Flow UI tool such as XBAYA
> ?
> > (In other words what are the things we can do with language which we
> cannot
> > do with UI)
> >
>
> Let's say you are going to write a new web base client for Airavata which
> generate workflows and launch it. What you need to do is do some magic
> with UI and finally come up with description language and send it to server
> side. Here you need to learn how to write a valid XWF file and write your
> own JS code to generate it. But if airavata has a JavaScript API which can
> be used to generate XWF for you by getting JSON objects as input then it
> would be great help for client side developers.
>
> Thank,
> Shameera.
>
>
>
> >
> > Sorry, if above questions are in-appropriate, just wanted to understand
> > what exactly needed.
> >
> > Thanks
> > -Thejaka Amila
> >
> >
> >
> > On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
> > wrote:
> >
> >> I think I'm not suggesting to create a Workflow interpreter using Python
> >> etc. What I'm suggesting is to remove the Worflow aspect from core
> Airavata
> >> and move it to a more higher level component. The more I think about it,
> >> the model I'm suggesting is similar to what Hadoop, Storm etc has done
> for
> >> distributed system computations. This model is proven to be successful
> over
> >> the years.
> >>
> >> Keeping what Airavata does at its core can help you to build a more
> >> robust system. If we look at Airavata as middleware to execute
> applications
> >> on computing resources we can simplify what Airavata does and focus on
> >> improving the core functionality. All the successful systems have
> thrived
> >> on defining what it does at its core and keeping it simple and being
> >> excellent at what it does. In that regard keeping workflow aspect out of
> >> Airavata can help you to focus on the core problem. That is to execute
> an
> >> application on a remote computing resource in a fault tolerant and
> scalable
> >> manner.
> >>
> >> What I'm suggesting is to give the Orchestrator the capability to
> execute
> >> a Driver program that is specified by the user. (This program can be
> >> written in Python, Java or any other language). This driver program is
> >> similar to what you define in a Hadoop or Storm configuration. The
> driver
> >> program specifies the flow of the computation. It specifies what are the
> >> applications needs to be executed, how to manipulate input output. The
> >> driver program is the workflow for the user. Because the user specifies
> the
> >> program he can program it to handle workflow steering etc. Every time
> the
> >> user wants to execute this program he can tell Airavata to execute the
> >> Driver program.
> >>
> >> I'm also not 100% sure about all the details. But this can be a new way
> >> to look at how systems like Airavata should behave. Your thoughts and
> >> suggestions are more than welcome.
> >>
> >> Thanks,
> >> Supun..
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
> >> shameerainfo@gmail.com> wrote:
> >>
> >>> Hi Supun,
> >>>
> >>> I think we were in two context, because I as suggesting a way to
> >>> serialize
> >>> and deserialize the workflow description while you are suggesting to
> >>> implement
> >>> some kind of workflow interpreter using Python, where Python client can
> >>> send
> >>> thrift calls to Airavata server to run the application. I can see with
> >>> your
> >>> suggested
> >>> approach we can control the workflow execution process from client side
> >>> which
> >>> make it easy to implement workflow debugger.As you mentioned this is a
> >>> major change
> >>> to Airavata. So we should neatly think as this will change our existing
> >>> architecture.
> >>>
> >>> Still if someone need to use different language java, php, JS etc ...
> to
> >>> run the same
> >>> workflow which generated by Python, we need a language independent
> >>> workflow
> >>> description.
> >>> My initial question was what is the best language for this?. But as I
> >>> have
> >>> explained in
> >>> one of my previous reply, It is not matter what language we used Either
> >>> we
> >>> can use
> >>> XML or JSON to write this description, what matters is how easy to
> >>> generate
> >>> workflow with the provided API. Hence it would be great to have set of
> >>> neat
> >>> APIs in
> >>> different languages.
> >>>
> >>> Thanks,
> >>> Shameera.
> >>>
> >>>
> >>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <
> supun06@gmail.com
> >>> >
> >>> wrote:
> >>>
> >>> > Hi Shameera,
> >>> >
> >>> > Using python is a radical approach for workflow I think. But I
> believe
> >>> this
> >>> > is a very beautiful and new approach.Here is a possible scenario for
> >>> > implementation and running using a Python based library.
> >>> >
> >>> > The Python library facilitates the creation of Applications and
> >>> submitting
> >>> > them to Airavata for execution. The underlying functionality is
> exactly
> >>> > similar to what Java clients provides.The only difference is that,
> >>> Python
> >>> > library should have a more fluent API than Java for easy creation of
> >>> > workflows. We can generate the Python clients that talk to Airavata
> >>> server
> >>> > using Thrift.
> >>> >
> >>> > Here is an example off the top of my head to a Python script created
> by
> >>> > user for a Workflow. This is a very crude example and we need to come
> >>> up
> >>> > with a much better API if we are going to go along this path. First
> we
> >>> need
> >>> > to write a Python script that can execute a workflow using Airavata.
> >>> >
> >>> > import airavata
> >>> >
> >>> > host = Host("localhost", ....)
> >>> > app1 = Application(host, ....)
> >>> > app2 = Application(host, ....)
> >>> >
> >>> > # we will connect these applications as a workflow using some
> topology
> >>> > builder or other constructs
> >>> >
> >>> > wb = WorkFlowBuilder()
> >>> > wb.setApp("name1", app1)
> >>> > # you can do a simple output transformation here etc
> >>> >
> >>> > # connects the input of app1 to app2 etc
> >>> > wb.setApp("name2", app2).connectInput("name1")
> >>> >
> >>> > wb.submit()
> >>> >
> >>> > Now we can load this Python script from XBaya. When XBaya loads this
> >>> script
> >>> > the Python script can output an XML configuration of the topology,
> >>> XBaya
> >>> > can render. There are other ways like directly executing the Python
> >>> script
> >>> > from command line and connecting XBaya indirectly as well. Now you
> can
> >>> run
> >>> > the workflow from XBaya. Running the Workflow means just executing
> the
> >>> > Python script.
> >>> >
> >>> > XBaya gets the notifications through messaging and update the UI
> >>> > accordingly.
> >>> >
> >>> > The users need to write the Python script by hand. XBaya cannot
> create
> >>> the
> >>> > script. Because workflow language is an actual python program the
> >>> benefits
> >>> > are immense. For example user can do workflow steering in the
> workflow
> >>> > itself by subscribing to messages from Airavata.
> >>> >
> >>> > Thanks,
> >>> > Supun..
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> >>> > shameerainfo@gmail.com> wrote:
> >>> >
> >>> > > Hi Supun,
> >>> > >
> >>> > > I meant to say JS is a well-known client side scripting language i
> >>> have
> >>> > > messed scripting part. Even we use Python, ultimately we should
> >>> convert
> >>> > > this to java model in server side, somehow we need to serialized
> >>> python
> >>> > > representation to the language which java can parse and generate
> that
> >>> > > model. In this case we need to parse python script in java isn't
> it?
> >>> I am
> >>> > > not exactly clear how you suggesting to use python here. More
> >>> details on
> >>> > > how end system works if we use Python would be great help to
> clearly
> >>> > > understand your points.
> >>> > >
> >>> > > Thanks,
> >>> > > Shameera.
> >>> > >
> >>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> >>> > chris.mattmann@gmail.com>
> >>> > > wrote:
> >>> > >
> >>> > > > Have you guys considered using JCC [1] as a means
> >>> > > > to expose the workflow API currently in Java as a
> >>> > > > Python API?
> >>> > > >
> >>> > > > We are exploring its use in OODT, and we have already
> >>> > > > created a Tika [2] JCC-based python API.
> >>> > > >
> >>> > > > Cheers,
> >>> > > > Chris
> >>> > > >
> >>> > > > [1] http://lucene.apache.org/pylucene/jcc/
> >>> > > > [2] http://github.com/chrismattmann/tika-python/
> >>> > > >
> >>> > > > ------------------------
> >>> > > > Chris Mattmann
> >>> > > > chris.mattmann@gmail.com
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > -----Original Message-----
> >>> > > > From: Supun Kamburugamuva <su...@gmail.com>
> >>> > > > Reply-To: <de...@oodt.apache.org>
> >>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
> >>> > > > To: dev <de...@airavata.apache.org>
> >>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> >>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
> >>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> >>> > > > <de...@oodt.apache.org>
> >>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> >>> > Airavata.
> >>> > > >
> >>> > > > >Once we had an offline discussion about the Airavata Workflow
> >>> language
> >>> > > > >(with Milinda, Saliya and Shameera). In that discussion one
> thing
> >>> came
> >>> > > out
> >>> > > > >was why we need to invent a different language when a simple
> >>> library
> >>> > > like
> >>> > > > >Python will full fill of Airavata requirements.
> >>> > > > >
> >>> > > > >There are many benefits in using a Python library as the API for
> >>> > > > >controlling Airavata workflows.
> >>> > > > >
> >>> > > > >1. It is a library, gives the ultimate control over the
> execution
> >>> and
> >>> > it
> >>> > > > >can be simpler than any domain specific language that we can
> come
> >>> with
> >>> > > > >like
> >>> > > > >XML, JSON etc
> >>> > > > >2. Most people use python and can learn it easily than any
> Domain
> >>> > > specific
> >>> > > > >language
> >>> > > > >3. You can easily create a Python library for Airavata because
> >>> all the
> >>> > > > >APIs
> >>> > > > >are thrift based.
> >>> > > > >4. If you design the constructs correctly you can plug an XBaya.
> >>> > > > >
> >>> > > > >Any thoughts?
> >>> > > > >
> >>> > > > >Thanks,
> >>> > > > >Supun..
> >>> > > > >
> >>> > > > >
> >>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> >>> > supun06@gmail.com
> >>> > > >
> >>> > > > >wrote:
> >>> > > > >
> >>> > > > >> Hi Shameera,
> >>> > > > >>
> >>> > > > >> Why you prefer JavaScript over a language like Python?
> >>> > > > >>
> >>> > > > >> Thanks,
> >>> > > > >> Supun..
> >>> > > > >>
> >>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> >>> > > > >> shameerainfo@gmail.com> wrote:
> >>> > > > >>
> >>> > > > >>> Hi,
> >>> > > > >>>
> >>> > > > >>> First of all thanks everyone for giving valuable inputs.
> After
> >>> > doing
> >>> > > > >>>some
> >>> > > > >>> background search and talking to different people in the
> >>> University
> >>> > > > >>>who has
> >>> > > > >>> used different workflow languages, I myself convinced that
> >>> > > introducing
> >>> > > > >>>an
> >>> > > > >>> another workflow language is not what actually they need. By
> >>> > changing
> >>> > > > >>> exiting workflow language to another will not solve problems.
> >>> What
> >>> > > they
> >>> > > > >>> asking is a easy way to construct the workflows. Indirectly
> >>> what
> >>> > they
> >>> > > > >>> asking for a sort of API which they can use to generate the
> >>> > workflows
> >>> > > > >>>and
> >>> > > > >>> run it. Correct me if i am wrong here.
> >>> > > > >>>
> >>> > > > >>> As most of above replies depict, if we can get a simple API,
> >>> as an
> >>> > > > >>> example, for a web based application, JavaScript API would
> be a
> >>> > good
> >>> > > > >>> solution, and probably JSON would be a good candidate for
> >>> language,
> >>> > > > >>>instead
> >>> > > > >>> of XML.
> >>> > > > >>>
> >>> > > > >>> Airavata community already have started to implement web base
> >>> GUI.
> >>> > > > >>>Hence
> >>> > > > >>> introducing a JSON base JavaScript API would be great help.
> >>> WDYT?
> >>> > > > >>>
> >>> > > > >>> Thanks,
> >>> > > > >>> Shameera.
> >>> > > > >>>
> >>> > > > >>>
> >>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >>> > > > >>> alekny7@gmail.com> wrote:
> >>> > > > >>>
> >>> > > > >>>> Hi,
> >>> > > > >>>>
> >>> > > > >>>> it is not dataflow instead focused on orchestrating REST
> >>> services
> >>> > > but
> >>> > > > >>>> you may find it useful datapoint - we created worfklow
> service
> >>> > that
> >>> > > > >>>>uses
> >>> > > > >>>> natively JavaScript and JSON to describe what happens during
> >>> > > workflow
> >>> > > > >>>> execution:
> >>> > > > >>>>
> >>> > > >
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>> > > > >>>>
> >>> > > > >>>> HTH,
> >>> > > > >>>>
> >>> > > > >>>> Alek
> >>> > > > >>>>
> >>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
> >>> smarru@apache.org>
> >>> > > > >>>>wrote:
> >>> > > > >>>>
> >>> > > > >>>>> Hi Chris,
> >>> > > > >>>>>
> >>> > > > >>>>> Great to hear OODT community will be interested in
> adopting a
> >>> > JSON
> >>> > > > >>>>> based workflow language and potentially a web based
> composer
> >>> as
> >>> > > well.
> >>> > > > >>>>> Airavata previously had BPEL support initially through a
> home
> >>> > grown
> >>> > > > >>>>> implementation [1] by Alek Slominski and later through
> >>> Apache ODE
> >>> > > > >>>>>[2]. Also
> >>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
> >>> read.
> >>> > > > >>>>>
> >>> > > > >>>>> I am of the same opinion that we should adopt something
> more
> >>> > modern
> >>> > > > >>>>>as
> >>> > > > >>>>> the challenges from scientific workflows seems to be
> >>> converging
> >>> > > with
> >>> > > > >>>>>the
> >>> > > > >>>>> data flow patterns in business workflows.
> >>> > > > >>>>>
> >>> > > > >>>>> It will be great if we can all compile a list of potential
> >>> > > candidates
> >>> > > > >>>>> and hack them through.
> >>> > > > >>>>>
> >>> > > > >>>>> Suresh
> >>> > > > >>>>> [1] -
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > >
> >>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>> > > > >>>>> [2] -
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > >
> >>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> >>> > > > >>>>>time_for_eScience_workflows
> >>> > > > >>>>> [3] -
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > >
> >>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> >>> > > > >>>>>6.pdf
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>> > > > >>>>>
> >>> > > > >>>>> > Hi Guys,
> >>> > > > >>>>> >
> >>> > > > >>>>> > I've been interested in this too - we don't per have a
> >>> specific
> >>> > > > >>>>> > OODT workflow language, but we specific workflows using
> >>> XML,
> >>> > and
> >>> > > > >>>>> > other configuration (we are also thinking of moving to
> >>> JSON for
> >>> > > > >>>>> > this).
> >>> > > > >>>>> >
> >>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
> >>> > complex
> >>> > > > >>>>> > to me.
> >>> > > > >>>>> >
> >>> > > > >>>>> > I wonder at the end of the day if we should adopt
> something
> >>> > more
> >>> > > > >>>>> > modern like PIG or some other data flow type of language
> >>> (PIG
> >>> > > > >>>>> > is really neat).
> >>> > > > >>>>> >
> >>> > > > >>>>> > Cheers,
> >>> > > > >>>>> > Chris
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> > > > >>>>> > Chris Mattmann, Ph.D.
> >>> > > > >>>>> > Chief Architect
> >>> > > > >>>>> > Instrument Software and Science Data Systems Section
> (398)
> >>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
> >>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
> >>> > > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
> >>> > > > >>>>> >
> >>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
> >>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089
> >>> USA
> >>> > > > >>>>> >
> >>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> > -----Original Message-----
> >>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
> >>> > > > >>>>> > <ar...@airavata.apache.org>
> >>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>> > > > >>>>> > To: "architecture@airavata.apache.org" <
> >>> > > > >>>>> architecture@airavata.apache.org>,
> >>> > > > >>>>> > dev <de...@airavata.apache.org>
> >>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language
> for
> >>> > > > >>>>>Airavata.
> >>> > > > >>>>> >
> >>> > > > >>>>> >> Hi All,
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> As we all know Airavata has its own workflow language
> call
> >>> > XWF.
> >>> > > > >>>>>When
> >>> > > > >>>>> XWF
> >>> > > > >>>>> >> was introduced, main focus points are interoperability
> and
> >>> > > > >>>>> convertibility.
> >>> > > > >>>>> >> But with years of experience it is convinced that above
> >>> > > > >>>>>requirements
> >>> > > > >>>>> are
> >>> > > > >>>>> >> not really useful when we come to real world use cases.
> >>> And
> >>> > XWF
> >>> > > is
> >>> > > > >>>>> XML
> >>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
> >>> image
> >>> > > it
> >>> > > > >>>>> self.
> >>> > > > >>>>> >> But
> >>> > > > >>>>> >> with the recent changes WSDL part is being removed from
> >>> XWF.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
> >>> languages in
> >>> > > > >>>>> industry
> >>> > > > >>>>> >> and
> >>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
> >>> need
> >>> > to
> >>> > > > >>>>> come up
> >>> > > > >>>>> >> with idea how we should improve Airavata workflow
> >>> language,
> >>> > > either
> >>> > > > >>>>> we can
> >>> > > > >>>>> >> improve existing XWF language, totally change to a new
> >>> > language
> >>> > > > >>>>> available
> >>> > > > >>>>> >> in industry or write a new light weight language. Basic
> >>> > > > >>>>>requirements
> >>> > > > >>>>> that
> >>> > > > >>>>> >> we expect from new improvement are, high usability,
> >>> flexible,
> >>> > > > >>>>>light
> >>> > > > >>>>> weight
> >>> > > > >>>>> >> and real time monitoring support. As you can see above
> >>> > > > >>>>>requirements
> >>> > > > >>>>> are
> >>> > > > >>>>> >> not
> >>> > > > >>>>> >> direct comes with workflow languages but we need
> workflow
> >>> > > language
> >>> > > > >>>>> which
> >>> > > > >>>>> >> help to support above requirements.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> After reading few papers and googling, initially i have
> >>> come
> >>> > up
> >>> > > > >>>>>with
> >>> > > > >>>>> >> following three existing languages,
> >>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>> > > > >>>>> >> 2. WS-BPEL
> >>> > > > >>>>> >> 3. SIDL
> >>> > > > >>>>> >> <
> >>> > > http://computation.llnl.gov/casc/components/index.html#page=home
> >>> > > > >
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific
> >>> domain,
> >>> > > > >>>>> Radical-SAGA
> >>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
> >>> above
> >>> > > > >>>>>three
> >>> > > > >>>>> >> languages we can come up with simple workflow language
> >>> base on
> >>> > > > >>>>> json(or
> >>> > > > >>>>> >> yaml) which support all our requirements for some
> extends.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> It would be grate if I can get more input regarding the
> >>> > $Subject
> >>> > > > >>>>> form the
> >>> > > > >>>>> >> airavata community. You all are more than welcome to
> >>> provide
> >>> > any
> >>> > > > >>>>> type of
> >>> > > > >>>>> >> suggestions.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> Thanks,
> >>> > > > >>>>> >> Shameera.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >>
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> --
> >>> > > > >>>>> >> Best Regards,
> >>> > > > >>>>> >> Shameera Rathnayaka.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT
> gmail.com
> >>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > > >>>>
> >>> > > > >>>>
> >>> > > > >>>> --
> >>> > > > >>>> The best way to predict the future is to invent it - Alan
> Kay
> >>> > > > >>>>
> >>> > > > >>>
> >>> > > > >>>
> >>> > > > >>>
> >>> > > > >>> --
> >>> > > > >>> Best Regards,
> >>> > > > >>> Shameera Rathnayaka.
> >>> > > > >>>
> >>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
> >>> > > > >>>
> >>> > > > >>
> >>> > > > >>
> >>> > > > >>
> >>> > > > >> --
> >>> > > > >> Supun Kamburugamuva
> >>> > > > >> Member, Apache Software Foundation; http://www.apache.org
> >>> > > > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> >>> > > > >> Blog: http://supunk.blogspot.com
> >>> > > > >>
> >>> > > > >>
> >>> > > > >
> >>> > > > >
> >>> > > > >--
> >>> > > > >Supun Kamburugamuva
> >>> > > > >Member, Apache Software Foundation; http://www.apache.org
> >>> > > > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> >>> > > > >Blog: http://supunk.blogspot.com
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > >
> >>> > >
> >>> > > --
> >>> > > Best Regards,
> >>> > > Shameera Rathnayaka.
> >>> > >
> >>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> > > Blog : http://shameerarathnayaka.blogspot.com/
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Supun Kamburugamuva
> >>> > Member, Apache Software Foundation; http://www.apache.org
> >>> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> >>> > Blog: http://supunk.blogspot.com
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards,
> >>> Shameera Rathnayaka.
> >>>
> >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> Blog : http://shameerarathnayaka.blogspot.com/
> >>>
> >>
> >>
> >>
> >> --
> >> Supun Kamburugamuva
> >> Member, Apache Software Foundation; http://www.apache.org
> >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> >> Blog: http://supunk.blogspot.com
> >>
> >>
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi Shameera,
I think the whole point of introducing thrift is to support language
bindings for gateway developers and I do not think we should limit this
freedom by introducing a python client just to build a workflow. I propose
to represent a workflow as a thrift data-structure (I think by using
thrift's list objects we can create an adjacency list and of course it will
be tricky because of thrift unsupport for inheritance and other limitations
but I think it will be worth the effort assuming thrift IDL will be
improved a lot in future). If we can represent a workflow as a thrift
object model in the client side all the workflow related CRUD operations
can be generated using thrift, so we can support all the languages which
thrift support.
To submit a workflow we can introduce another thrift operation which accept
the big workflow thrift object and workflow interpreter can parse this and
execute the workflow. Internally we can use XWF and we can convert thrift
model to XWF and XWF to a thrift object.
Regards
Lahiru
On Mon, Oct 20, 2014 at 10:35 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:
> Hi Amila,
>
> Please see my comments inline.
>
> On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara <thejaka.amila@gmail.com
> > wrote:
>
>> Hello,
>>
>> I am sorry, I am bit late on this thread. But when reading through this
>> thread I simply got lost, what this thread is discussing. I have few
>> questions.
>>
>> 1. @Shameera : Is XWF actually a language to define workflow ? To my
>> understanding it was an intermediate representation to convert workflow
>> defined in UI to java object model. Was XWF ever used by any airavata user
>> to define a workflow graph ?
>>
>
> Yes, XWF is the language defined and used by Airavata to explain the
> workflows but it is not well documented. Both server and client sides read
> this description language to generate runtime java representation. so
> when you used XBaya to create a workflow with multiple applications, under
> the hood XBaya generate xwf which describe that workflow to server.
>
>
>
>>
>> From initial description what I understood is we are looking for a
>> improved "intermediate representation", not a language which describes
>> workflows.
>>
>>
>>
>
>> 2. So what is the exact purpose of this proposed language ?
>> - Is it to hide parallelism in workflows ?
>> - Is it to replace the XBAYA functionalities ? (i hope not)
>>
>
>
> Actually initial idea was to identify good, well-defined and scientific
> domain friendly language. Whole purpose of this effort is reduce the entry
> barrier of the end user. But later it is understood that introducing a new
> language won't fix our issue.
>
>
>>
>> 3. What are we trying to achieve by this proposed language which we
>> cannot achieved through workflow UI tool ?
>>
>> 4. Who is going to use this language ?
>>
>
> As I explained, our direction has been changed.By introducing a new
> language we are get nothing but nice description file. No functional
> improvements etc ... The current language should use all airavata
> client(Currently we have only XBaya) side applications to explain the
> workflow to the server side.
>
>
>
>>
>> 5. Why would a user prefer (assuming intended audience of proposed
>> language is end users) a language over a Work Flow UI tool such as XBAYA ?
>> (In other words what are the things we can do with language which we cannot
>> do with UI)
>>
>
> Let's say you are going to write a new web base client for Airavata which
> generate workflows and launch it. What you need to do is do some magic
> with UI and finally come up with description language and send it to server
> side. Here you need to learn how to write a valid XWF file and write your
> own JS code to generate it. But if airavata has a JavaScript API which can
> be used to generate XWF for you by getting JSON objects as input then it
> would be great help for client side developers.
>
> Thank,
> Shameera.
>
>
>
>>
>> Sorry, if above questions are in-appropriate, just wanted to understand
>> what exactly needed.
>>
>> Thanks
>> -Thejaka Amila
>>
>>
>>
>> On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
>> wrote:
>>
>>> I think I'm not suggesting to create a Workflow interpreter using Python
>>> etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
>>> and move it to a more higher level component. The more I think about it,
>>> the model I'm suggesting is similar to what Hadoop, Storm etc has done for
>>> distributed system computations. This model is proven to be successful over
>>> the years.
>>>
>>> Keeping what Airavata does at its core can help you to build a more
>>> robust system. If we look at Airavata as middleware to execute applications
>>> on computing resources we can simplify what Airavata does and focus on
>>> improving the core functionality. All the successful systems have thrived
>>> on defining what it does at its core and keeping it simple and being
>>> excellent at what it does. In that regard keeping workflow aspect out of
>>> Airavata can help you to focus on the core problem. That is to execute an
>>> application on a remote computing resource in a fault tolerant and scalable
>>> manner.
>>>
>>> What I'm suggesting is to give the Orchestrator the capability to
>>> execute a Driver program that is specified by the user. (This program can
>>> be written in Python, Java or any other language). This driver program is
>>> similar to what you define in a Hadoop or Storm configuration. The driver
>>> program specifies the flow of the computation. It specifies what are the
>>> applications needs to be executed, how to manipulate input output. The
>>> driver program is the workflow for the user. Because the user specifies the
>>> program he can program it to handle workflow steering etc. Every time the
>>> user wants to execute this program he can tell Airavata to execute the
>>> Driver program.
>>>
>>> I'm also not 100% sure about all the details. But this can be a new way
>>> to look at how systems like Airavata should behave. Your thoughts and
>>> suggestions are more than welcome.
>>>
>>> Thanks,
>>> Supun..
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
>>> shameerainfo@gmail.com> wrote:
>>>
>>>> Hi Supun,
>>>>
>>>> I think we were in two context, because I as suggesting a way to
>>>> serialize
>>>> and deserialize the workflow description while you are suggesting to
>>>> implement
>>>> some kind of workflow interpreter using Python, where Python client can
>>>> send
>>>> thrift calls to Airavata server to run the application. I can see with
>>>> your
>>>> suggested
>>>> approach we can control the workflow execution process from client side
>>>> which
>>>> make it easy to implement workflow debugger.As you mentioned this is a
>>>> major change
>>>> to Airavata. So we should neatly think as this will change our existing
>>>> architecture.
>>>>
>>>> Still if someone need to use different language java, php, JS etc ... to
>>>> run the same
>>>> workflow which generated by Python, we need a language independent
>>>> workflow
>>>> description.
>>>> My initial question was what is the best language for this?. But as I
>>>> have
>>>> explained in
>>>> one of my previous reply, It is not matter what language we used Either
>>>> we
>>>> can use
>>>> XML or JSON to write this description, what matters is how easy to
>>>> generate
>>>> workflow with the provided API. Hence it would be great to have set of
>>>> neat
>>>> APIs in
>>>> different languages.
>>>>
>>>> Thanks,
>>>> Shameera.
>>>>
>>>>
>>>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <
>>>> supun06@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi Shameera,
>>>> >
>>>> > Using python is a radical approach for workflow I think. But I
>>>> believe this
>>>> > is a very beautiful and new approach.Here is a possible scenario for
>>>> > implementation and running using a Python based library.
>>>> >
>>>> > The Python library facilitates the creation of Applications and
>>>> submitting
>>>> > them to Airavata for execution. The underlying functionality is
>>>> exactly
>>>> > similar to what Java clients provides.The only difference is that,
>>>> Python
>>>> > library should have a more fluent API than Java for easy creation of
>>>> > workflows. We can generate the Python clients that talk to Airavata
>>>> server
>>>> > using Thrift.
>>>> >
>>>> > Here is an example off the top of my head to a Python script created
>>>> by
>>>> > user for a Workflow. This is a very crude example and we need to come
>>>> up
>>>> > with a much better API if we are going to go along this path. First
>>>> we need
>>>> > to write a Python script that can execute a workflow using Airavata.
>>>> >
>>>> > import airavata
>>>> >
>>>> > host = Host("localhost", ....)
>>>> > app1 = Application(host, ....)
>>>> > app2 = Application(host, ....)
>>>> >
>>>> > # we will connect these applications as a workflow using some topology
>>>> > builder or other constructs
>>>> >
>>>> > wb = WorkFlowBuilder()
>>>> > wb.setApp("name1", app1)
>>>> > # you can do a simple output transformation here etc
>>>> >
>>>> > # connects the input of app1 to app2 etc
>>>> > wb.setApp("name2", app2).connectInput("name1")
>>>> >
>>>> > wb.submit()
>>>> >
>>>> > Now we can load this Python script from XBaya. When XBaya loads this
>>>> script
>>>> > the Python script can output an XML configuration of the topology,
>>>> XBaya
>>>> > can render. There are other ways like directly executing the Python
>>>> script
>>>> > from command line and connecting XBaya indirectly as well. Now you
>>>> can run
>>>> > the workflow from XBaya. Running the Workflow means just executing the
>>>> > Python script.
>>>> >
>>>> > XBaya gets the notifications through messaging and update the UI
>>>> > accordingly.
>>>> >
>>>> > The users need to write the Python script by hand. XBaya cannot
>>>> create the
>>>> > script. Because workflow language is an actual python program the
>>>> benefits
>>>> > are immense. For example user can do workflow steering in the workflow
>>>> > itself by subscribing to messages from Airavata.
>>>> >
>>>> > Thanks,
>>>> > Supun..
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>>>> > shameerainfo@gmail.com> wrote:
>>>> >
>>>> > > Hi Supun,
>>>> > >
>>>> > > I meant to say JS is a well-known client side scripting language i
>>>> have
>>>> > > messed scripting part. Even we use Python, ultimately we should
>>>> convert
>>>> > > this to java model in server side, somehow we need to serialized
>>>> python
>>>> > > representation to the language which java can parse and generate
>>>> that
>>>> > > model. In this case we need to parse python script in java isn't
>>>> it? I am
>>>> > > not exactly clear how you suggesting to use python here. More
>>>> details on
>>>> > > how end system works if we use Python would be great help to clearly
>>>> > > understand your points.
>>>> > >
>>>> > > Thanks,
>>>> > > Shameera.
>>>> > >
>>>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>>>> > chris.mattmann@gmail.com>
>>>> > > wrote:
>>>> > >
>>>> > > > Have you guys considered using JCC [1] as a means
>>>> > > > to expose the workflow API currently in Java as a
>>>> > > > Python API?
>>>> > > >
>>>> > > > We are exploring its use in OODT, and we have already
>>>> > > > created a Tika [2] JCC-based python API.
>>>> > > >
>>>> > > > Cheers,
>>>> > > > Chris
>>>> > > >
>>>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>>>> > > > [2] http://github.com/chrismattmann/tika-python/
>>>> > > >
>>>> > > > ------------------------
>>>> > > > Chris Mattmann
>>>> > > > chris.mattmann@gmail.com
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > > -----Original Message-----
>>>> > > > From: Supun Kamburugamuva <su...@gmail.com>
>>>> > > > Reply-To: <de...@oodt.apache.org>
>>>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>>>> > > > To: dev <de...@airavata.apache.org>
>>>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
>>>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
>>>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
>>>> > > > <de...@oodt.apache.org>
>>>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>>>> > Airavata.
>>>> > > >
>>>> > > > >Once we had an offline discussion about the Airavata Workflow
>>>> language
>>>> > > > >(with Milinda, Saliya and Shameera). In that discussion one
>>>> thing came
>>>> > > out
>>>> > > > >was why we need to invent a different language when a simple
>>>> library
>>>> > > like
>>>> > > > >Python will full fill of Airavata requirements.
>>>> > > > >
>>>> > > > >There are many benefits in using a Python library as the API for
>>>> > > > >controlling Airavata workflows.
>>>> > > > >
>>>> > > > >1. It is a library, gives the ultimate control over the
>>>> execution and
>>>> > it
>>>> > > > >can be simpler than any domain specific language that we can
>>>> come with
>>>> > > > >like
>>>> > > > >XML, JSON etc
>>>> > > > >2. Most people use python and can learn it easily than any Domain
>>>> > > specific
>>>> > > > >language
>>>> > > > >3. You can easily create a Python library for Airavata because
>>>> all the
>>>> > > > >APIs
>>>> > > > >are thrift based.
>>>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>>>> > > > >
>>>> > > > >Any thoughts?
>>>> > > > >
>>>> > > > >Thanks,
>>>> > > > >Supun..
>>>> > > > >
>>>> > > > >
>>>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
>>>> > supun06@gmail.com
>>>> > > >
>>>> > > > >wrote:
>>>> > > > >
>>>> > > > >> Hi Shameera,
>>>> > > > >>
>>>> > > > >> Why you prefer JavaScript over a language like Python?
>>>> > > > >>
>>>> > > > >> Thanks,
>>>> > > > >> Supun..
>>>> > > > >>
>>>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>>>> > > > >> shameerainfo@gmail.com> wrote:
>>>> > > > >>
>>>> > > > >>> Hi,
>>>> > > > >>>
>>>> > > > >>> First of all thanks everyone for giving valuable inputs. After
>>>> > doing
>>>> > > > >>>some
>>>> > > > >>> background search and talking to different people in the
>>>> University
>>>> > > > >>>who has
>>>> > > > >>> used different workflow languages, I myself convinced that
>>>> > > introducing
>>>> > > > >>>an
>>>> > > > >>> another workflow language is not what actually they need. By
>>>> > changing
>>>> > > > >>> exiting workflow language to another will not solve problems.
>>>> What
>>>> > > they
>>>> > > > >>> asking is a easy way to construct the workflows. Indirectly
>>>> what
>>>> > they
>>>> > > > >>> asking for a sort of API which they can use to generate the
>>>> > workflows
>>>> > > > >>>and
>>>> > > > >>> run it. Correct me if i am wrong here.
>>>> > > > >>>
>>>> > > > >>> As most of above replies depict, if we can get a simple API,
>>>> as an
>>>> > > > >>> example, for a web based application, JavaScript API would be
>>>> a
>>>> > good
>>>> > > > >>> solution, and probably JSON would be a good candidate for
>>>> language,
>>>> > > > >>>instead
>>>> > > > >>> of XML.
>>>> > > > >>>
>>>> > > > >>> Airavata community already have started to implement web base
>>>> GUI.
>>>> > > > >>>Hence
>>>> > > > >>> introducing a JSON base JavaScript API would be great help.
>>>> WDYT?
>>>> > > > >>>
>>>> > > > >>> Thanks,
>>>> > > > >>> Shameera.
>>>> > > > >>>
>>>> > > > >>>
>>>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>>> > > > >>> alekny7@gmail.com> wrote:
>>>> > > > >>>
>>>> > > > >>>> Hi,
>>>> > > > >>>>
>>>> > > > >>>> it is not dataflow instead focused on orchestrating REST
>>>> services
>>>> > > but
>>>> > > > >>>> you may find it useful datapoint - we created worfklow
>>>> service
>>>> > that
>>>> > > > >>>>uses
>>>> > > > >>>> natively JavaScript and JSON to describe what happens during
>>>> > > workflow
>>>> > > > >>>> execution:
>>>> > > > >>>>
>>>> > > >
>>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>> > > > >>>>
>>>> > > > >>>> HTH,
>>>> > > > >>>>
>>>> > > > >>>> Alek
>>>> > > > >>>>
>>>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
>>>> smarru@apache.org>
>>>> > > > >>>>wrote:
>>>> > > > >>>>
>>>> > > > >>>>> Hi Chris,
>>>> > > > >>>>>
>>>> > > > >>>>> Great to hear OODT community will be interested in adopting
>>>> a
>>>> > JSON
>>>> > > > >>>>> based workflow language and potentially a web based
>>>> composer as
>>>> > > well.
>>>> > > > >>>>> Airavata previously had BPEL support initially through a
>>>> home
>>>> > grown
>>>> > > > >>>>> implementation [1] by Alek Slominski and later through
>>>> Apache ODE
>>>> > > > >>>>>[2]. Also
>>>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
>>>> read.
>>>> > > > >>>>>
>>>> > > > >>>>> I am of the same opinion that we should adopt something more
>>>> > modern
>>>> > > > >>>>>as
>>>> > > > >>>>> the challenges from scientific workflows seems to be
>>>> converging
>>>> > > with
>>>> > > > >>>>>the
>>>> > > > >>>>> data flow patterns in business workflows.
>>>> > > > >>>>>
>>>> > > > >>>>> It will be great if we can all compile a list of potential
>>>> > > candidates
>>>> > > > >>>>> and hack them through.
>>>> > > > >>>>>
>>>> > > > >>>>> Suresh
>>>> > > > >>>>> [1] -
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > >
>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>> > > > >>>>> [2] -
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > >
>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>>>> > > > >>>>>time_for_eScience_workflows
>>>> > > > >>>>> [3] -
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > >
>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>>>> > > > >>>>>6.pdf
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>> > > > >>>>>
>>>> > > > >>>>> > Hi Guys,
>>>> > > > >>>>> >
>>>> > > > >>>>> > I've been interested in this too - we don't per have a
>>>> specific
>>>> > > > >>>>> > OODT workflow language, but we specific workflows using
>>>> XML,
>>>> > and
>>>> > > > >>>>> > other configuration (we are also thinking of moving to
>>>> JSON for
>>>> > > > >>>>> > this).
>>>> > > > >>>>> >
>>>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
>>>> > complex
>>>> > > > >>>>> > to me.
>>>> > > > >>>>> >
>>>> > > > >>>>> > I wonder at the end of the day if we should adopt
>>>> something
>>>> > more
>>>> > > > >>>>> > modern like PIG or some other data flow type of language
>>>> (PIG
>>>> > > > >>>>> > is really neat).
>>>> > > > >>>>> >
>>>> > > > >>>>> > Cheers,
>>>> > > > >>>>> > Chris
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > > > >>>>> > Chris Mattmann, Ph.D.
>>>> > > > >>>>> > Chief Architect
>>>> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
>>>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
>>>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
>>>> > > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
>>>> > > > >>>>> >
>>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
>>>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089
>>>> USA
>>>> > > > >>>>> >
>>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> > -----Original Message-----
>>>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
>>>> > > > >>>>> > <ar...@airavata.apache.org>
>>>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>> > > > >>>>> > To: "architecture@airavata.apache.org" <
>>>> > > > >>>>> architecture@airavata.apache.org>,
>>>> > > > >>>>> > dev <de...@airavata.apache.org>
>>>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language
>>>> for
>>>> > > > >>>>>Airavata.
>>>> > > > >>>>> >
>>>> > > > >>>>> >> Hi All,
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> As we all know Airavata has its own workflow language
>>>> call
>>>> > XWF.
>>>> > > > >>>>>When
>>>> > > > >>>>> XWF
>>>> > > > >>>>> >> was introduced, main focus points are interoperability
>>>> and
>>>> > > > >>>>> convertibility.
>>>> > > > >>>>> >> But with years of experience it is convinced that above
>>>> > > > >>>>>requirements
>>>> > > > >>>>> are
>>>> > > > >>>>> >> not really useful when we come to real world use cases.
>>>> And
>>>> > XWF
>>>> > > is
>>>> > > > >>>>> XML
>>>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
>>>> image
>>>> > > it
>>>> > > > >>>>> self.
>>>> > > > >>>>> >> But
>>>> > > > >>>>> >> with the recent changes WSDL part is being removed from
>>>> XWF.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
>>>> languages in
>>>> > > > >>>>> industry
>>>> > > > >>>>> >> and
>>>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
>>>> need
>>>> > to
>>>> > > > >>>>> come up
>>>> > > > >>>>> >> with idea how we should improve Airavata workflow
>>>> language,
>>>> > > either
>>>> > > > >>>>> we can
>>>> > > > >>>>> >> improve existing XWF language, totally change to a new
>>>> > language
>>>> > > > >>>>> available
>>>> > > > >>>>> >> in industry or write a new light weight language. Basic
>>>> > > > >>>>>requirements
>>>> > > > >>>>> that
>>>> > > > >>>>> >> we expect from new improvement are, high usability,
>>>> flexible,
>>>> > > > >>>>>light
>>>> > > > >>>>> weight
>>>> > > > >>>>> >> and real time monitoring support. As you can see above
>>>> > > > >>>>>requirements
>>>> > > > >>>>> are
>>>> > > > >>>>> >> not
>>>> > > > >>>>> >> direct comes with workflow languages but we need workflow
>>>> > > language
>>>> > > > >>>>> which
>>>> > > > >>>>> >> help to support above requirements.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> After reading few papers and googling, initially i have
>>>> come
>>>> > up
>>>> > > > >>>>>with
>>>> > > > >>>>> >> following three existing languages,
>>>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>> > > > >>>>> >> 2. WS-BPEL
>>>> > > > >>>>> >> 3. SIDL
>>>> > > > >>>>> >> <
>>>> > > http://computation.llnl.gov/casc/components/index.html#page=home
>>>> > > > >
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific
>>>> domain,
>>>> > > > >>>>> Radical-SAGA
>>>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
>>>> above
>>>> > > > >>>>>three
>>>> > > > >>>>> >> languages we can come up with simple workflow language
>>>> base on
>>>> > > > >>>>> json(or
>>>> > > > >>>>> >> yaml) which support all our requirements for some
>>>> extends.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> It would be grate if I can get more input regarding the
>>>> > $Subject
>>>> > > > >>>>> form the
>>>> > > > >>>>> >> airavata community. You all are more than welcome to
>>>> provide
>>>> > any
>>>> > > > >>>>> type of
>>>> > > > >>>>> >> suggestions.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> Thanks,
>>>> > > > >>>>> >> Shameera.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >>
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> --
>>>> > > > >>>>> >> Best Regards,
>>>> > > > >>>>> >> Shameera Rathnayaka.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT
>>>> gmail.com
>>>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > > >>>>
>>>> > > > >>>>
>>>> > > > >>>> --
>>>> > > > >>>> The best way to predict the future is to invent it - Alan Kay
>>>> > > > >>>>
>>>> > > > >>>
>>>> > > > >>>
>>>> > > > >>>
>>>> > > > >>> --
>>>> > > > >>> Best Regards,
>>>> > > > >>> Shameera Rathnayaka.
>>>> > > > >>>
>>>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
>>>> > > > >>>
>>>> > > > >>
>>>> > > > >>
>>>> > > > >>
>>>> > > > >> --
>>>> > > > >> Supun Kamburugamuva
>>>> > > > >> Member, Apache Software Foundation; http://www.apache.org
>>>> > > > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>>> > > > >> Blog: http://supunk.blogspot.com
>>>> > > > >>
>>>> > > > >>
>>>> > > > >
>>>> > > > >
>>>> > > > >--
>>>> > > > >Supun Kamburugamuva
>>>> > > > >Member, Apache Software Foundation; http://www.apache.org
>>>> > > > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>>> > > > >Blog: http://supunk.blogspot.com
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > > Best Regards,
>>>> > > Shameera Rathnayaka.
>>>> > >
>>>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> > > Blog : http://shameerarathnayaka.blogspot.com/
>>>> > >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Supun Kamburugamuva
>>>> > Member, Apache Software Foundation; http://www.apache.org
>>>> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>>> > Blog: http://supunk.blogspot.com
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Shameera Rathnayaka.
>>>>
>>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>>
>>>
>>>
>>>
>>> --
>>> Supun Kamburugamuva
>>> Member, Apache Software Foundation; http://www.apache.org
>>> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>> Blog: http://supunk.blogspot.com
>>>
>>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
--
Research Assistant
Science Gateways Group
Indiana University
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Amila Jayasekara <th...@gmail.com>.
Hi Shameera,
Thanks for the reply.
I wouldnt call XWF a "language". It is not a language in any sense but an
intermediate representation of a workflow graph.
To my understanding what you need is not a language but an efficient
portable intermediate representation. Seems you also have realized it. JSON
might be a good candidate provided most of the frontends are web based.
Thanks
-Thejaka
On Mon, Oct 20, 2014 at 10:35 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:
> Hi Amila,
>
> Please see my comments inline.
>
> On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara <thejaka.amila@gmail.com
> >
> wrote:
>
> > Hello,
> >
> > I am sorry, I am bit late on this thread. But when reading through this
> > thread I simply got lost, what this thread is discussing. I have few
> > questions.
> >
> > 1. @Shameera : Is XWF actually a language to define workflow ? To my
> > understanding it was an intermediate representation to convert workflow
> > defined in UI to java object model. Was XWF ever used by any airavata
> user
> > to define a workflow graph ?
> >
>
> Yes, XWF is the language defined and used by Airavata to explain the
> workflows but it is not well documented. Both server and client sides read
> this description language to generate runtime java representation. so
> when you used XBaya to create a workflow with multiple applications, under
> the hood XBaya generate xwf which describe that workflow to server.
>
>
>
> >
> > From initial description what I understood is we are looking for a
> > improved "intermediate representation", not a language which describes
> > workflows.
> >
> >
> >
>
> > 2. So what is the exact purpose of this proposed language ?
> > - Is it to hide parallelism in workflows ?
> > - Is it to replace the XBAYA functionalities ? (i hope not)
> >
>
>
> Actually initial idea was to identify good, well-defined and scientific
> domain friendly language. Whole purpose of this effort is reduce the entry
> barrier of the end user. But later it is understood that introducing a new
> language won't fix our issue.
>
>
> >
> > 3. What are we trying to achieve by this proposed language which we
> cannot
> > achieved through workflow UI tool ?
> >
> > 4. Who is going to use this language ?
> >
>
> As I explained, our direction has been changed.By introducing a new
> language we are get nothing but nice description file. No functional
> improvements etc ... The current language should use all airavata
> client(Currently we have only XBaya) side applications to explain the
> workflow to the server side.
>
>
>
> >
> > 5. Why would a user prefer (assuming intended audience of proposed
> > language is end users) a language over a Work Flow UI tool such as XBAYA
> ?
> > (In other words what are the things we can do with language which we
> cannot
> > do with UI)
> >
>
> Let's say you are going to write a new web base client for Airavata which
> generate workflows and launch it. What you need to do is do some magic
> with UI and finally come up with description language and send it to server
> side. Here you need to learn how to write a valid XWF file and write your
> own JS code to generate it. But if airavata has a JavaScript API which can
> be used to generate XWF for you by getting JSON objects as input then it
> would be great help for client side developers.
>
> Thank,
> Shameera.
>
>
>
> >
> > Sorry, if above questions are in-appropriate, just wanted to understand
> > what exactly needed.
> >
> > Thanks
> > -Thejaka Amila
> >
> >
> >
> > On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
> > wrote:
> >
> >> I think I'm not suggesting to create a Workflow interpreter using Python
> >> etc. What I'm suggesting is to remove the Worflow aspect from core
> Airavata
> >> and move it to a more higher level component. The more I think about it,
> >> the model I'm suggesting is similar to what Hadoop, Storm etc has done
> for
> >> distributed system computations. This model is proven to be successful
> over
> >> the years.
> >>
> >> Keeping what Airavata does at its core can help you to build a more
> >> robust system. If we look at Airavata as middleware to execute
> applications
> >> on computing resources we can simplify what Airavata does and focus on
> >> improving the core functionality. All the successful systems have
> thrived
> >> on defining what it does at its core and keeping it simple and being
> >> excellent at what it does. In that regard keeping workflow aspect out of
> >> Airavata can help you to focus on the core problem. That is to execute
> an
> >> application on a remote computing resource in a fault tolerant and
> scalable
> >> manner.
> >>
> >> What I'm suggesting is to give the Orchestrator the capability to
> execute
> >> a Driver program that is specified by the user. (This program can be
> >> written in Python, Java or any other language). This driver program is
> >> similar to what you define in a Hadoop or Storm configuration. The
> driver
> >> program specifies the flow of the computation. It specifies what are the
> >> applications needs to be executed, how to manipulate input output. The
> >> driver program is the workflow for the user. Because the user specifies
> the
> >> program he can program it to handle workflow steering etc. Every time
> the
> >> user wants to execute this program he can tell Airavata to execute the
> >> Driver program.
> >>
> >> I'm also not 100% sure about all the details. But this can be a new way
> >> to look at how systems like Airavata should behave. Your thoughts and
> >> suggestions are more than welcome.
> >>
> >> Thanks,
> >> Supun..
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
> >> shameerainfo@gmail.com> wrote:
> >>
> >>> Hi Supun,
> >>>
> >>> I think we were in two context, because I as suggesting a way to
> >>> serialize
> >>> and deserialize the workflow description while you are suggesting to
> >>> implement
> >>> some kind of workflow interpreter using Python, where Python client can
> >>> send
> >>> thrift calls to Airavata server to run the application. I can see with
> >>> your
> >>> suggested
> >>> approach we can control the workflow execution process from client side
> >>> which
> >>> make it easy to implement workflow debugger.As you mentioned this is a
> >>> major change
> >>> to Airavata. So we should neatly think as this will change our existing
> >>> architecture.
> >>>
> >>> Still if someone need to use different language java, php, JS etc ...
> to
> >>> run the same
> >>> workflow which generated by Python, we need a language independent
> >>> workflow
> >>> description.
> >>> My initial question was what is the best language for this?. But as I
> >>> have
> >>> explained in
> >>> one of my previous reply, It is not matter what language we used Either
> >>> we
> >>> can use
> >>> XML or JSON to write this description, what matters is how easy to
> >>> generate
> >>> workflow with the provided API. Hence it would be great to have set of
> >>> neat
> >>> APIs in
> >>> different languages.
> >>>
> >>> Thanks,
> >>> Shameera.
> >>>
> >>>
> >>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <
> supun06@gmail.com
> >>> >
> >>> wrote:
> >>>
> >>> > Hi Shameera,
> >>> >
> >>> > Using python is a radical approach for workflow I think. But I
> believe
> >>> this
> >>> > is a very beautiful and new approach.Here is a possible scenario for
> >>> > implementation and running using a Python based library.
> >>> >
> >>> > The Python library facilitates the creation of Applications and
> >>> submitting
> >>> > them to Airavata for execution. The underlying functionality is
> exactly
> >>> > similar to what Java clients provides.The only difference is that,
> >>> Python
> >>> > library should have a more fluent API than Java for easy creation of
> >>> > workflows. We can generate the Python clients that talk to Airavata
> >>> server
> >>> > using Thrift.
> >>> >
> >>> > Here is an example off the top of my head to a Python script created
> by
> >>> > user for a Workflow. This is a very crude example and we need to come
> >>> up
> >>> > with a much better API if we are going to go along this path. First
> we
> >>> need
> >>> > to write a Python script that can execute a workflow using Airavata.
> >>> >
> >>> > import airavata
> >>> >
> >>> > host = Host("localhost", ....)
> >>> > app1 = Application(host, ....)
> >>> > app2 = Application(host, ....)
> >>> >
> >>> > # we will connect these applications as a workflow using some
> topology
> >>> > builder or other constructs
> >>> >
> >>> > wb = WorkFlowBuilder()
> >>> > wb.setApp("name1", app1)
> >>> > # you can do a simple output transformation here etc
> >>> >
> >>> > # connects the input of app1 to app2 etc
> >>> > wb.setApp("name2", app2).connectInput("name1")
> >>> >
> >>> > wb.submit()
> >>> >
> >>> > Now we can load this Python script from XBaya. When XBaya loads this
> >>> script
> >>> > the Python script can output an XML configuration of the topology,
> >>> XBaya
> >>> > can render. There are other ways like directly executing the Python
> >>> script
> >>> > from command line and connecting XBaya indirectly as well. Now you
> can
> >>> run
> >>> > the workflow from XBaya. Running the Workflow means just executing
> the
> >>> > Python script.
> >>> >
> >>> > XBaya gets the notifications through messaging and update the UI
> >>> > accordingly.
> >>> >
> >>> > The users need to write the Python script by hand. XBaya cannot
> create
> >>> the
> >>> > script. Because workflow language is an actual python program the
> >>> benefits
> >>> > are immense. For example user can do workflow steering in the
> workflow
> >>> > itself by subscribing to messages from Airavata.
> >>> >
> >>> > Thanks,
> >>> > Supun..
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> >>> > shameerainfo@gmail.com> wrote:
> >>> >
> >>> > > Hi Supun,
> >>> > >
> >>> > > I meant to say JS is a well-known client side scripting language i
> >>> have
> >>> > > messed scripting part. Even we use Python, ultimately we should
> >>> convert
> >>> > > this to java model in server side, somehow we need to serialized
> >>> python
> >>> > > representation to the language which java can parse and generate
> that
> >>> > > model. In this case we need to parse python script in java isn't
> it?
> >>> I am
> >>> > > not exactly clear how you suggesting to use python here. More
> >>> details on
> >>> > > how end system works if we use Python would be great help to
> clearly
> >>> > > understand your points.
> >>> > >
> >>> > > Thanks,
> >>> > > Shameera.
> >>> > >
> >>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> >>> > chris.mattmann@gmail.com>
> >>> > > wrote:
> >>> > >
> >>> > > > Have you guys considered using JCC [1] as a means
> >>> > > > to expose the workflow API currently in Java as a
> >>> > > > Python API?
> >>> > > >
> >>> > > > We are exploring its use in OODT, and we have already
> >>> > > > created a Tika [2] JCC-based python API.
> >>> > > >
> >>> > > > Cheers,
> >>> > > > Chris
> >>> > > >
> >>> > > > [1] http://lucene.apache.org/pylucene/jcc/
> >>> > > > [2] http://github.com/chrismattmann/tika-python/
> >>> > > >
> >>> > > > ------------------------
> >>> > > > Chris Mattmann
> >>> > > > chris.mattmann@gmail.com
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > -----Original Message-----
> >>> > > > From: Supun Kamburugamuva <su...@gmail.com>
> >>> > > > Reply-To: <de...@oodt.apache.org>
> >>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
> >>> > > > To: dev <de...@airavata.apache.org>
> >>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> >>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
> >>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> >>> > > > <de...@oodt.apache.org>
> >>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> >>> > Airavata.
> >>> > > >
> >>> > > > >Once we had an offline discussion about the Airavata Workflow
> >>> language
> >>> > > > >(with Milinda, Saliya and Shameera). In that discussion one
> thing
> >>> came
> >>> > > out
> >>> > > > >was why we need to invent a different language when a simple
> >>> library
> >>> > > like
> >>> > > > >Python will full fill of Airavata requirements.
> >>> > > > >
> >>> > > > >There are many benefits in using a Python library as the API for
> >>> > > > >controlling Airavata workflows.
> >>> > > > >
> >>> > > > >1. It is a library, gives the ultimate control over the
> execution
> >>> and
> >>> > it
> >>> > > > >can be simpler than any domain specific language that we can
> come
> >>> with
> >>> > > > >like
> >>> > > > >XML, JSON etc
> >>> > > > >2. Most people use python and can learn it easily than any
> Domain
> >>> > > specific
> >>> > > > >language
> >>> > > > >3. You can easily create a Python library for Airavata because
> >>> all the
> >>> > > > >APIs
> >>> > > > >are thrift based.
> >>> > > > >4. If you design the constructs correctly you can plug an XBaya.
> >>> > > > >
> >>> > > > >Any thoughts?
> >>> > > > >
> >>> > > > >Thanks,
> >>> > > > >Supun..
> >>> > > > >
> >>> > > > >
> >>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> >>> > supun06@gmail.com
> >>> > > >
> >>> > > > >wrote:
> >>> > > > >
> >>> > > > >> Hi Shameera,
> >>> > > > >>
> >>> > > > >> Why you prefer JavaScript over a language like Python?
> >>> > > > >>
> >>> > > > >> Thanks,
> >>> > > > >> Supun..
> >>> > > > >>
> >>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> >>> > > > >> shameerainfo@gmail.com> wrote:
> >>> > > > >>
> >>> > > > >>> Hi,
> >>> > > > >>>
> >>> > > > >>> First of all thanks everyone for giving valuable inputs.
> After
> >>> > doing
> >>> > > > >>>some
> >>> > > > >>> background search and talking to different people in the
> >>> University
> >>> > > > >>>who has
> >>> > > > >>> used different workflow languages, I myself convinced that
> >>> > > introducing
> >>> > > > >>>an
> >>> > > > >>> another workflow language is not what actually they need. By
> >>> > changing
> >>> > > > >>> exiting workflow language to another will not solve problems.
> >>> What
> >>> > > they
> >>> > > > >>> asking is a easy way to construct the workflows. Indirectly
> >>> what
> >>> > they
> >>> > > > >>> asking for a sort of API which they can use to generate the
> >>> > workflows
> >>> > > > >>>and
> >>> > > > >>> run it. Correct me if i am wrong here.
> >>> > > > >>>
> >>> > > > >>> As most of above replies depict, if we can get a simple API,
> >>> as an
> >>> > > > >>> example, for a web based application, JavaScript API would
> be a
> >>> > good
> >>> > > > >>> solution, and probably JSON would be a good candidate for
> >>> language,
> >>> > > > >>>instead
> >>> > > > >>> of XML.
> >>> > > > >>>
> >>> > > > >>> Airavata community already have started to implement web base
> >>> GUI.
> >>> > > > >>>Hence
> >>> > > > >>> introducing a JSON base JavaScript API would be great help.
> >>> WDYT?
> >>> > > > >>>
> >>> > > > >>> Thanks,
> >>> > > > >>> Shameera.
> >>> > > > >>>
> >>> > > > >>>
> >>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >>> > > > >>> alekny7@gmail.com> wrote:
> >>> > > > >>>
> >>> > > > >>>> Hi,
> >>> > > > >>>>
> >>> > > > >>>> it is not dataflow instead focused on orchestrating REST
> >>> services
> >>> > > but
> >>> > > > >>>> you may find it useful datapoint - we created worfklow
> service
> >>> > that
> >>> > > > >>>>uses
> >>> > > > >>>> natively JavaScript and JSON to describe what happens during
> >>> > > workflow
> >>> > > > >>>> execution:
> >>> > > > >>>>
> >>> > > >
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>> > > > >>>>
> >>> > > > >>>> HTH,
> >>> > > > >>>>
> >>> > > > >>>> Alek
> >>> > > > >>>>
> >>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
> >>> smarru@apache.org>
> >>> > > > >>>>wrote:
> >>> > > > >>>>
> >>> > > > >>>>> Hi Chris,
> >>> > > > >>>>>
> >>> > > > >>>>> Great to hear OODT community will be interested in
> adopting a
> >>> > JSON
> >>> > > > >>>>> based workflow language and potentially a web based
> composer
> >>> as
> >>> > > well.
> >>> > > > >>>>> Airavata previously had BPEL support initially through a
> home
> >>> > grown
> >>> > > > >>>>> implementation [1] by Alek Slominski and later through
> >>> Apache ODE
> >>> > > > >>>>>[2]. Also
> >>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
> >>> read.
> >>> > > > >>>>>
> >>> > > > >>>>> I am of the same opinion that we should adopt something
> more
> >>> > modern
> >>> > > > >>>>>as
> >>> > > > >>>>> the challenges from scientific workflows seems to be
> >>> converging
> >>> > > with
> >>> > > > >>>>>the
> >>> > > > >>>>> data flow patterns in business workflows.
> >>> > > > >>>>>
> >>> > > > >>>>> It will be great if we can all compile a list of potential
> >>> > > candidates
> >>> > > > >>>>> and hack them through.
> >>> > > > >>>>>
> >>> > > > >>>>> Suresh
> >>> > > > >>>>> [1] -
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > >
> >>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>> > > > >>>>> [2] -
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > >
> >>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> >>> > > > >>>>>time_for_eScience_workflows
> >>> > > > >>>>> [3] -
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > >
> >>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> >>> > > > >>>>>6.pdf
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>> > > > >>>>>
> >>> > > > >>>>> > Hi Guys,
> >>> > > > >>>>> >
> >>> > > > >>>>> > I've been interested in this too - we don't per have a
> >>> specific
> >>> > > > >>>>> > OODT workflow language, but we specific workflows using
> >>> XML,
> >>> > and
> >>> > > > >>>>> > other configuration (we are also thinking of moving to
> >>> JSON for
> >>> > > > >>>>> > this).
> >>> > > > >>>>> >
> >>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
> >>> > complex
> >>> > > > >>>>> > to me.
> >>> > > > >>>>> >
> >>> > > > >>>>> > I wonder at the end of the day if we should adopt
> something
> >>> > more
> >>> > > > >>>>> > modern like PIG or some other data flow type of language
> >>> (PIG
> >>> > > > >>>>> > is really neat).
> >>> > > > >>>>> >
> >>> > > > >>>>> > Cheers,
> >>> > > > >>>>> > Chris
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> > > > >>>>> > Chris Mattmann, Ph.D.
> >>> > > > >>>>> > Chief Architect
> >>> > > > >>>>> > Instrument Software and Science Data Systems Section
> (398)
> >>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
> >>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
> >>> > > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
> >>> > > > >>>>> >
> >>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
> >>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089
> >>> USA
> >>> > > > >>>>> >
> >>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> > -----Original Message-----
> >>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
> >>> > > > >>>>> > <ar...@airavata.apache.org>
> >>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>> > > > >>>>> > To: "architecture@airavata.apache.org" <
> >>> > > > >>>>> architecture@airavata.apache.org>,
> >>> > > > >>>>> > dev <de...@airavata.apache.org>
> >>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language
> for
> >>> > > > >>>>>Airavata.
> >>> > > > >>>>> >
> >>> > > > >>>>> >> Hi All,
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> As we all know Airavata has its own workflow language
> call
> >>> > XWF.
> >>> > > > >>>>>When
> >>> > > > >>>>> XWF
> >>> > > > >>>>> >> was introduced, main focus points are interoperability
> and
> >>> > > > >>>>> convertibility.
> >>> > > > >>>>> >> But with years of experience it is convinced that above
> >>> > > > >>>>>requirements
> >>> > > > >>>>> are
> >>> > > > >>>>> >> not really useful when we come to real world use cases.
> >>> And
> >>> > XWF
> >>> > > is
> >>> > > > >>>>> XML
> >>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
> >>> image
> >>> > > it
> >>> > > > >>>>> self.
> >>> > > > >>>>> >> But
> >>> > > > >>>>> >> with the recent changes WSDL part is being removed from
> >>> XWF.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
> >>> languages in
> >>> > > > >>>>> industry
> >>> > > > >>>>> >> and
> >>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
> >>> need
> >>> > to
> >>> > > > >>>>> come up
> >>> > > > >>>>> >> with idea how we should improve Airavata workflow
> >>> language,
> >>> > > either
> >>> > > > >>>>> we can
> >>> > > > >>>>> >> improve existing XWF language, totally change to a new
> >>> > language
> >>> > > > >>>>> available
> >>> > > > >>>>> >> in industry or write a new light weight language. Basic
> >>> > > > >>>>>requirements
> >>> > > > >>>>> that
> >>> > > > >>>>> >> we expect from new improvement are, high usability,
> >>> flexible,
> >>> > > > >>>>>light
> >>> > > > >>>>> weight
> >>> > > > >>>>> >> and real time monitoring support. As you can see above
> >>> > > > >>>>>requirements
> >>> > > > >>>>> are
> >>> > > > >>>>> >> not
> >>> > > > >>>>> >> direct comes with workflow languages but we need
> workflow
> >>> > > language
> >>> > > > >>>>> which
> >>> > > > >>>>> >> help to support above requirements.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> After reading few papers and googling, initially i have
> >>> come
> >>> > up
> >>> > > > >>>>>with
> >>> > > > >>>>> >> following three existing languages,
> >>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>> > > > >>>>> >> 2. WS-BPEL
> >>> > > > >>>>> >> 3. SIDL
> >>> > > > >>>>> >> <
> >>> > > http://computation.llnl.gov/casc/components/index.html#page=home
> >>> > > > >
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific
> >>> domain,
> >>> > > > >>>>> Radical-SAGA
> >>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
> >>> above
> >>> > > > >>>>>three
> >>> > > > >>>>> >> languages we can come up with simple workflow language
> >>> base on
> >>> > > > >>>>> json(or
> >>> > > > >>>>> >> yaml) which support all our requirements for some
> extends.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> It would be grate if I can get more input regarding the
> >>> > $Subject
> >>> > > > >>>>> form the
> >>> > > > >>>>> >> airavata community. You all are more than welcome to
> >>> provide
> >>> > any
> >>> > > > >>>>> type of
> >>> > > > >>>>> >> suggestions.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> Thanks,
> >>> > > > >>>>> >> Shameera.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >>
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> --
> >>> > > > >>>>> >> Best Regards,
> >>> > > > >>>>> >> Shameera Rathnayaka.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT
> gmail.com
> >>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > > >>>>
> >>> > > > >>>>
> >>> > > > >>>> --
> >>> > > > >>>> The best way to predict the future is to invent it - Alan
> Kay
> >>> > > > >>>>
> >>> > > > >>>
> >>> > > > >>>
> >>> > > > >>>
> >>> > > > >>> --
> >>> > > > >>> Best Regards,
> >>> > > > >>> Shameera Rathnayaka.
> >>> > > > >>>
> >>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
> >>> > > > >>>
> >>> > > > >>
> >>> > > > >>
> >>> > > > >>
> >>> > > > >> --
> >>> > > > >> Supun Kamburugamuva
> >>> > > > >> Member, Apache Software Foundation; http://www.apache.org
> >>> > > > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> >>> > > > >> Blog: http://supunk.blogspot.com
> >>> > > > >>
> >>> > > > >>
> >>> > > > >
> >>> > > > >
> >>> > > > >--
> >>> > > > >Supun Kamburugamuva
> >>> > > > >Member, Apache Software Foundation; http://www.apache.org
> >>> > > > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> >>> > > > >Blog: http://supunk.blogspot.com
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > >
> >>> > >
> >>> > > --
> >>> > > Best Regards,
> >>> > > Shameera Rathnayaka.
> >>> > >
> >>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> > > Blog : http://shameerarathnayaka.blogspot.com/
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Supun Kamburugamuva
> >>> > Member, Apache Software Foundation; http://www.apache.org
> >>> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> >>> > Blog: http://supunk.blogspot.com
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards,
> >>> Shameera Rathnayaka.
> >>>
> >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> Blog : http://shameerarathnayaka.blogspot.com/
> >>>
> >>
> >>
> >>
> >> --
> >> Supun Kamburugamuva
> >> Member, Apache Software Foundation; http://www.apache.org
> >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> >> Blog: http://supunk.blogspot.com
> >>
> >>
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi Shameera,
I think the whole point of introducing thrift is to support language
bindings for gateway developers and I do not think we should limit this
freedom by introducing a python client just to build a workflow. I propose
to represent a workflow as a thrift data-structure (I think by using
thrift's list objects we can create an adjacency list and of course it will
be tricky because of thrift unsupport for inheritance and other limitations
but I think it will be worth the effort assuming thrift IDL will be
improved a lot in future). If we can represent a workflow as a thrift
object model in the client side all the workflow related CRUD operations
can be generated using thrift, so we can support all the languages which
thrift support.
To submit a workflow we can introduce another thrift operation which accept
the big workflow thrift object and workflow interpreter can parse this and
execute the workflow. Internally we can use XWF and we can convert thrift
model to XWF and XWF to a thrift object.
Regards
Lahiru
On Mon, Oct 20, 2014 at 10:35 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:
> Hi Amila,
>
> Please see my comments inline.
>
> On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara <thejaka.amila@gmail.com
> > wrote:
>
>> Hello,
>>
>> I am sorry, I am bit late on this thread. But when reading through this
>> thread I simply got lost, what this thread is discussing. I have few
>> questions.
>>
>> 1. @Shameera : Is XWF actually a language to define workflow ? To my
>> understanding it was an intermediate representation to convert workflow
>> defined in UI to java object model. Was XWF ever used by any airavata user
>> to define a workflow graph ?
>>
>
> Yes, XWF is the language defined and used by Airavata to explain the
> workflows but it is not well documented. Both server and client sides read
> this description language to generate runtime java representation. so
> when you used XBaya to create a workflow with multiple applications, under
> the hood XBaya generate xwf which describe that workflow to server.
>
>
>
>>
>> From initial description what I understood is we are looking for a
>> improved "intermediate representation", not a language which describes
>> workflows.
>>
>>
>>
>
>> 2. So what is the exact purpose of this proposed language ?
>> - Is it to hide parallelism in workflows ?
>> - Is it to replace the XBAYA functionalities ? (i hope not)
>>
>
>
> Actually initial idea was to identify good, well-defined and scientific
> domain friendly language. Whole purpose of this effort is reduce the entry
> barrier of the end user. But later it is understood that introducing a new
> language won't fix our issue.
>
>
>>
>> 3. What are we trying to achieve by this proposed language which we
>> cannot achieved through workflow UI tool ?
>>
>> 4. Who is going to use this language ?
>>
>
> As I explained, our direction has been changed.By introducing a new
> language we are get nothing but nice description file. No functional
> improvements etc ... The current language should use all airavata
> client(Currently we have only XBaya) side applications to explain the
> workflow to the server side.
>
>
>
>>
>> 5. Why would a user prefer (assuming intended audience of proposed
>> language is end users) a language over a Work Flow UI tool such as XBAYA ?
>> (In other words what are the things we can do with language which we cannot
>> do with UI)
>>
>
> Let's say you are going to write a new web base client for Airavata which
> generate workflows and launch it. What you need to do is do some magic
> with UI and finally come up with description language and send it to server
> side. Here you need to learn how to write a valid XWF file and write your
> own JS code to generate it. But if airavata has a JavaScript API which can
> be used to generate XWF for you by getting JSON objects as input then it
> would be great help for client side developers.
>
> Thank,
> Shameera.
>
>
>
>>
>> Sorry, if above questions are in-appropriate, just wanted to understand
>> what exactly needed.
>>
>> Thanks
>> -Thejaka Amila
>>
>>
>>
>> On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
>> wrote:
>>
>>> I think I'm not suggesting to create a Workflow interpreter using Python
>>> etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
>>> and move it to a more higher level component. The more I think about it,
>>> the model I'm suggesting is similar to what Hadoop, Storm etc has done for
>>> distributed system computations. This model is proven to be successful over
>>> the years.
>>>
>>> Keeping what Airavata does at its core can help you to build a more
>>> robust system. If we look at Airavata as middleware to execute applications
>>> on computing resources we can simplify what Airavata does and focus on
>>> improving the core functionality. All the successful systems have thrived
>>> on defining what it does at its core and keeping it simple and being
>>> excellent at what it does. In that regard keeping workflow aspect out of
>>> Airavata can help you to focus on the core problem. That is to execute an
>>> application on a remote computing resource in a fault tolerant and scalable
>>> manner.
>>>
>>> What I'm suggesting is to give the Orchestrator the capability to
>>> execute a Driver program that is specified by the user. (This program can
>>> be written in Python, Java or any other language). This driver program is
>>> similar to what you define in a Hadoop or Storm configuration. The driver
>>> program specifies the flow of the computation. It specifies what are the
>>> applications needs to be executed, how to manipulate input output. The
>>> driver program is the workflow for the user. Because the user specifies the
>>> program he can program it to handle workflow steering etc. Every time the
>>> user wants to execute this program he can tell Airavata to execute the
>>> Driver program.
>>>
>>> I'm also not 100% sure about all the details. But this can be a new way
>>> to look at how systems like Airavata should behave. Your thoughts and
>>> suggestions are more than welcome.
>>>
>>> Thanks,
>>> Supun..
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
>>> shameerainfo@gmail.com> wrote:
>>>
>>>> Hi Supun,
>>>>
>>>> I think we were in two context, because I as suggesting a way to
>>>> serialize
>>>> and deserialize the workflow description while you are suggesting to
>>>> implement
>>>> some kind of workflow interpreter using Python, where Python client can
>>>> send
>>>> thrift calls to Airavata server to run the application. I can see with
>>>> your
>>>> suggested
>>>> approach we can control the workflow execution process from client side
>>>> which
>>>> make it easy to implement workflow debugger.As you mentioned this is a
>>>> major change
>>>> to Airavata. So we should neatly think as this will change our existing
>>>> architecture.
>>>>
>>>> Still if someone need to use different language java, php, JS etc ... to
>>>> run the same
>>>> workflow which generated by Python, we need a language independent
>>>> workflow
>>>> description.
>>>> My initial question was what is the best language for this?. But as I
>>>> have
>>>> explained in
>>>> one of my previous reply, It is not matter what language we used Either
>>>> we
>>>> can use
>>>> XML or JSON to write this description, what matters is how easy to
>>>> generate
>>>> workflow with the provided API. Hence it would be great to have set of
>>>> neat
>>>> APIs in
>>>> different languages.
>>>>
>>>> Thanks,
>>>> Shameera.
>>>>
>>>>
>>>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <
>>>> supun06@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi Shameera,
>>>> >
>>>> > Using python is a radical approach for workflow I think. But I
>>>> believe this
>>>> > is a very beautiful and new approach.Here is a possible scenario for
>>>> > implementation and running using a Python based library.
>>>> >
>>>> > The Python library facilitates the creation of Applications and
>>>> submitting
>>>> > them to Airavata for execution. The underlying functionality is
>>>> exactly
>>>> > similar to what Java clients provides.The only difference is that,
>>>> Python
>>>> > library should have a more fluent API than Java for easy creation of
>>>> > workflows. We can generate the Python clients that talk to Airavata
>>>> server
>>>> > using Thrift.
>>>> >
>>>> > Here is an example off the top of my head to a Python script created
>>>> by
>>>> > user for a Workflow. This is a very crude example and we need to come
>>>> up
>>>> > with a much better API if we are going to go along this path. First
>>>> we need
>>>> > to write a Python script that can execute a workflow using Airavata.
>>>> >
>>>> > import airavata
>>>> >
>>>> > host = Host("localhost", ....)
>>>> > app1 = Application(host, ....)
>>>> > app2 = Application(host, ....)
>>>> >
>>>> > # we will connect these applications as a workflow using some topology
>>>> > builder or other constructs
>>>> >
>>>> > wb = WorkFlowBuilder()
>>>> > wb.setApp("name1", app1)
>>>> > # you can do a simple output transformation here etc
>>>> >
>>>> > # connects the input of app1 to app2 etc
>>>> > wb.setApp("name2", app2).connectInput("name1")
>>>> >
>>>> > wb.submit()
>>>> >
>>>> > Now we can load this Python script from XBaya. When XBaya loads this
>>>> script
>>>> > the Python script can output an XML configuration of the topology,
>>>> XBaya
>>>> > can render. There are other ways like directly executing the Python
>>>> script
>>>> > from command line and connecting XBaya indirectly as well. Now you
>>>> can run
>>>> > the workflow from XBaya. Running the Workflow means just executing the
>>>> > Python script.
>>>> >
>>>> > XBaya gets the notifications through messaging and update the UI
>>>> > accordingly.
>>>> >
>>>> > The users need to write the Python script by hand. XBaya cannot
>>>> create the
>>>> > script. Because workflow language is an actual python program the
>>>> benefits
>>>> > are immense. For example user can do workflow steering in the workflow
>>>> > itself by subscribing to messages from Airavata.
>>>> >
>>>> > Thanks,
>>>> > Supun..
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>>>> > shameerainfo@gmail.com> wrote:
>>>> >
>>>> > > Hi Supun,
>>>> > >
>>>> > > I meant to say JS is a well-known client side scripting language i
>>>> have
>>>> > > messed scripting part. Even we use Python, ultimately we should
>>>> convert
>>>> > > this to java model in server side, somehow we need to serialized
>>>> python
>>>> > > representation to the language which java can parse and generate
>>>> that
>>>> > > model. In this case we need to parse python script in java isn't
>>>> it? I am
>>>> > > not exactly clear how you suggesting to use python here. More
>>>> details on
>>>> > > how end system works if we use Python would be great help to clearly
>>>> > > understand your points.
>>>> > >
>>>> > > Thanks,
>>>> > > Shameera.
>>>> > >
>>>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>>>> > chris.mattmann@gmail.com>
>>>> > > wrote:
>>>> > >
>>>> > > > Have you guys considered using JCC [1] as a means
>>>> > > > to expose the workflow API currently in Java as a
>>>> > > > Python API?
>>>> > > >
>>>> > > > We are exploring its use in OODT, and we have already
>>>> > > > created a Tika [2] JCC-based python API.
>>>> > > >
>>>> > > > Cheers,
>>>> > > > Chris
>>>> > > >
>>>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>>>> > > > [2] http://github.com/chrismattmann/tika-python/
>>>> > > >
>>>> > > > ------------------------
>>>> > > > Chris Mattmann
>>>> > > > chris.mattmann@gmail.com
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > > -----Original Message-----
>>>> > > > From: Supun Kamburugamuva <su...@gmail.com>
>>>> > > > Reply-To: <de...@oodt.apache.org>
>>>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>>>> > > > To: dev <de...@airavata.apache.org>
>>>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
>>>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
>>>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
>>>> > > > <de...@oodt.apache.org>
>>>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>>>> > Airavata.
>>>> > > >
>>>> > > > >Once we had an offline discussion about the Airavata Workflow
>>>> language
>>>> > > > >(with Milinda, Saliya and Shameera). In that discussion one
>>>> thing came
>>>> > > out
>>>> > > > >was why we need to invent a different language when a simple
>>>> library
>>>> > > like
>>>> > > > >Python will full fill of Airavata requirements.
>>>> > > > >
>>>> > > > >There are many benefits in using a Python library as the API for
>>>> > > > >controlling Airavata workflows.
>>>> > > > >
>>>> > > > >1. It is a library, gives the ultimate control over the
>>>> execution and
>>>> > it
>>>> > > > >can be simpler than any domain specific language that we can
>>>> come with
>>>> > > > >like
>>>> > > > >XML, JSON etc
>>>> > > > >2. Most people use python and can learn it easily than any Domain
>>>> > > specific
>>>> > > > >language
>>>> > > > >3. You can easily create a Python library for Airavata because
>>>> all the
>>>> > > > >APIs
>>>> > > > >are thrift based.
>>>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>>>> > > > >
>>>> > > > >Any thoughts?
>>>> > > > >
>>>> > > > >Thanks,
>>>> > > > >Supun..
>>>> > > > >
>>>> > > > >
>>>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
>>>> > supun06@gmail.com
>>>> > > >
>>>> > > > >wrote:
>>>> > > > >
>>>> > > > >> Hi Shameera,
>>>> > > > >>
>>>> > > > >> Why you prefer JavaScript over a language like Python?
>>>> > > > >>
>>>> > > > >> Thanks,
>>>> > > > >> Supun..
>>>> > > > >>
>>>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>>>> > > > >> shameerainfo@gmail.com> wrote:
>>>> > > > >>
>>>> > > > >>> Hi,
>>>> > > > >>>
>>>> > > > >>> First of all thanks everyone for giving valuable inputs. After
>>>> > doing
>>>> > > > >>>some
>>>> > > > >>> background search and talking to different people in the
>>>> University
>>>> > > > >>>who has
>>>> > > > >>> used different workflow languages, I myself convinced that
>>>> > > introducing
>>>> > > > >>>an
>>>> > > > >>> another workflow language is not what actually they need. By
>>>> > changing
>>>> > > > >>> exiting workflow language to another will not solve problems.
>>>> What
>>>> > > they
>>>> > > > >>> asking is a easy way to construct the workflows. Indirectly
>>>> what
>>>> > they
>>>> > > > >>> asking for a sort of API which they can use to generate the
>>>> > workflows
>>>> > > > >>>and
>>>> > > > >>> run it. Correct me if i am wrong here.
>>>> > > > >>>
>>>> > > > >>> As most of above replies depict, if we can get a simple API,
>>>> as an
>>>> > > > >>> example, for a web based application, JavaScript API would be
>>>> a
>>>> > good
>>>> > > > >>> solution, and probably JSON would be a good candidate for
>>>> language,
>>>> > > > >>>instead
>>>> > > > >>> of XML.
>>>> > > > >>>
>>>> > > > >>> Airavata community already have started to implement web base
>>>> GUI.
>>>> > > > >>>Hence
>>>> > > > >>> introducing a JSON base JavaScript API would be great help.
>>>> WDYT?
>>>> > > > >>>
>>>> > > > >>> Thanks,
>>>> > > > >>> Shameera.
>>>> > > > >>>
>>>> > > > >>>
>>>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>>> > > > >>> alekny7@gmail.com> wrote:
>>>> > > > >>>
>>>> > > > >>>> Hi,
>>>> > > > >>>>
>>>> > > > >>>> it is not dataflow instead focused on orchestrating REST
>>>> services
>>>> > > but
>>>> > > > >>>> you may find it useful datapoint - we created worfklow
>>>> service
>>>> > that
>>>> > > > >>>>uses
>>>> > > > >>>> natively JavaScript and JSON to describe what happens during
>>>> > > workflow
>>>> > > > >>>> execution:
>>>> > > > >>>>
>>>> > > >
>>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>> > > > >>>>
>>>> > > > >>>> HTH,
>>>> > > > >>>>
>>>> > > > >>>> Alek
>>>> > > > >>>>
>>>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
>>>> smarru@apache.org>
>>>> > > > >>>>wrote:
>>>> > > > >>>>
>>>> > > > >>>>> Hi Chris,
>>>> > > > >>>>>
>>>> > > > >>>>> Great to hear OODT community will be interested in adopting
>>>> a
>>>> > JSON
>>>> > > > >>>>> based workflow language and potentially a web based
>>>> composer as
>>>> > > well.
>>>> > > > >>>>> Airavata previously had BPEL support initially through a
>>>> home
>>>> > grown
>>>> > > > >>>>> implementation [1] by Alek Slominski and later through
>>>> Apache ODE
>>>> > > > >>>>>[2]. Also
>>>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
>>>> read.
>>>> > > > >>>>>
>>>> > > > >>>>> I am of the same opinion that we should adopt something more
>>>> > modern
>>>> > > > >>>>>as
>>>> > > > >>>>> the challenges from scientific workflows seems to be
>>>> converging
>>>> > > with
>>>> > > > >>>>>the
>>>> > > > >>>>> data flow patterns in business workflows.
>>>> > > > >>>>>
>>>> > > > >>>>> It will be great if we can all compile a list of potential
>>>> > > candidates
>>>> > > > >>>>> and hack them through.
>>>> > > > >>>>>
>>>> > > > >>>>> Suresh
>>>> > > > >>>>> [1] -
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > >
>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>> > > > >>>>> [2] -
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > >
>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>>>> > > > >>>>>time_for_eScience_workflows
>>>> > > > >>>>> [3] -
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > >
>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>>>> > > > >>>>>6.pdf
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>> > > > >>>>>
>>>> > > > >>>>> > Hi Guys,
>>>> > > > >>>>> >
>>>> > > > >>>>> > I've been interested in this too - we don't per have a
>>>> specific
>>>> > > > >>>>> > OODT workflow language, but we specific workflows using
>>>> XML,
>>>> > and
>>>> > > > >>>>> > other configuration (we are also thinking of moving to
>>>> JSON for
>>>> > > > >>>>> > this).
>>>> > > > >>>>> >
>>>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
>>>> > complex
>>>> > > > >>>>> > to me.
>>>> > > > >>>>> >
>>>> > > > >>>>> > I wonder at the end of the day if we should adopt
>>>> something
>>>> > more
>>>> > > > >>>>> > modern like PIG or some other data flow type of language
>>>> (PIG
>>>> > > > >>>>> > is really neat).
>>>> > > > >>>>> >
>>>> > > > >>>>> > Cheers,
>>>> > > > >>>>> > Chris
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > > > >>>>> > Chris Mattmann, Ph.D.
>>>> > > > >>>>> > Chief Architect
>>>> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
>>>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
>>>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
>>>> > > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
>>>> > > > >>>>> >
>>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
>>>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089
>>>> USA
>>>> > > > >>>>> >
>>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> > -----Original Message-----
>>>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
>>>> > > > >>>>> > <ar...@airavata.apache.org>
>>>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>> > > > >>>>> > To: "architecture@airavata.apache.org" <
>>>> > > > >>>>> architecture@airavata.apache.org>,
>>>> > > > >>>>> > dev <de...@airavata.apache.org>
>>>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language
>>>> for
>>>> > > > >>>>>Airavata.
>>>> > > > >>>>> >
>>>> > > > >>>>> >> Hi All,
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> As we all know Airavata has its own workflow language
>>>> call
>>>> > XWF.
>>>> > > > >>>>>When
>>>> > > > >>>>> XWF
>>>> > > > >>>>> >> was introduced, main focus points are interoperability
>>>> and
>>>> > > > >>>>> convertibility.
>>>> > > > >>>>> >> But with years of experience it is convinced that above
>>>> > > > >>>>>requirements
>>>> > > > >>>>> are
>>>> > > > >>>>> >> not really useful when we come to real world use cases.
>>>> And
>>>> > XWF
>>>> > > is
>>>> > > > >>>>> XML
>>>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
>>>> image
>>>> > > it
>>>> > > > >>>>> self.
>>>> > > > >>>>> >> But
>>>> > > > >>>>> >> with the recent changes WSDL part is being removed from
>>>> XWF.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
>>>> languages in
>>>> > > > >>>>> industry
>>>> > > > >>>>> >> and
>>>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
>>>> need
>>>> > to
>>>> > > > >>>>> come up
>>>> > > > >>>>> >> with idea how we should improve Airavata workflow
>>>> language,
>>>> > > either
>>>> > > > >>>>> we can
>>>> > > > >>>>> >> improve existing XWF language, totally change to a new
>>>> > language
>>>> > > > >>>>> available
>>>> > > > >>>>> >> in industry or write a new light weight language. Basic
>>>> > > > >>>>>requirements
>>>> > > > >>>>> that
>>>> > > > >>>>> >> we expect from new improvement are, high usability,
>>>> flexible,
>>>> > > > >>>>>light
>>>> > > > >>>>> weight
>>>> > > > >>>>> >> and real time monitoring support. As you can see above
>>>> > > > >>>>>requirements
>>>> > > > >>>>> are
>>>> > > > >>>>> >> not
>>>> > > > >>>>> >> direct comes with workflow languages but we need workflow
>>>> > > language
>>>> > > > >>>>> which
>>>> > > > >>>>> >> help to support above requirements.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> After reading few papers and googling, initially i have
>>>> come
>>>> > up
>>>> > > > >>>>>with
>>>> > > > >>>>> >> following three existing languages,
>>>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>> > > > >>>>> >> 2. WS-BPEL
>>>> > > > >>>>> >> 3. SIDL
>>>> > > > >>>>> >> <
>>>> > > http://computation.llnl.gov/casc/components/index.html#page=home
>>>> > > > >
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific
>>>> domain,
>>>> > > > >>>>> Radical-SAGA
>>>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
>>>> above
>>>> > > > >>>>>three
>>>> > > > >>>>> >> languages we can come up with simple workflow language
>>>> base on
>>>> > > > >>>>> json(or
>>>> > > > >>>>> >> yaml) which support all our requirements for some
>>>> extends.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> It would be grate if I can get more input regarding the
>>>> > $Subject
>>>> > > > >>>>> form the
>>>> > > > >>>>> >> airavata community. You all are more than welcome to
>>>> provide
>>>> > any
>>>> > > > >>>>> type of
>>>> > > > >>>>> >> suggestions.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> Thanks,
>>>> > > > >>>>> >> Shameera.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >>
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> --
>>>> > > > >>>>> >> Best Regards,
>>>> > > > >>>>> >> Shameera Rathnayaka.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT
>>>> gmail.com
>>>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > > >>>>
>>>> > > > >>>>
>>>> > > > >>>> --
>>>> > > > >>>> The best way to predict the future is to invent it - Alan Kay
>>>> > > > >>>>
>>>> > > > >>>
>>>> > > > >>>
>>>> > > > >>>
>>>> > > > >>> --
>>>> > > > >>> Best Regards,
>>>> > > > >>> Shameera Rathnayaka.
>>>> > > > >>>
>>>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
>>>> > > > >>>
>>>> > > > >>
>>>> > > > >>
>>>> > > > >>
>>>> > > > >> --
>>>> > > > >> Supun Kamburugamuva
>>>> > > > >> Member, Apache Software Foundation; http://www.apache.org
>>>> > > > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>>> > > > >> Blog: http://supunk.blogspot.com
>>>> > > > >>
>>>> > > > >>
>>>> > > > >
>>>> > > > >
>>>> > > > >--
>>>> > > > >Supun Kamburugamuva
>>>> > > > >Member, Apache Software Foundation; http://www.apache.org
>>>> > > > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>>> > > > >Blog: http://supunk.blogspot.com
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > > Best Regards,
>>>> > > Shameera Rathnayaka.
>>>> > >
>>>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> > > Blog : http://shameerarathnayaka.blogspot.com/
>>>> > >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Supun Kamburugamuva
>>>> > Member, Apache Software Foundation; http://www.apache.org
>>>> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>>> > Blog: http://supunk.blogspot.com
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Shameera Rathnayaka.
>>>>
>>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>>
>>>
>>>
>>>
>>> --
>>> Supun Kamburugamuva
>>> Member, Apache Software Foundation; http://www.apache.org
>>> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>> Blog: http://supunk.blogspot.com
>>>
>>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
--
Research Assistant
Science Gateways Group
Indiana University
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi Shameera,
I think the whole point of introducing thrift is to support language
bindings for gateway developers and I do not think we should limit this
freedom by introducing a python client just to build a workflow. I propose
to represent a workflow as a thrift data-structure (I think by using
thrift's list objects we can create an adjacency list and of course it will
be tricky because of thrift unsupport for inheritance and other limitations
but I think it will be worth the effort assuming thrift IDL will be
improved a lot in future). If we can represent a workflow as a thrift
object model in the client side all the workflow related CRUD operations
can be generated using thrift, so we can support all the languages which
thrift support.
To submit a workflow we can introduce another thrift operation which accept
the big workflow thrift object and workflow interpreter can parse this and
execute the workflow. Internally we can use XWF and we can convert thrift
model to XWF and XWF to a thrift object.
Regards
Lahiru
On Mon, Oct 20, 2014 at 10:35 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:
> Hi Amila,
>
> Please see my comments inline.
>
> On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara <thejaka.amila@gmail.com
> > wrote:
>
>> Hello,
>>
>> I am sorry, I am bit late on this thread. But when reading through this
>> thread I simply got lost, what this thread is discussing. I have few
>> questions.
>>
>> 1. @Shameera : Is XWF actually a language to define workflow ? To my
>> understanding it was an intermediate representation to convert workflow
>> defined in UI to java object model. Was XWF ever used by any airavata user
>> to define a workflow graph ?
>>
>
> Yes, XWF is the language defined and used by Airavata to explain the
> workflows but it is not well documented. Both server and client sides read
> this description language to generate runtime java representation. so
> when you used XBaya to create a workflow with multiple applications, under
> the hood XBaya generate xwf which describe that workflow to server.
>
>
>
>>
>> From initial description what I understood is we are looking for a
>> improved "intermediate representation", not a language which describes
>> workflows.
>>
>>
>>
>
>> 2. So what is the exact purpose of this proposed language ?
>> - Is it to hide parallelism in workflows ?
>> - Is it to replace the XBAYA functionalities ? (i hope not)
>>
>
>
> Actually initial idea was to identify good, well-defined and scientific
> domain friendly language. Whole purpose of this effort is reduce the entry
> barrier of the end user. But later it is understood that introducing a new
> language won't fix our issue.
>
>
>>
>> 3. What are we trying to achieve by this proposed language which we
>> cannot achieved through workflow UI tool ?
>>
>> 4. Who is going to use this language ?
>>
>
> As I explained, our direction has been changed.By introducing a new
> language we are get nothing but nice description file. No functional
> improvements etc ... The current language should use all airavata
> client(Currently we have only XBaya) side applications to explain the
> workflow to the server side.
>
>
>
>>
>> 5. Why would a user prefer (assuming intended audience of proposed
>> language is end users) a language over a Work Flow UI tool such as XBAYA ?
>> (In other words what are the things we can do with language which we cannot
>> do with UI)
>>
>
> Let's say you are going to write a new web base client for Airavata which
> generate workflows and launch it. What you need to do is do some magic
> with UI and finally come up with description language and send it to server
> side. Here you need to learn how to write a valid XWF file and write your
> own JS code to generate it. But if airavata has a JavaScript API which can
> be used to generate XWF for you by getting JSON objects as input then it
> would be great help for client side developers.
>
> Thank,
> Shameera.
>
>
>
>>
>> Sorry, if above questions are in-appropriate, just wanted to understand
>> what exactly needed.
>>
>> Thanks
>> -Thejaka Amila
>>
>>
>>
>> On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
>> wrote:
>>
>>> I think I'm not suggesting to create a Workflow interpreter using Python
>>> etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
>>> and move it to a more higher level component. The more I think about it,
>>> the model I'm suggesting is similar to what Hadoop, Storm etc has done for
>>> distributed system computations. This model is proven to be successful over
>>> the years.
>>>
>>> Keeping what Airavata does at its core can help you to build a more
>>> robust system. If we look at Airavata as middleware to execute applications
>>> on computing resources we can simplify what Airavata does and focus on
>>> improving the core functionality. All the successful systems have thrived
>>> on defining what it does at its core and keeping it simple and being
>>> excellent at what it does. In that regard keeping workflow aspect out of
>>> Airavata can help you to focus on the core problem. That is to execute an
>>> application on a remote computing resource in a fault tolerant and scalable
>>> manner.
>>>
>>> What I'm suggesting is to give the Orchestrator the capability to
>>> execute a Driver program that is specified by the user. (This program can
>>> be written in Python, Java or any other language). This driver program is
>>> similar to what you define in a Hadoop or Storm configuration. The driver
>>> program specifies the flow of the computation. It specifies what are the
>>> applications needs to be executed, how to manipulate input output. The
>>> driver program is the workflow for the user. Because the user specifies the
>>> program he can program it to handle workflow steering etc. Every time the
>>> user wants to execute this program he can tell Airavata to execute the
>>> Driver program.
>>>
>>> I'm also not 100% sure about all the details. But this can be a new way
>>> to look at how systems like Airavata should behave. Your thoughts and
>>> suggestions are more than welcome.
>>>
>>> Thanks,
>>> Supun..
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
>>> shameerainfo@gmail.com> wrote:
>>>
>>>> Hi Supun,
>>>>
>>>> I think we were in two context, because I as suggesting a way to
>>>> serialize
>>>> and deserialize the workflow description while you are suggesting to
>>>> implement
>>>> some kind of workflow interpreter using Python, where Python client can
>>>> send
>>>> thrift calls to Airavata server to run the application. I can see with
>>>> your
>>>> suggested
>>>> approach we can control the workflow execution process from client side
>>>> which
>>>> make it easy to implement workflow debugger.As you mentioned this is a
>>>> major change
>>>> to Airavata. So we should neatly think as this will change our existing
>>>> architecture.
>>>>
>>>> Still if someone need to use different language java, php, JS etc ... to
>>>> run the same
>>>> workflow which generated by Python, we need a language independent
>>>> workflow
>>>> description.
>>>> My initial question was what is the best language for this?. But as I
>>>> have
>>>> explained in
>>>> one of my previous reply, It is not matter what language we used Either
>>>> we
>>>> can use
>>>> XML or JSON to write this description, what matters is how easy to
>>>> generate
>>>> workflow with the provided API. Hence it would be great to have set of
>>>> neat
>>>> APIs in
>>>> different languages.
>>>>
>>>> Thanks,
>>>> Shameera.
>>>>
>>>>
>>>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <
>>>> supun06@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi Shameera,
>>>> >
>>>> > Using python is a radical approach for workflow I think. But I
>>>> believe this
>>>> > is a very beautiful and new approach.Here is a possible scenario for
>>>> > implementation and running using a Python based library.
>>>> >
>>>> > The Python library facilitates the creation of Applications and
>>>> submitting
>>>> > them to Airavata for execution. The underlying functionality is
>>>> exactly
>>>> > similar to what Java clients provides.The only difference is that,
>>>> Python
>>>> > library should have a more fluent API than Java for easy creation of
>>>> > workflows. We can generate the Python clients that talk to Airavata
>>>> server
>>>> > using Thrift.
>>>> >
>>>> > Here is an example off the top of my head to a Python script created
>>>> by
>>>> > user for a Workflow. This is a very crude example and we need to come
>>>> up
>>>> > with a much better API if we are going to go along this path. First
>>>> we need
>>>> > to write a Python script that can execute a workflow using Airavata.
>>>> >
>>>> > import airavata
>>>> >
>>>> > host = Host("localhost", ....)
>>>> > app1 = Application(host, ....)
>>>> > app2 = Application(host, ....)
>>>> >
>>>> > # we will connect these applications as a workflow using some topology
>>>> > builder or other constructs
>>>> >
>>>> > wb = WorkFlowBuilder()
>>>> > wb.setApp("name1", app1)
>>>> > # you can do a simple output transformation here etc
>>>> >
>>>> > # connects the input of app1 to app2 etc
>>>> > wb.setApp("name2", app2).connectInput("name1")
>>>> >
>>>> > wb.submit()
>>>> >
>>>> > Now we can load this Python script from XBaya. When XBaya loads this
>>>> script
>>>> > the Python script can output an XML configuration of the topology,
>>>> XBaya
>>>> > can render. There are other ways like directly executing the Python
>>>> script
>>>> > from command line and connecting XBaya indirectly as well. Now you
>>>> can run
>>>> > the workflow from XBaya. Running the Workflow means just executing the
>>>> > Python script.
>>>> >
>>>> > XBaya gets the notifications through messaging and update the UI
>>>> > accordingly.
>>>> >
>>>> > The users need to write the Python script by hand. XBaya cannot
>>>> create the
>>>> > script. Because workflow language is an actual python program the
>>>> benefits
>>>> > are immense. For example user can do workflow steering in the workflow
>>>> > itself by subscribing to messages from Airavata.
>>>> >
>>>> > Thanks,
>>>> > Supun..
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>>>> > shameerainfo@gmail.com> wrote:
>>>> >
>>>> > > Hi Supun,
>>>> > >
>>>> > > I meant to say JS is a well-known client side scripting language i
>>>> have
>>>> > > messed scripting part. Even we use Python, ultimately we should
>>>> convert
>>>> > > this to java model in server side, somehow we need to serialized
>>>> python
>>>> > > representation to the language which java can parse and generate
>>>> that
>>>> > > model. In this case we need to parse python script in java isn't
>>>> it? I am
>>>> > > not exactly clear how you suggesting to use python here. More
>>>> details on
>>>> > > how end system works if we use Python would be great help to clearly
>>>> > > understand your points.
>>>> > >
>>>> > > Thanks,
>>>> > > Shameera.
>>>> > >
>>>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>>>> > chris.mattmann@gmail.com>
>>>> > > wrote:
>>>> > >
>>>> > > > Have you guys considered using JCC [1] as a means
>>>> > > > to expose the workflow API currently in Java as a
>>>> > > > Python API?
>>>> > > >
>>>> > > > We are exploring its use in OODT, and we have already
>>>> > > > created a Tika [2] JCC-based python API.
>>>> > > >
>>>> > > > Cheers,
>>>> > > > Chris
>>>> > > >
>>>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>>>> > > > [2] http://github.com/chrismattmann/tika-python/
>>>> > > >
>>>> > > > ------------------------
>>>> > > > Chris Mattmann
>>>> > > > chris.mattmann@gmail.com
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > > -----Original Message-----
>>>> > > > From: Supun Kamburugamuva <su...@gmail.com>
>>>> > > > Reply-To: <de...@oodt.apache.org>
>>>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>>>> > > > To: dev <de...@airavata.apache.org>
>>>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
>>>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
>>>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
>>>> > > > <de...@oodt.apache.org>
>>>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>>>> > Airavata.
>>>> > > >
>>>> > > > >Once we had an offline discussion about the Airavata Workflow
>>>> language
>>>> > > > >(with Milinda, Saliya and Shameera). In that discussion one
>>>> thing came
>>>> > > out
>>>> > > > >was why we need to invent a different language when a simple
>>>> library
>>>> > > like
>>>> > > > >Python will full fill of Airavata requirements.
>>>> > > > >
>>>> > > > >There are many benefits in using a Python library as the API for
>>>> > > > >controlling Airavata workflows.
>>>> > > > >
>>>> > > > >1. It is a library, gives the ultimate control over the
>>>> execution and
>>>> > it
>>>> > > > >can be simpler than any domain specific language that we can
>>>> come with
>>>> > > > >like
>>>> > > > >XML, JSON etc
>>>> > > > >2. Most people use python and can learn it easily than any Domain
>>>> > > specific
>>>> > > > >language
>>>> > > > >3. You can easily create a Python library for Airavata because
>>>> all the
>>>> > > > >APIs
>>>> > > > >are thrift based.
>>>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>>>> > > > >
>>>> > > > >Any thoughts?
>>>> > > > >
>>>> > > > >Thanks,
>>>> > > > >Supun..
>>>> > > > >
>>>> > > > >
>>>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
>>>> > supun06@gmail.com
>>>> > > >
>>>> > > > >wrote:
>>>> > > > >
>>>> > > > >> Hi Shameera,
>>>> > > > >>
>>>> > > > >> Why you prefer JavaScript over a language like Python?
>>>> > > > >>
>>>> > > > >> Thanks,
>>>> > > > >> Supun..
>>>> > > > >>
>>>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>>>> > > > >> shameerainfo@gmail.com> wrote:
>>>> > > > >>
>>>> > > > >>> Hi,
>>>> > > > >>>
>>>> > > > >>> First of all thanks everyone for giving valuable inputs. After
>>>> > doing
>>>> > > > >>>some
>>>> > > > >>> background search and talking to different people in the
>>>> University
>>>> > > > >>>who has
>>>> > > > >>> used different workflow languages, I myself convinced that
>>>> > > introducing
>>>> > > > >>>an
>>>> > > > >>> another workflow language is not what actually they need. By
>>>> > changing
>>>> > > > >>> exiting workflow language to another will not solve problems.
>>>> What
>>>> > > they
>>>> > > > >>> asking is a easy way to construct the workflows. Indirectly
>>>> what
>>>> > they
>>>> > > > >>> asking for a sort of API which they can use to generate the
>>>> > workflows
>>>> > > > >>>and
>>>> > > > >>> run it. Correct me if i am wrong here.
>>>> > > > >>>
>>>> > > > >>> As most of above replies depict, if we can get a simple API,
>>>> as an
>>>> > > > >>> example, for a web based application, JavaScript API would be
>>>> a
>>>> > good
>>>> > > > >>> solution, and probably JSON would be a good candidate for
>>>> language,
>>>> > > > >>>instead
>>>> > > > >>> of XML.
>>>> > > > >>>
>>>> > > > >>> Airavata community already have started to implement web base
>>>> GUI.
>>>> > > > >>>Hence
>>>> > > > >>> introducing a JSON base JavaScript API would be great help.
>>>> WDYT?
>>>> > > > >>>
>>>> > > > >>> Thanks,
>>>> > > > >>> Shameera.
>>>> > > > >>>
>>>> > > > >>>
>>>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>>> > > > >>> alekny7@gmail.com> wrote:
>>>> > > > >>>
>>>> > > > >>>> Hi,
>>>> > > > >>>>
>>>> > > > >>>> it is not dataflow instead focused on orchestrating REST
>>>> services
>>>> > > but
>>>> > > > >>>> you may find it useful datapoint - we created worfklow
>>>> service
>>>> > that
>>>> > > > >>>>uses
>>>> > > > >>>> natively JavaScript and JSON to describe what happens during
>>>> > > workflow
>>>> > > > >>>> execution:
>>>> > > > >>>>
>>>> > > >
>>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>> > > > >>>>
>>>> > > > >>>> HTH,
>>>> > > > >>>>
>>>> > > > >>>> Alek
>>>> > > > >>>>
>>>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
>>>> smarru@apache.org>
>>>> > > > >>>>wrote:
>>>> > > > >>>>
>>>> > > > >>>>> Hi Chris,
>>>> > > > >>>>>
>>>> > > > >>>>> Great to hear OODT community will be interested in adopting
>>>> a
>>>> > JSON
>>>> > > > >>>>> based workflow language and potentially a web based
>>>> composer as
>>>> > > well.
>>>> > > > >>>>> Airavata previously had BPEL support initially through a
>>>> home
>>>> > grown
>>>> > > > >>>>> implementation [1] by Alek Slominski and later through
>>>> Apache ODE
>>>> > > > >>>>>[2]. Also
>>>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
>>>> read.
>>>> > > > >>>>>
>>>> > > > >>>>> I am of the same opinion that we should adopt something more
>>>> > modern
>>>> > > > >>>>>as
>>>> > > > >>>>> the challenges from scientific workflows seems to be
>>>> converging
>>>> > > with
>>>> > > > >>>>>the
>>>> > > > >>>>> data flow patterns in business workflows.
>>>> > > > >>>>>
>>>> > > > >>>>> It will be great if we can all compile a list of potential
>>>> > > candidates
>>>> > > > >>>>> and hack them through.
>>>> > > > >>>>>
>>>> > > > >>>>> Suresh
>>>> > > > >>>>> [1] -
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > >
>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>> > > > >>>>> [2] -
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > >
>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>>>> > > > >>>>>time_for_eScience_workflows
>>>> > > > >>>>> [3] -
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > >
>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>>>> > > > >>>>>6.pdf
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>> > > > >>>>>
>>>> > > > >>>>> > Hi Guys,
>>>> > > > >>>>> >
>>>> > > > >>>>> > I've been interested in this too - we don't per have a
>>>> specific
>>>> > > > >>>>> > OODT workflow language, but we specific workflows using
>>>> XML,
>>>> > and
>>>> > > > >>>>> > other configuration (we are also thinking of moving to
>>>> JSON for
>>>> > > > >>>>> > this).
>>>> > > > >>>>> >
>>>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
>>>> > complex
>>>> > > > >>>>> > to me.
>>>> > > > >>>>> >
>>>> > > > >>>>> > I wonder at the end of the day if we should adopt
>>>> something
>>>> > more
>>>> > > > >>>>> > modern like PIG or some other data flow type of language
>>>> (PIG
>>>> > > > >>>>> > is really neat).
>>>> > > > >>>>> >
>>>> > > > >>>>> > Cheers,
>>>> > > > >>>>> > Chris
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > > > >>>>> > Chris Mattmann, Ph.D.
>>>> > > > >>>>> > Chief Architect
>>>> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
>>>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
>>>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
>>>> > > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
>>>> > > > >>>>> >
>>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
>>>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089
>>>> USA
>>>> > > > >>>>> >
>>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> > -----Original Message-----
>>>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
>>>> > > > >>>>> > <ar...@airavata.apache.org>
>>>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>> > > > >>>>> > To: "architecture@airavata.apache.org" <
>>>> > > > >>>>> architecture@airavata.apache.org>,
>>>> > > > >>>>> > dev <de...@airavata.apache.org>
>>>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language
>>>> for
>>>> > > > >>>>>Airavata.
>>>> > > > >>>>> >
>>>> > > > >>>>> >> Hi All,
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> As we all know Airavata has its own workflow language
>>>> call
>>>> > XWF.
>>>> > > > >>>>>When
>>>> > > > >>>>> XWF
>>>> > > > >>>>> >> was introduced, main focus points are interoperability
>>>> and
>>>> > > > >>>>> convertibility.
>>>> > > > >>>>> >> But with years of experience it is convinced that above
>>>> > > > >>>>>requirements
>>>> > > > >>>>> are
>>>> > > > >>>>> >> not really useful when we come to real world use cases.
>>>> And
>>>> > XWF
>>>> > > is
>>>> > > > >>>>> XML
>>>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
>>>> image
>>>> > > it
>>>> > > > >>>>> self.
>>>> > > > >>>>> >> But
>>>> > > > >>>>> >> with the recent changes WSDL part is being removed from
>>>> XWF.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
>>>> languages in
>>>> > > > >>>>> industry
>>>> > > > >>>>> >> and
>>>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
>>>> need
>>>> > to
>>>> > > > >>>>> come up
>>>> > > > >>>>> >> with idea how we should improve Airavata workflow
>>>> language,
>>>> > > either
>>>> > > > >>>>> we can
>>>> > > > >>>>> >> improve existing XWF language, totally change to a new
>>>> > language
>>>> > > > >>>>> available
>>>> > > > >>>>> >> in industry or write a new light weight language. Basic
>>>> > > > >>>>>requirements
>>>> > > > >>>>> that
>>>> > > > >>>>> >> we expect from new improvement are, high usability,
>>>> flexible,
>>>> > > > >>>>>light
>>>> > > > >>>>> weight
>>>> > > > >>>>> >> and real time monitoring support. As you can see above
>>>> > > > >>>>>requirements
>>>> > > > >>>>> are
>>>> > > > >>>>> >> not
>>>> > > > >>>>> >> direct comes with workflow languages but we need workflow
>>>> > > language
>>>> > > > >>>>> which
>>>> > > > >>>>> >> help to support above requirements.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> After reading few papers and googling, initially i have
>>>> come
>>>> > up
>>>> > > > >>>>>with
>>>> > > > >>>>> >> following three existing languages,
>>>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>> > > > >>>>> >> 2. WS-BPEL
>>>> > > > >>>>> >> 3. SIDL
>>>> > > > >>>>> >> <
>>>> > > http://computation.llnl.gov/casc/components/index.html#page=home
>>>> > > > >
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific
>>>> domain,
>>>> > > > >>>>> Radical-SAGA
>>>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
>>>> above
>>>> > > > >>>>>three
>>>> > > > >>>>> >> languages we can come up with simple workflow language
>>>> base on
>>>> > > > >>>>> json(or
>>>> > > > >>>>> >> yaml) which support all our requirements for some
>>>> extends.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> It would be grate if I can get more input regarding the
>>>> > $Subject
>>>> > > > >>>>> form the
>>>> > > > >>>>> >> airavata community. You all are more than welcome to
>>>> provide
>>>> > any
>>>> > > > >>>>> type of
>>>> > > > >>>>> >> suggestions.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> Thanks,
>>>> > > > >>>>> >> Shameera.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >>
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> --
>>>> > > > >>>>> >> Best Regards,
>>>> > > > >>>>> >> Shameera Rathnayaka.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT
>>>> gmail.com
>>>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > > >>>>
>>>> > > > >>>>
>>>> > > > >>>> --
>>>> > > > >>>> The best way to predict the future is to invent it - Alan Kay
>>>> > > > >>>>
>>>> > > > >>>
>>>> > > > >>>
>>>> > > > >>>
>>>> > > > >>> --
>>>> > > > >>> Best Regards,
>>>> > > > >>> Shameera Rathnayaka.
>>>> > > > >>>
>>>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
>>>> > > > >>>
>>>> > > > >>
>>>> > > > >>
>>>> > > > >>
>>>> > > > >> --
>>>> > > > >> Supun Kamburugamuva
>>>> > > > >> Member, Apache Software Foundation; http://www.apache.org
>>>> > > > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>>> > > > >> Blog: http://supunk.blogspot.com
>>>> > > > >>
>>>> > > > >>
>>>> > > > >
>>>> > > > >
>>>> > > > >--
>>>> > > > >Supun Kamburugamuva
>>>> > > > >Member, Apache Software Foundation; http://www.apache.org
>>>> > > > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>>> > > > >Blog: http://supunk.blogspot.com
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > > Best Regards,
>>>> > > Shameera Rathnayaka.
>>>> > >
>>>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> > > Blog : http://shameerarathnayaka.blogspot.com/
>>>> > >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Supun Kamburugamuva
>>>> > Member, Apache Software Foundation; http://www.apache.org
>>>> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>>> > Blog: http://supunk.blogspot.com
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Shameera Rathnayaka.
>>>>
>>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>>
>>>
>>>
>>>
>>> --
>>> Supun Kamburugamuva
>>> Member, Apache Software Foundation; http://www.apache.org
>>> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>> Blog: http://supunk.blogspot.com
>>>
>>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
--
Research Assistant
Science Gateways Group
Indiana University
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Amila Jayasekara <th...@gmail.com>.
Hi Shameera,
Thanks for the reply.
I wouldnt call XWF a "language". It is not a language in any sense but an
intermediate representation of a workflow graph.
To my understanding what you need is not a language but an efficient
portable intermediate representation. Seems you also have realized it. JSON
might be a good candidate provided most of the frontends are web based.
Thanks
-Thejaka
On Mon, Oct 20, 2014 at 10:35 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:
> Hi Amila,
>
> Please see my comments inline.
>
> On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara <thejaka.amila@gmail.com
> >
> wrote:
>
> > Hello,
> >
> > I am sorry, I am bit late on this thread. But when reading through this
> > thread I simply got lost, what this thread is discussing. I have few
> > questions.
> >
> > 1. @Shameera : Is XWF actually a language to define workflow ? To my
> > understanding it was an intermediate representation to convert workflow
> > defined in UI to java object model. Was XWF ever used by any airavata
> user
> > to define a workflow graph ?
> >
>
> Yes, XWF is the language defined and used by Airavata to explain the
> workflows but it is not well documented. Both server and client sides read
> this description language to generate runtime java representation. so
> when you used XBaya to create a workflow with multiple applications, under
> the hood XBaya generate xwf which describe that workflow to server.
>
>
>
> >
> > From initial description what I understood is we are looking for a
> > improved "intermediate representation", not a language which describes
> > workflows.
> >
> >
> >
>
> > 2. So what is the exact purpose of this proposed language ?
> > - Is it to hide parallelism in workflows ?
> > - Is it to replace the XBAYA functionalities ? (i hope not)
> >
>
>
> Actually initial idea was to identify good, well-defined and scientific
> domain friendly language. Whole purpose of this effort is reduce the entry
> barrier of the end user. But later it is understood that introducing a new
> language won't fix our issue.
>
>
> >
> > 3. What are we trying to achieve by this proposed language which we
> cannot
> > achieved through workflow UI tool ?
> >
> > 4. Who is going to use this language ?
> >
>
> As I explained, our direction has been changed.By introducing a new
> language we are get nothing but nice description file. No functional
> improvements etc ... The current language should use all airavata
> client(Currently we have only XBaya) side applications to explain the
> workflow to the server side.
>
>
>
> >
> > 5. Why would a user prefer (assuming intended audience of proposed
> > language is end users) a language over a Work Flow UI tool such as XBAYA
> ?
> > (In other words what are the things we can do with language which we
> cannot
> > do with UI)
> >
>
> Let's say you are going to write a new web base client for Airavata which
> generate workflows and launch it. What you need to do is do some magic
> with UI and finally come up with description language and send it to server
> side. Here you need to learn how to write a valid XWF file and write your
> own JS code to generate it. But if airavata has a JavaScript API which can
> be used to generate XWF for you by getting JSON objects as input then it
> would be great help for client side developers.
>
> Thank,
> Shameera.
>
>
>
> >
> > Sorry, if above questions are in-appropriate, just wanted to understand
> > what exactly needed.
> >
> > Thanks
> > -Thejaka Amila
> >
> >
> >
> > On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
> > wrote:
> >
> >> I think I'm not suggesting to create a Workflow interpreter using Python
> >> etc. What I'm suggesting is to remove the Worflow aspect from core
> Airavata
> >> and move it to a more higher level component. The more I think about it,
> >> the model I'm suggesting is similar to what Hadoop, Storm etc has done
> for
> >> distributed system computations. This model is proven to be successful
> over
> >> the years.
> >>
> >> Keeping what Airavata does at its core can help you to build a more
> >> robust system. If we look at Airavata as middleware to execute
> applications
> >> on computing resources we can simplify what Airavata does and focus on
> >> improving the core functionality. All the successful systems have
> thrived
> >> on defining what it does at its core and keeping it simple and being
> >> excellent at what it does. In that regard keeping workflow aspect out of
> >> Airavata can help you to focus on the core problem. That is to execute
> an
> >> application on a remote computing resource in a fault tolerant and
> scalable
> >> manner.
> >>
> >> What I'm suggesting is to give the Orchestrator the capability to
> execute
> >> a Driver program that is specified by the user. (This program can be
> >> written in Python, Java or any other language). This driver program is
> >> similar to what you define in a Hadoop or Storm configuration. The
> driver
> >> program specifies the flow of the computation. It specifies what are the
> >> applications needs to be executed, how to manipulate input output. The
> >> driver program is the workflow for the user. Because the user specifies
> the
> >> program he can program it to handle workflow steering etc. Every time
> the
> >> user wants to execute this program he can tell Airavata to execute the
> >> Driver program.
> >>
> >> I'm also not 100% sure about all the details. But this can be a new way
> >> to look at how systems like Airavata should behave. Your thoughts and
> >> suggestions are more than welcome.
> >>
> >> Thanks,
> >> Supun..
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
> >> shameerainfo@gmail.com> wrote:
> >>
> >>> Hi Supun,
> >>>
> >>> I think we were in two context, because I as suggesting a way to
> >>> serialize
> >>> and deserialize the workflow description while you are suggesting to
> >>> implement
> >>> some kind of workflow interpreter using Python, where Python client can
> >>> send
> >>> thrift calls to Airavata server to run the application. I can see with
> >>> your
> >>> suggested
> >>> approach we can control the workflow execution process from client side
> >>> which
> >>> make it easy to implement workflow debugger.As you mentioned this is a
> >>> major change
> >>> to Airavata. So we should neatly think as this will change our existing
> >>> architecture.
> >>>
> >>> Still if someone need to use different language java, php, JS etc ...
> to
> >>> run the same
> >>> workflow which generated by Python, we need a language independent
> >>> workflow
> >>> description.
> >>> My initial question was what is the best language for this?. But as I
> >>> have
> >>> explained in
> >>> one of my previous reply, It is not matter what language we used Either
> >>> we
> >>> can use
> >>> XML or JSON to write this description, what matters is how easy to
> >>> generate
> >>> workflow with the provided API. Hence it would be great to have set of
> >>> neat
> >>> APIs in
> >>> different languages.
> >>>
> >>> Thanks,
> >>> Shameera.
> >>>
> >>>
> >>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <
> supun06@gmail.com
> >>> >
> >>> wrote:
> >>>
> >>> > Hi Shameera,
> >>> >
> >>> > Using python is a radical approach for workflow I think. But I
> believe
> >>> this
> >>> > is a very beautiful and new approach.Here is a possible scenario for
> >>> > implementation and running using a Python based library.
> >>> >
> >>> > The Python library facilitates the creation of Applications and
> >>> submitting
> >>> > them to Airavata for execution. The underlying functionality is
> exactly
> >>> > similar to what Java clients provides.The only difference is that,
> >>> Python
> >>> > library should have a more fluent API than Java for easy creation of
> >>> > workflows. We can generate the Python clients that talk to Airavata
> >>> server
> >>> > using Thrift.
> >>> >
> >>> > Here is an example off the top of my head to a Python script created
> by
> >>> > user for a Workflow. This is a very crude example and we need to come
> >>> up
> >>> > with a much better API if we are going to go along this path. First
> we
> >>> need
> >>> > to write a Python script that can execute a workflow using Airavata.
> >>> >
> >>> > import airavata
> >>> >
> >>> > host = Host("localhost", ....)
> >>> > app1 = Application(host, ....)
> >>> > app2 = Application(host, ....)
> >>> >
> >>> > # we will connect these applications as a workflow using some
> topology
> >>> > builder or other constructs
> >>> >
> >>> > wb = WorkFlowBuilder()
> >>> > wb.setApp("name1", app1)
> >>> > # you can do a simple output transformation here etc
> >>> >
> >>> > # connects the input of app1 to app2 etc
> >>> > wb.setApp("name2", app2).connectInput("name1")
> >>> >
> >>> > wb.submit()
> >>> >
> >>> > Now we can load this Python script from XBaya. When XBaya loads this
> >>> script
> >>> > the Python script can output an XML configuration of the topology,
> >>> XBaya
> >>> > can render. There are other ways like directly executing the Python
> >>> script
> >>> > from command line and connecting XBaya indirectly as well. Now you
> can
> >>> run
> >>> > the workflow from XBaya. Running the Workflow means just executing
> the
> >>> > Python script.
> >>> >
> >>> > XBaya gets the notifications through messaging and update the UI
> >>> > accordingly.
> >>> >
> >>> > The users need to write the Python script by hand. XBaya cannot
> create
> >>> the
> >>> > script. Because workflow language is an actual python program the
> >>> benefits
> >>> > are immense. For example user can do workflow steering in the
> workflow
> >>> > itself by subscribing to messages from Airavata.
> >>> >
> >>> > Thanks,
> >>> > Supun..
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> >>> > shameerainfo@gmail.com> wrote:
> >>> >
> >>> > > Hi Supun,
> >>> > >
> >>> > > I meant to say JS is a well-known client side scripting language i
> >>> have
> >>> > > messed scripting part. Even we use Python, ultimately we should
> >>> convert
> >>> > > this to java model in server side, somehow we need to serialized
> >>> python
> >>> > > representation to the language which java can parse and generate
> that
> >>> > > model. In this case we need to parse python script in java isn't
> it?
> >>> I am
> >>> > > not exactly clear how you suggesting to use python here. More
> >>> details on
> >>> > > how end system works if we use Python would be great help to
> clearly
> >>> > > understand your points.
> >>> > >
> >>> > > Thanks,
> >>> > > Shameera.
> >>> > >
> >>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> >>> > chris.mattmann@gmail.com>
> >>> > > wrote:
> >>> > >
> >>> > > > Have you guys considered using JCC [1] as a means
> >>> > > > to expose the workflow API currently in Java as a
> >>> > > > Python API?
> >>> > > >
> >>> > > > We are exploring its use in OODT, and we have already
> >>> > > > created a Tika [2] JCC-based python API.
> >>> > > >
> >>> > > > Cheers,
> >>> > > > Chris
> >>> > > >
> >>> > > > [1] http://lucene.apache.org/pylucene/jcc/
> >>> > > > [2] http://github.com/chrismattmann/tika-python/
> >>> > > >
> >>> > > > ------------------------
> >>> > > > Chris Mattmann
> >>> > > > chris.mattmann@gmail.com
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > -----Original Message-----
> >>> > > > From: Supun Kamburugamuva <su...@gmail.com>
> >>> > > > Reply-To: <de...@oodt.apache.org>
> >>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
> >>> > > > To: dev <de...@airavata.apache.org>
> >>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> >>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
> >>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> >>> > > > <de...@oodt.apache.org>
> >>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> >>> > Airavata.
> >>> > > >
> >>> > > > >Once we had an offline discussion about the Airavata Workflow
> >>> language
> >>> > > > >(with Milinda, Saliya and Shameera). In that discussion one
> thing
> >>> came
> >>> > > out
> >>> > > > >was why we need to invent a different language when a simple
> >>> library
> >>> > > like
> >>> > > > >Python will full fill of Airavata requirements.
> >>> > > > >
> >>> > > > >There are many benefits in using a Python library as the API for
> >>> > > > >controlling Airavata workflows.
> >>> > > > >
> >>> > > > >1. It is a library, gives the ultimate control over the
> execution
> >>> and
> >>> > it
> >>> > > > >can be simpler than any domain specific language that we can
> come
> >>> with
> >>> > > > >like
> >>> > > > >XML, JSON etc
> >>> > > > >2. Most people use python and can learn it easily than any
> Domain
> >>> > > specific
> >>> > > > >language
> >>> > > > >3. You can easily create a Python library for Airavata because
> >>> all the
> >>> > > > >APIs
> >>> > > > >are thrift based.
> >>> > > > >4. If you design the constructs correctly you can plug an XBaya.
> >>> > > > >
> >>> > > > >Any thoughts?
> >>> > > > >
> >>> > > > >Thanks,
> >>> > > > >Supun..
> >>> > > > >
> >>> > > > >
> >>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> >>> > supun06@gmail.com
> >>> > > >
> >>> > > > >wrote:
> >>> > > > >
> >>> > > > >> Hi Shameera,
> >>> > > > >>
> >>> > > > >> Why you prefer JavaScript over a language like Python?
> >>> > > > >>
> >>> > > > >> Thanks,
> >>> > > > >> Supun..
> >>> > > > >>
> >>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> >>> > > > >> shameerainfo@gmail.com> wrote:
> >>> > > > >>
> >>> > > > >>> Hi,
> >>> > > > >>>
> >>> > > > >>> First of all thanks everyone for giving valuable inputs.
> After
> >>> > doing
> >>> > > > >>>some
> >>> > > > >>> background search and talking to different people in the
> >>> University
> >>> > > > >>>who has
> >>> > > > >>> used different workflow languages, I myself convinced that
> >>> > > introducing
> >>> > > > >>>an
> >>> > > > >>> another workflow language is not what actually they need. By
> >>> > changing
> >>> > > > >>> exiting workflow language to another will not solve problems.
> >>> What
> >>> > > they
> >>> > > > >>> asking is a easy way to construct the workflows. Indirectly
> >>> what
> >>> > they
> >>> > > > >>> asking for a sort of API which they can use to generate the
> >>> > workflows
> >>> > > > >>>and
> >>> > > > >>> run it. Correct me if i am wrong here.
> >>> > > > >>>
> >>> > > > >>> As most of above replies depict, if we can get a simple API,
> >>> as an
> >>> > > > >>> example, for a web based application, JavaScript API would
> be a
> >>> > good
> >>> > > > >>> solution, and probably JSON would be a good candidate for
> >>> language,
> >>> > > > >>>instead
> >>> > > > >>> of XML.
> >>> > > > >>>
> >>> > > > >>> Airavata community already have started to implement web base
> >>> GUI.
> >>> > > > >>>Hence
> >>> > > > >>> introducing a JSON base JavaScript API would be great help.
> >>> WDYT?
> >>> > > > >>>
> >>> > > > >>> Thanks,
> >>> > > > >>> Shameera.
> >>> > > > >>>
> >>> > > > >>>
> >>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >>> > > > >>> alekny7@gmail.com> wrote:
> >>> > > > >>>
> >>> > > > >>>> Hi,
> >>> > > > >>>>
> >>> > > > >>>> it is not dataflow instead focused on orchestrating REST
> >>> services
> >>> > > but
> >>> > > > >>>> you may find it useful datapoint - we created worfklow
> service
> >>> > that
> >>> > > > >>>>uses
> >>> > > > >>>> natively JavaScript and JSON to describe what happens during
> >>> > > workflow
> >>> > > > >>>> execution:
> >>> > > > >>>>
> >>> > > >
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>> > > > >>>>
> >>> > > > >>>> HTH,
> >>> > > > >>>>
> >>> > > > >>>> Alek
> >>> > > > >>>>
> >>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
> >>> smarru@apache.org>
> >>> > > > >>>>wrote:
> >>> > > > >>>>
> >>> > > > >>>>> Hi Chris,
> >>> > > > >>>>>
> >>> > > > >>>>> Great to hear OODT community will be interested in
> adopting a
> >>> > JSON
> >>> > > > >>>>> based workflow language and potentially a web based
> composer
> >>> as
> >>> > > well.
> >>> > > > >>>>> Airavata previously had BPEL support initially through a
> home
> >>> > grown
> >>> > > > >>>>> implementation [1] by Alek Slominski and later through
> >>> Apache ODE
> >>> > > > >>>>>[2]. Also
> >>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
> >>> read.
> >>> > > > >>>>>
> >>> > > > >>>>> I am of the same opinion that we should adopt something
> more
> >>> > modern
> >>> > > > >>>>>as
> >>> > > > >>>>> the challenges from scientific workflows seems to be
> >>> converging
> >>> > > with
> >>> > > > >>>>>the
> >>> > > > >>>>> data flow patterns in business workflows.
> >>> > > > >>>>>
> >>> > > > >>>>> It will be great if we can all compile a list of potential
> >>> > > candidates
> >>> > > > >>>>> and hack them through.
> >>> > > > >>>>>
> >>> > > > >>>>> Suresh
> >>> > > > >>>>> [1] -
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > >
> >>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>> > > > >>>>> [2] -
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > >
> >>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> >>> > > > >>>>>time_for_eScience_workflows
> >>> > > > >>>>> [3] -
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > >
> >>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> >>> > > > >>>>>6.pdf
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>> > > > >>>>>
> >>> > > > >>>>> > Hi Guys,
> >>> > > > >>>>> >
> >>> > > > >>>>> > I've been interested in this too - we don't per have a
> >>> specific
> >>> > > > >>>>> > OODT workflow language, but we specific workflows using
> >>> XML,
> >>> > and
> >>> > > > >>>>> > other configuration (we are also thinking of moving to
> >>> JSON for
> >>> > > > >>>>> > this).
> >>> > > > >>>>> >
> >>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
> >>> > complex
> >>> > > > >>>>> > to me.
> >>> > > > >>>>> >
> >>> > > > >>>>> > I wonder at the end of the day if we should adopt
> something
> >>> > more
> >>> > > > >>>>> > modern like PIG or some other data flow type of language
> >>> (PIG
> >>> > > > >>>>> > is really neat).
> >>> > > > >>>>> >
> >>> > > > >>>>> > Cheers,
> >>> > > > >>>>> > Chris
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> > > > >>>>> > Chris Mattmann, Ph.D.
> >>> > > > >>>>> > Chief Architect
> >>> > > > >>>>> > Instrument Software and Science Data Systems Section
> (398)
> >>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
> >>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
> >>> > > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
> >>> > > > >>>>> >
> >>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
> >>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089
> >>> USA
> >>> > > > >>>>> >
> >>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> > -----Original Message-----
> >>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
> >>> > > > >>>>> > <ar...@airavata.apache.org>
> >>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>> > > > >>>>> > To: "architecture@airavata.apache.org" <
> >>> > > > >>>>> architecture@airavata.apache.org>,
> >>> > > > >>>>> > dev <de...@airavata.apache.org>
> >>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language
> for
> >>> > > > >>>>>Airavata.
> >>> > > > >>>>> >
> >>> > > > >>>>> >> Hi All,
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> As we all know Airavata has its own workflow language
> call
> >>> > XWF.
> >>> > > > >>>>>When
> >>> > > > >>>>> XWF
> >>> > > > >>>>> >> was introduced, main focus points are interoperability
> and
> >>> > > > >>>>> convertibility.
> >>> > > > >>>>> >> But with years of experience it is convinced that above
> >>> > > > >>>>>requirements
> >>> > > > >>>>> are
> >>> > > > >>>>> >> not really useful when we come to real world use cases.
> >>> And
> >>> > XWF
> >>> > > is
> >>> > > > >>>>> XML
> >>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
> >>> image
> >>> > > it
> >>> > > > >>>>> self.
> >>> > > > >>>>> >> But
> >>> > > > >>>>> >> with the recent changes WSDL part is being removed from
> >>> XWF.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
> >>> languages in
> >>> > > > >>>>> industry
> >>> > > > >>>>> >> and
> >>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
> >>> need
> >>> > to
> >>> > > > >>>>> come up
> >>> > > > >>>>> >> with idea how we should improve Airavata workflow
> >>> language,
> >>> > > either
> >>> > > > >>>>> we can
> >>> > > > >>>>> >> improve existing XWF language, totally change to a new
> >>> > language
> >>> > > > >>>>> available
> >>> > > > >>>>> >> in industry or write a new light weight language. Basic
> >>> > > > >>>>>requirements
> >>> > > > >>>>> that
> >>> > > > >>>>> >> we expect from new improvement are, high usability,
> >>> flexible,
> >>> > > > >>>>>light
> >>> > > > >>>>> weight
> >>> > > > >>>>> >> and real time monitoring support. As you can see above
> >>> > > > >>>>>requirements
> >>> > > > >>>>> are
> >>> > > > >>>>> >> not
> >>> > > > >>>>> >> direct comes with workflow languages but we need
> workflow
> >>> > > language
> >>> > > > >>>>> which
> >>> > > > >>>>> >> help to support above requirements.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> After reading few papers and googling, initially i have
> >>> come
> >>> > up
> >>> > > > >>>>>with
> >>> > > > >>>>> >> following three existing languages,
> >>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>> > > > >>>>> >> 2. WS-BPEL
> >>> > > > >>>>> >> 3. SIDL
> >>> > > > >>>>> >> <
> >>> > > http://computation.llnl.gov/casc/components/index.html#page=home
> >>> > > > >
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific
> >>> domain,
> >>> > > > >>>>> Radical-SAGA
> >>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
> >>> above
> >>> > > > >>>>>three
> >>> > > > >>>>> >> languages we can come up with simple workflow language
> >>> base on
> >>> > > > >>>>> json(or
> >>> > > > >>>>> >> yaml) which support all our requirements for some
> extends.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> It would be grate if I can get more input regarding the
> >>> > $Subject
> >>> > > > >>>>> form the
> >>> > > > >>>>> >> airavata community. You all are more than welcome to
> >>> provide
> >>> > any
> >>> > > > >>>>> type of
> >>> > > > >>>>> >> suggestions.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> Thanks,
> >>> > > > >>>>> >> Shameera.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >>
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> --
> >>> > > > >>>>> >> Best Regards,
> >>> > > > >>>>> >> Shameera Rathnayaka.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT
> gmail.com
> >>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > > >>>>
> >>> > > > >>>>
> >>> > > > >>>> --
> >>> > > > >>>> The best way to predict the future is to invent it - Alan
> Kay
> >>> > > > >>>>
> >>> > > > >>>
> >>> > > > >>>
> >>> > > > >>>
> >>> > > > >>> --
> >>> > > > >>> Best Regards,
> >>> > > > >>> Shameera Rathnayaka.
> >>> > > > >>>
> >>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
> >>> > > > >>>
> >>> > > > >>
> >>> > > > >>
> >>> > > > >>
> >>> > > > >> --
> >>> > > > >> Supun Kamburugamuva
> >>> > > > >> Member, Apache Software Foundation; http://www.apache.org
> >>> > > > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> >>> > > > >> Blog: http://supunk.blogspot.com
> >>> > > > >>
> >>> > > > >>
> >>> > > > >
> >>> > > > >
> >>> > > > >--
> >>> > > > >Supun Kamburugamuva
> >>> > > > >Member, Apache Software Foundation; http://www.apache.org
> >>> > > > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> >>> > > > >Blog: http://supunk.blogspot.com
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > >
> >>> > >
> >>> > > --
> >>> > > Best Regards,
> >>> > > Shameera Rathnayaka.
> >>> > >
> >>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> > > Blog : http://shameerarathnayaka.blogspot.com/
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Supun Kamburugamuva
> >>> > Member, Apache Software Foundation; http://www.apache.org
> >>> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> >>> > Blog: http://supunk.blogspot.com
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards,
> >>> Shameera Rathnayaka.
> >>>
> >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> Blog : http://shameerarathnayaka.blogspot.com/
> >>>
> >>
> >>
> >>
> >> --
> >> Supun Kamburugamuva
> >> Member, Apache Software Foundation; http://www.apache.org
> >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> >> Blog: http://supunk.blogspot.com
> >>
> >>
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi Amila,
Please see my comments inline.
On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara <th...@gmail.com>
wrote:
> Hello,
>
> I am sorry, I am bit late on this thread. But when reading through this
> thread I simply got lost, what this thread is discussing. I have few
> questions.
>
> 1. @Shameera : Is XWF actually a language to define workflow ? To my
> understanding it was an intermediate representation to convert workflow
> defined in UI to java object model. Was XWF ever used by any airavata user
> to define a workflow graph ?
>
Yes, XWF is the language defined and used by Airavata to explain the
workflows but it is not well documented. Both server and client sides read
this description language to generate runtime java representation. so
when you used XBaya to create a workflow with multiple applications, under
the hood XBaya generate xwf which describe that workflow to server.
>
> From initial description what I understood is we are looking for a
> improved "intermediate representation", not a language which describes
> workflows.
>
>
>
> 2. So what is the exact purpose of this proposed language ?
> - Is it to hide parallelism in workflows ?
> - Is it to replace the XBAYA functionalities ? (i hope not)
>
Actually initial idea was to identify good, well-defined and scientific
domain friendly language. Whole purpose of this effort is reduce the entry
barrier of the end user. But later it is understood that introducing a new
language won't fix our issue.
>
> 3. What are we trying to achieve by this proposed language which we cannot
> achieved through workflow UI tool ?
>
> 4. Who is going to use this language ?
>
As I explained, our direction has been changed.By introducing a new
language we are get nothing but nice description file. No functional
improvements etc ... The current language should use all airavata
client(Currently we have only XBaya) side applications to explain the
workflow to the server side.
>
> 5. Why would a user prefer (assuming intended audience of proposed
> language is end users) a language over a Work Flow UI tool such as XBAYA ?
> (In other words what are the things we can do with language which we cannot
> do with UI)
>
Let's say you are going to write a new web base client for Airavata which
generate workflows and launch it. What you need to do is do some magic
with UI and finally come up with description language and send it to server
side. Here you need to learn how to write a valid XWF file and write your
own JS code to generate it. But if airavata has a JavaScript API which can
be used to generate XWF for you by getting JSON objects as input then it
would be great help for client side developers.
Thank,
Shameera.
>
> Sorry, if above questions are in-appropriate, just wanted to understand
> what exactly needed.
>
> Thanks
> -Thejaka Amila
>
>
>
> On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
>> I think I'm not suggesting to create a Workflow interpreter using Python
>> etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
>> and move it to a more higher level component. The more I think about it,
>> the model I'm suggesting is similar to what Hadoop, Storm etc has done for
>> distributed system computations. This model is proven to be successful over
>> the years.
>>
>> Keeping what Airavata does at its core can help you to build a more
>> robust system. If we look at Airavata as middleware to execute applications
>> on computing resources we can simplify what Airavata does and focus on
>> improving the core functionality. All the successful systems have thrived
>> on defining what it does at its core and keeping it simple and being
>> excellent at what it does. In that regard keeping workflow aspect out of
>> Airavata can help you to focus on the core problem. That is to execute an
>> application on a remote computing resource in a fault tolerant and scalable
>> manner.
>>
>> What I'm suggesting is to give the Orchestrator the capability to execute
>> a Driver program that is specified by the user. (This program can be
>> written in Python, Java or any other language). This driver program is
>> similar to what you define in a Hadoop or Storm configuration. The driver
>> program specifies the flow of the computation. It specifies what are the
>> applications needs to be executed, how to manipulate input output. The
>> driver program is the workflow for the user. Because the user specifies the
>> program he can program it to handle workflow steering etc. Every time the
>> user wants to execute this program he can tell Airavata to execute the
>> Driver program.
>>
>> I'm also not 100% sure about all the details. But this can be a new way
>> to look at how systems like Airavata should behave. Your thoughts and
>> suggestions are more than welcome.
>>
>> Thanks,
>> Supun..
>>
>>
>>
>>
>>
>>
>> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>>
>>> Hi Supun,
>>>
>>> I think we were in two context, because I as suggesting a way to
>>> serialize
>>> and deserialize the workflow description while you are suggesting to
>>> implement
>>> some kind of workflow interpreter using Python, where Python client can
>>> send
>>> thrift calls to Airavata server to run the application. I can see with
>>> your
>>> suggested
>>> approach we can control the workflow execution process from client side
>>> which
>>> make it easy to implement workflow debugger.As you mentioned this is a
>>> major change
>>> to Airavata. So we should neatly think as this will change our existing
>>> architecture.
>>>
>>> Still if someone need to use different language java, php, JS etc ... to
>>> run the same
>>> workflow which generated by Python, we need a language independent
>>> workflow
>>> description.
>>> My initial question was what is the best language for this?. But as I
>>> have
>>> explained in
>>> one of my previous reply, It is not matter what language we used Either
>>> we
>>> can use
>>> XML or JSON to write this description, what matters is how easy to
>>> generate
>>> workflow with the provided API. Hence it would be great to have set of
>>> neat
>>> APIs in
>>> different languages.
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <supun06@gmail.com
>>> >
>>> wrote:
>>>
>>> > Hi Shameera,
>>> >
>>> > Using python is a radical approach for workflow I think. But I believe
>>> this
>>> > is a very beautiful and new approach.Here is a possible scenario for
>>> > implementation and running using a Python based library.
>>> >
>>> > The Python library facilitates the creation of Applications and
>>> submitting
>>> > them to Airavata for execution. The underlying functionality is exactly
>>> > similar to what Java clients provides.The only difference is that,
>>> Python
>>> > library should have a more fluent API than Java for easy creation of
>>> > workflows. We can generate the Python clients that talk to Airavata
>>> server
>>> > using Thrift.
>>> >
>>> > Here is an example off the top of my head to a Python script created by
>>> > user for a Workflow. This is a very crude example and we need to come
>>> up
>>> > with a much better API if we are going to go along this path. First we
>>> need
>>> > to write a Python script that can execute a workflow using Airavata.
>>> >
>>> > import airavata
>>> >
>>> > host = Host("localhost", ....)
>>> > app1 = Application(host, ....)
>>> > app2 = Application(host, ....)
>>> >
>>> > # we will connect these applications as a workflow using some topology
>>> > builder or other constructs
>>> >
>>> > wb = WorkFlowBuilder()
>>> > wb.setApp("name1", app1)
>>> > # you can do a simple output transformation here etc
>>> >
>>> > # connects the input of app1 to app2 etc
>>> > wb.setApp("name2", app2).connectInput("name1")
>>> >
>>> > wb.submit()
>>> >
>>> > Now we can load this Python script from XBaya. When XBaya loads this
>>> script
>>> > the Python script can output an XML configuration of the topology,
>>> XBaya
>>> > can render. There are other ways like directly executing the Python
>>> script
>>> > from command line and connecting XBaya indirectly as well. Now you can
>>> run
>>> > the workflow from XBaya. Running the Workflow means just executing the
>>> > Python script.
>>> >
>>> > XBaya gets the notifications through messaging and update the UI
>>> > accordingly.
>>> >
>>> > The users need to write the Python script by hand. XBaya cannot create
>>> the
>>> > script. Because workflow language is an actual python program the
>>> benefits
>>> > are immense. For example user can do workflow steering in the workflow
>>> > itself by subscribing to messages from Airavata.
>>> >
>>> > Thanks,
>>> > Supun..
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>>> > shameerainfo@gmail.com> wrote:
>>> >
>>> > > Hi Supun,
>>> > >
>>> > > I meant to say JS is a well-known client side scripting language i
>>> have
>>> > > messed scripting part. Even we use Python, ultimately we should
>>> convert
>>> > > this to java model in server side, somehow we need to serialized
>>> python
>>> > > representation to the language which java can parse and generate that
>>> > > model. In this case we need to parse python script in java isn't it?
>>> I am
>>> > > not exactly clear how you suggesting to use python here. More
>>> details on
>>> > > how end system works if we use Python would be great help to clearly
>>> > > understand your points.
>>> > >
>>> > > Thanks,
>>> > > Shameera.
>>> > >
>>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>>> > chris.mattmann@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > Have you guys considered using JCC [1] as a means
>>> > > > to expose the workflow API currently in Java as a
>>> > > > Python API?
>>> > > >
>>> > > > We are exploring its use in OODT, and we have already
>>> > > > created a Tika [2] JCC-based python API.
>>> > > >
>>> > > > Cheers,
>>> > > > Chris
>>> > > >
>>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>>> > > > [2] http://github.com/chrismattmann/tika-python/
>>> > > >
>>> > > > ------------------------
>>> > > > Chris Mattmann
>>> > > > chris.mattmann@gmail.com
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > -----Original Message-----
>>> > > > From: Supun Kamburugamuva <su...@gmail.com>
>>> > > > Reply-To: <de...@oodt.apache.org>
>>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>>> > > > To: dev <de...@airavata.apache.org>
>>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
>>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
>>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
>>> > > > <de...@oodt.apache.org>
>>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>>> > Airavata.
>>> > > >
>>> > > > >Once we had an offline discussion about the Airavata Workflow
>>> language
>>> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
>>> came
>>> > > out
>>> > > > >was why we need to invent a different language when a simple
>>> library
>>> > > like
>>> > > > >Python will full fill of Airavata requirements.
>>> > > > >
>>> > > > >There are many benefits in using a Python library as the API for
>>> > > > >controlling Airavata workflows.
>>> > > > >
>>> > > > >1. It is a library, gives the ultimate control over the execution
>>> and
>>> > it
>>> > > > >can be simpler than any domain specific language that we can come
>>> with
>>> > > > >like
>>> > > > >XML, JSON etc
>>> > > > >2. Most people use python and can learn it easily than any Domain
>>> > > specific
>>> > > > >language
>>> > > > >3. You can easily create a Python library for Airavata because
>>> all the
>>> > > > >APIs
>>> > > > >are thrift based.
>>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>>> > > > >
>>> > > > >Any thoughts?
>>> > > > >
>>> > > > >Thanks,
>>> > > > >Supun..
>>> > > > >
>>> > > > >
>>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
>>> > supun06@gmail.com
>>> > > >
>>> > > > >wrote:
>>> > > > >
>>> > > > >> Hi Shameera,
>>> > > > >>
>>> > > > >> Why you prefer JavaScript over a language like Python?
>>> > > > >>
>>> > > > >> Thanks,
>>> > > > >> Supun..
>>> > > > >>
>>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>>> > > > >> shameerainfo@gmail.com> wrote:
>>> > > > >>
>>> > > > >>> Hi,
>>> > > > >>>
>>> > > > >>> First of all thanks everyone for giving valuable inputs. After
>>> > doing
>>> > > > >>>some
>>> > > > >>> background search and talking to different people in the
>>> University
>>> > > > >>>who has
>>> > > > >>> used different workflow languages, I myself convinced that
>>> > > introducing
>>> > > > >>>an
>>> > > > >>> another workflow language is not what actually they need. By
>>> > changing
>>> > > > >>> exiting workflow language to another will not solve problems.
>>> What
>>> > > they
>>> > > > >>> asking is a easy way to construct the workflows. Indirectly
>>> what
>>> > they
>>> > > > >>> asking for a sort of API which they can use to generate the
>>> > workflows
>>> > > > >>>and
>>> > > > >>> run it. Correct me if i am wrong here.
>>> > > > >>>
>>> > > > >>> As most of above replies depict, if we can get a simple API,
>>> as an
>>> > > > >>> example, for a web based application, JavaScript API would be a
>>> > good
>>> > > > >>> solution, and probably JSON would be a good candidate for
>>> language,
>>> > > > >>>instead
>>> > > > >>> of XML.
>>> > > > >>>
>>> > > > >>> Airavata community already have started to implement web base
>>> GUI.
>>> > > > >>>Hence
>>> > > > >>> introducing a JSON base JavaScript API would be great help.
>>> WDYT?
>>> > > > >>>
>>> > > > >>> Thanks,
>>> > > > >>> Shameera.
>>> > > > >>>
>>> > > > >>>
>>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> > > > >>> alekny7@gmail.com> wrote:
>>> > > > >>>
>>> > > > >>>> Hi,
>>> > > > >>>>
>>> > > > >>>> it is not dataflow instead focused on orchestrating REST
>>> services
>>> > > but
>>> > > > >>>> you may find it useful datapoint - we created worfklow service
>>> > that
>>> > > > >>>>uses
>>> > > > >>>> natively JavaScript and JSON to describe what happens during
>>> > > workflow
>>> > > > >>>> execution:
>>> > > > >>>>
>>> > > >
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>> > > > >>>>
>>> > > > >>>> HTH,
>>> > > > >>>>
>>> > > > >>>> Alek
>>> > > > >>>>
>>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
>>> smarru@apache.org>
>>> > > > >>>>wrote:
>>> > > > >>>>
>>> > > > >>>>> Hi Chris,
>>> > > > >>>>>
>>> > > > >>>>> Great to hear OODT community will be interested in adopting a
>>> > JSON
>>> > > > >>>>> based workflow language and potentially a web based composer
>>> as
>>> > > well.
>>> > > > >>>>> Airavata previously had BPEL support initially through a home
>>> > grown
>>> > > > >>>>> implementation [1] by Alek Slominski and later through
>>> Apache ODE
>>> > > > >>>>>[2]. Also
>>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
>>> read.
>>> > > > >>>>>
>>> > > > >>>>> I am of the same opinion that we should adopt something more
>>> > modern
>>> > > > >>>>>as
>>> > > > >>>>> the challenges from scientific workflows seems to be
>>> converging
>>> > > with
>>> > > > >>>>>the
>>> > > > >>>>> data flow patterns in business workflows.
>>> > > > >>>>>
>>> > > > >>>>> It will be great if we can all compile a list of potential
>>> > > candidates
>>> > > > >>>>> and hack them through.
>>> > > > >>>>>
>>> > > > >>>>> Suresh
>>> > > > >>>>> [1] -
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > >
>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>> > > > >>>>> [2] -
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > >
>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>>> > > > >>>>>time_for_eScience_workflows
>>> > > > >>>>> [3] -
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > >
>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>>> > > > >>>>>6.pdf
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>> > > > >>>>>
>>> > > > >>>>> > Hi Guys,
>>> > > > >>>>> >
>>> > > > >>>>> > I've been interested in this too - we don't per have a
>>> specific
>>> > > > >>>>> > OODT workflow language, but we specific workflows using
>>> XML,
>>> > and
>>> > > > >>>>> > other configuration (we are also thinking of moving to
>>> JSON for
>>> > > > >>>>> > this).
>>> > > > >>>>> >
>>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
>>> > complex
>>> > > > >>>>> > to me.
>>> > > > >>>>> >
>>> > > > >>>>> > I wonder at the end of the day if we should adopt something
>>> > more
>>> > > > >>>>> > modern like PIG or some other data flow type of language
>>> (PIG
>>> > > > >>>>> > is really neat).
>>> > > > >>>>> >
>>> > > > >>>>> > Cheers,
>>> > > > >>>>> > Chris
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > > > >>>>> > Chris Mattmann, Ph.D.
>>> > > > >>>>> > Chief Architect
>>> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
>>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
>>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
>>> > > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
>>> > > > >>>>> >
>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
>>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089
>>> USA
>>> > > > >>>>> >
>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> > -----Original Message-----
>>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
>>> > > > >>>>> > <ar...@airavata.apache.org>
>>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>> > > > >>>>> > To: "architecture@airavata.apache.org" <
>>> > > > >>>>> architecture@airavata.apache.org>,
>>> > > > >>>>> > dev <de...@airavata.apache.org>
>>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
>>> > > > >>>>>Airavata.
>>> > > > >>>>> >
>>> > > > >>>>> >> Hi All,
>>> > > > >>>>> >>
>>> > > > >>>>> >> As we all know Airavata has its own workflow language call
>>> > XWF.
>>> > > > >>>>>When
>>> > > > >>>>> XWF
>>> > > > >>>>> >> was introduced, main focus points are interoperability and
>>> > > > >>>>> convertibility.
>>> > > > >>>>> >> But with years of experience it is convinced that above
>>> > > > >>>>>requirements
>>> > > > >>>>> are
>>> > > > >>>>> >> not really useful when we come to real world use cases.
>>> And
>>> > XWF
>>> > > is
>>> > > > >>>>> XML
>>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
>>> image
>>> > > it
>>> > > > >>>>> self.
>>> > > > >>>>> >> But
>>> > > > >>>>> >> with the recent changes WSDL part is being removed from
>>> XWF.
>>> > > > >>>>> >>
>>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
>>> languages in
>>> > > > >>>>> industry
>>> > > > >>>>> >> and
>>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
>>> need
>>> > to
>>> > > > >>>>> come up
>>> > > > >>>>> >> with idea how we should improve Airavata workflow
>>> language,
>>> > > either
>>> > > > >>>>> we can
>>> > > > >>>>> >> improve existing XWF language, totally change to a new
>>> > language
>>> > > > >>>>> available
>>> > > > >>>>> >> in industry or write a new light weight language. Basic
>>> > > > >>>>>requirements
>>> > > > >>>>> that
>>> > > > >>>>> >> we expect from new improvement are, high usability,
>>> flexible,
>>> > > > >>>>>light
>>> > > > >>>>> weight
>>> > > > >>>>> >> and real time monitoring support. As you can see above
>>> > > > >>>>>requirements
>>> > > > >>>>> are
>>> > > > >>>>> >> not
>>> > > > >>>>> >> direct comes with workflow languages but we need workflow
>>> > > language
>>> > > > >>>>> which
>>> > > > >>>>> >> help to support above requirements.
>>> > > > >>>>> >>
>>> > > > >>>>> >> After reading few papers and googling, initially i have
>>> come
>>> > up
>>> > > > >>>>>with
>>> > > > >>>>> >> following three existing languages,
>>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>> > > > >>>>> >> 2. WS-BPEL
>>> > > > >>>>> >> 3. SIDL
>>> > > > >>>>> >> <
>>> > > http://computation.llnl.gov/casc/components/index.html#page=home
>>> > > > >
>>> > > > >>>>> >>
>>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific
>>> domain,
>>> > > > >>>>> Radical-SAGA
>>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
>>> above
>>> > > > >>>>>three
>>> > > > >>>>> >> languages we can come up with simple workflow language
>>> base on
>>> > > > >>>>> json(or
>>> > > > >>>>> >> yaml) which support all our requirements for some extends.
>>> > > > >>>>> >>
>>> > > > >>>>> >> It would be grate if I can get more input regarding the
>>> > $Subject
>>> > > > >>>>> form the
>>> > > > >>>>> >> airavata community. You all are more than welcome to
>>> provide
>>> > any
>>> > > > >>>>> type of
>>> > > > >>>>> >> suggestions.
>>> > > > >>>>> >>
>>> > > > >>>>> >> Thanks,
>>> > > > >>>>> >> Shameera.
>>> > > > >>>>> >>
>>> > > > >>>>> >>
>>> > > > >>>>> >>
>>> > > > >>>>> >> --
>>> > > > >>>>> >> Best Regards,
>>> > > > >>>>> >> Shameera Rathnayaka.
>>> > > > >>>>> >>
>>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > > >>>>
>>> > > > >>>>
>>> > > > >>>> --
>>> > > > >>>> The best way to predict the future is to invent it - Alan Kay
>>> > > > >>>>
>>> > > > >>>
>>> > > > >>>
>>> > > > >>>
>>> > > > >>> --
>>> > > > >>> Best Regards,
>>> > > > >>> Shameera Rathnayaka.
>>> > > > >>>
>>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
>>> > > > >>>
>>> > > > >>
>>> > > > >>
>>> > > > >>
>>> > > > >> --
>>> > > > >> Supun Kamburugamuva
>>> > > > >> Member, Apache Software Foundation; http://www.apache.org
>>> > > > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>> > > > >> Blog: http://supunk.blogspot.com
>>> > > > >>
>>> > > > >>
>>> > > > >
>>> > > > >
>>> > > > >--
>>> > > > >Supun Kamburugamuva
>>> > > > >Member, Apache Software Foundation; http://www.apache.org
>>> > > > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>> > > > >Blog: http://supunk.blogspot.com
>>> > > >
>>> > > >
>>> > > >
>>> > >
>>> > >
>>> > > --
>>> > > Best Regards,
>>> > > Shameera Rathnayaka.
>>> > >
>>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
>>> > > Blog : http://shameerarathnayaka.blogspot.com/
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Supun Kamburugamuva
>>> > Member, Apache Software Foundation; http://www.apache.org
>>> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>> > Blog: http://supunk.blogspot.com
>>> >
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>>
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>
--
Best Regards,
Shameera Rathnayaka.
email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi Amila,
Please see my comments inline.
On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara <th...@gmail.com>
wrote:
> Hello,
>
> I am sorry, I am bit late on this thread. But when reading through this
> thread I simply got lost, what this thread is discussing. I have few
> questions.
>
> 1. @Shameera : Is XWF actually a language to define workflow ? To my
> understanding it was an intermediate representation to convert workflow
> defined in UI to java object model. Was XWF ever used by any airavata user
> to define a workflow graph ?
>
Yes, XWF is the language defined and used by Airavata to explain the
workflows but it is not well documented. Both server and client sides read
this description language to generate runtime java representation. so
when you used XBaya to create a workflow with multiple applications, under
the hood XBaya generate xwf which describe that workflow to server.
>
> From initial description what I understood is we are looking for a
> improved "intermediate representation", not a language which describes
> workflows.
>
>
>
> 2. So what is the exact purpose of this proposed language ?
> - Is it to hide parallelism in workflows ?
> - Is it to replace the XBAYA functionalities ? (i hope not)
>
Actually initial idea was to identify good, well-defined and scientific
domain friendly language. Whole purpose of this effort is reduce the entry
barrier of the end user. But later it is understood that introducing a new
language won't fix our issue.
>
> 3. What are we trying to achieve by this proposed language which we cannot
> achieved through workflow UI tool ?
>
> 4. Who is going to use this language ?
>
As I explained, our direction has been changed.By introducing a new
language we are get nothing but nice description file. No functional
improvements etc ... The current language should use all airavata
client(Currently we have only XBaya) side applications to explain the
workflow to the server side.
>
> 5. Why would a user prefer (assuming intended audience of proposed
> language is end users) a language over a Work Flow UI tool such as XBAYA ?
> (In other words what are the things we can do with language which we cannot
> do with UI)
>
Let's say you are going to write a new web base client for Airavata which
generate workflows and launch it. What you need to do is do some magic
with UI and finally come up with description language and send it to server
side. Here you need to learn how to write a valid XWF file and write your
own JS code to generate it. But if airavata has a JavaScript API which can
be used to generate XWF for you by getting JSON objects as input then it
would be great help for client side developers.
Thank,
Shameera.
>
> Sorry, if above questions are in-appropriate, just wanted to understand
> what exactly needed.
>
> Thanks
> -Thejaka Amila
>
>
>
> On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
>> I think I'm not suggesting to create a Workflow interpreter using Python
>> etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
>> and move it to a more higher level component. The more I think about it,
>> the model I'm suggesting is similar to what Hadoop, Storm etc has done for
>> distributed system computations. This model is proven to be successful over
>> the years.
>>
>> Keeping what Airavata does at its core can help you to build a more
>> robust system. If we look at Airavata as middleware to execute applications
>> on computing resources we can simplify what Airavata does and focus on
>> improving the core functionality. All the successful systems have thrived
>> on defining what it does at its core and keeping it simple and being
>> excellent at what it does. In that regard keeping workflow aspect out of
>> Airavata can help you to focus on the core problem. That is to execute an
>> application on a remote computing resource in a fault tolerant and scalable
>> manner.
>>
>> What I'm suggesting is to give the Orchestrator the capability to execute
>> a Driver program that is specified by the user. (This program can be
>> written in Python, Java or any other language). This driver program is
>> similar to what you define in a Hadoop or Storm configuration. The driver
>> program specifies the flow of the computation. It specifies what are the
>> applications needs to be executed, how to manipulate input output. The
>> driver program is the workflow for the user. Because the user specifies the
>> program he can program it to handle workflow steering etc. Every time the
>> user wants to execute this program he can tell Airavata to execute the
>> Driver program.
>>
>> I'm also not 100% sure about all the details. But this can be a new way
>> to look at how systems like Airavata should behave. Your thoughts and
>> suggestions are more than welcome.
>>
>> Thanks,
>> Supun..
>>
>>
>>
>>
>>
>>
>> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>>
>>> Hi Supun,
>>>
>>> I think we were in two context, because I as suggesting a way to
>>> serialize
>>> and deserialize the workflow description while you are suggesting to
>>> implement
>>> some kind of workflow interpreter using Python, where Python client can
>>> send
>>> thrift calls to Airavata server to run the application. I can see with
>>> your
>>> suggested
>>> approach we can control the workflow execution process from client side
>>> which
>>> make it easy to implement workflow debugger.As you mentioned this is a
>>> major change
>>> to Airavata. So we should neatly think as this will change our existing
>>> architecture.
>>>
>>> Still if someone need to use different language java, php, JS etc ... to
>>> run the same
>>> workflow which generated by Python, we need a language independent
>>> workflow
>>> description.
>>> My initial question was what is the best language for this?. But as I
>>> have
>>> explained in
>>> one of my previous reply, It is not matter what language we used Either
>>> we
>>> can use
>>> XML or JSON to write this description, what matters is how easy to
>>> generate
>>> workflow with the provided API. Hence it would be great to have set of
>>> neat
>>> APIs in
>>> different languages.
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <supun06@gmail.com
>>> >
>>> wrote:
>>>
>>> > Hi Shameera,
>>> >
>>> > Using python is a radical approach for workflow I think. But I believe
>>> this
>>> > is a very beautiful and new approach.Here is a possible scenario for
>>> > implementation and running using a Python based library.
>>> >
>>> > The Python library facilitates the creation of Applications and
>>> submitting
>>> > them to Airavata for execution. The underlying functionality is exactly
>>> > similar to what Java clients provides.The only difference is that,
>>> Python
>>> > library should have a more fluent API than Java for easy creation of
>>> > workflows. We can generate the Python clients that talk to Airavata
>>> server
>>> > using Thrift.
>>> >
>>> > Here is an example off the top of my head to a Python script created by
>>> > user for a Workflow. This is a very crude example and we need to come
>>> up
>>> > with a much better API if we are going to go along this path. First we
>>> need
>>> > to write a Python script that can execute a workflow using Airavata.
>>> >
>>> > import airavata
>>> >
>>> > host = Host("localhost", ....)
>>> > app1 = Application(host, ....)
>>> > app2 = Application(host, ....)
>>> >
>>> > # we will connect these applications as a workflow using some topology
>>> > builder or other constructs
>>> >
>>> > wb = WorkFlowBuilder()
>>> > wb.setApp("name1", app1)
>>> > # you can do a simple output transformation here etc
>>> >
>>> > # connects the input of app1 to app2 etc
>>> > wb.setApp("name2", app2).connectInput("name1")
>>> >
>>> > wb.submit()
>>> >
>>> > Now we can load this Python script from XBaya. When XBaya loads this
>>> script
>>> > the Python script can output an XML configuration of the topology,
>>> XBaya
>>> > can render. There are other ways like directly executing the Python
>>> script
>>> > from command line and connecting XBaya indirectly as well. Now you can
>>> run
>>> > the workflow from XBaya. Running the Workflow means just executing the
>>> > Python script.
>>> >
>>> > XBaya gets the notifications through messaging and update the UI
>>> > accordingly.
>>> >
>>> > The users need to write the Python script by hand. XBaya cannot create
>>> the
>>> > script. Because workflow language is an actual python program the
>>> benefits
>>> > are immense. For example user can do workflow steering in the workflow
>>> > itself by subscribing to messages from Airavata.
>>> >
>>> > Thanks,
>>> > Supun..
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>>> > shameerainfo@gmail.com> wrote:
>>> >
>>> > > Hi Supun,
>>> > >
>>> > > I meant to say JS is a well-known client side scripting language i
>>> have
>>> > > messed scripting part. Even we use Python, ultimately we should
>>> convert
>>> > > this to java model in server side, somehow we need to serialized
>>> python
>>> > > representation to the language which java can parse and generate that
>>> > > model. In this case we need to parse python script in java isn't it?
>>> I am
>>> > > not exactly clear how you suggesting to use python here. More
>>> details on
>>> > > how end system works if we use Python would be great help to clearly
>>> > > understand your points.
>>> > >
>>> > > Thanks,
>>> > > Shameera.
>>> > >
>>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>>> > chris.mattmann@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > Have you guys considered using JCC [1] as a means
>>> > > > to expose the workflow API currently in Java as a
>>> > > > Python API?
>>> > > >
>>> > > > We are exploring its use in OODT, and we have already
>>> > > > created a Tika [2] JCC-based python API.
>>> > > >
>>> > > > Cheers,
>>> > > > Chris
>>> > > >
>>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>>> > > > [2] http://github.com/chrismattmann/tika-python/
>>> > > >
>>> > > > ------------------------
>>> > > > Chris Mattmann
>>> > > > chris.mattmann@gmail.com
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > -----Original Message-----
>>> > > > From: Supun Kamburugamuva <su...@gmail.com>
>>> > > > Reply-To: <de...@oodt.apache.org>
>>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>>> > > > To: dev <de...@airavata.apache.org>
>>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
>>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
>>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
>>> > > > <de...@oodt.apache.org>
>>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>>> > Airavata.
>>> > > >
>>> > > > >Once we had an offline discussion about the Airavata Workflow
>>> language
>>> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
>>> came
>>> > > out
>>> > > > >was why we need to invent a different language when a simple
>>> library
>>> > > like
>>> > > > >Python will full fill of Airavata requirements.
>>> > > > >
>>> > > > >There are many benefits in using a Python library as the API for
>>> > > > >controlling Airavata workflows.
>>> > > > >
>>> > > > >1. It is a library, gives the ultimate control over the execution
>>> and
>>> > it
>>> > > > >can be simpler than any domain specific language that we can come
>>> with
>>> > > > >like
>>> > > > >XML, JSON etc
>>> > > > >2. Most people use python and can learn it easily than any Domain
>>> > > specific
>>> > > > >language
>>> > > > >3. You can easily create a Python library for Airavata because
>>> all the
>>> > > > >APIs
>>> > > > >are thrift based.
>>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>>> > > > >
>>> > > > >Any thoughts?
>>> > > > >
>>> > > > >Thanks,
>>> > > > >Supun..
>>> > > > >
>>> > > > >
>>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
>>> > supun06@gmail.com
>>> > > >
>>> > > > >wrote:
>>> > > > >
>>> > > > >> Hi Shameera,
>>> > > > >>
>>> > > > >> Why you prefer JavaScript over a language like Python?
>>> > > > >>
>>> > > > >> Thanks,
>>> > > > >> Supun..
>>> > > > >>
>>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>>> > > > >> shameerainfo@gmail.com> wrote:
>>> > > > >>
>>> > > > >>> Hi,
>>> > > > >>>
>>> > > > >>> First of all thanks everyone for giving valuable inputs. After
>>> > doing
>>> > > > >>>some
>>> > > > >>> background search and talking to different people in the
>>> University
>>> > > > >>>who has
>>> > > > >>> used different workflow languages, I myself convinced that
>>> > > introducing
>>> > > > >>>an
>>> > > > >>> another workflow language is not what actually they need. By
>>> > changing
>>> > > > >>> exiting workflow language to another will not solve problems.
>>> What
>>> > > they
>>> > > > >>> asking is a easy way to construct the workflows. Indirectly
>>> what
>>> > they
>>> > > > >>> asking for a sort of API which they can use to generate the
>>> > workflows
>>> > > > >>>and
>>> > > > >>> run it. Correct me if i am wrong here.
>>> > > > >>>
>>> > > > >>> As most of above replies depict, if we can get a simple API,
>>> as an
>>> > > > >>> example, for a web based application, JavaScript API would be a
>>> > good
>>> > > > >>> solution, and probably JSON would be a good candidate for
>>> language,
>>> > > > >>>instead
>>> > > > >>> of XML.
>>> > > > >>>
>>> > > > >>> Airavata community already have started to implement web base
>>> GUI.
>>> > > > >>>Hence
>>> > > > >>> introducing a JSON base JavaScript API would be great help.
>>> WDYT?
>>> > > > >>>
>>> > > > >>> Thanks,
>>> > > > >>> Shameera.
>>> > > > >>>
>>> > > > >>>
>>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> > > > >>> alekny7@gmail.com> wrote:
>>> > > > >>>
>>> > > > >>>> Hi,
>>> > > > >>>>
>>> > > > >>>> it is not dataflow instead focused on orchestrating REST
>>> services
>>> > > but
>>> > > > >>>> you may find it useful datapoint - we created worfklow service
>>> > that
>>> > > > >>>>uses
>>> > > > >>>> natively JavaScript and JSON to describe what happens during
>>> > > workflow
>>> > > > >>>> execution:
>>> > > > >>>>
>>> > > >
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>> > > > >>>>
>>> > > > >>>> HTH,
>>> > > > >>>>
>>> > > > >>>> Alek
>>> > > > >>>>
>>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
>>> smarru@apache.org>
>>> > > > >>>>wrote:
>>> > > > >>>>
>>> > > > >>>>> Hi Chris,
>>> > > > >>>>>
>>> > > > >>>>> Great to hear OODT community will be interested in adopting a
>>> > JSON
>>> > > > >>>>> based workflow language and potentially a web based composer
>>> as
>>> > > well.
>>> > > > >>>>> Airavata previously had BPEL support initially through a home
>>> > grown
>>> > > > >>>>> implementation [1] by Alek Slominski and later through
>>> Apache ODE
>>> > > > >>>>>[2]. Also
>>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
>>> read.
>>> > > > >>>>>
>>> > > > >>>>> I am of the same opinion that we should adopt something more
>>> > modern
>>> > > > >>>>>as
>>> > > > >>>>> the challenges from scientific workflows seems to be
>>> converging
>>> > > with
>>> > > > >>>>>the
>>> > > > >>>>> data flow patterns in business workflows.
>>> > > > >>>>>
>>> > > > >>>>> It will be great if we can all compile a list of potential
>>> > > candidates
>>> > > > >>>>> and hack them through.
>>> > > > >>>>>
>>> > > > >>>>> Suresh
>>> > > > >>>>> [1] -
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > >
>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>> > > > >>>>> [2] -
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > >
>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>>> > > > >>>>>time_for_eScience_workflows
>>> > > > >>>>> [3] -
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > >
>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>>> > > > >>>>>6.pdf
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>> > > > >>>>>
>>> > > > >>>>> > Hi Guys,
>>> > > > >>>>> >
>>> > > > >>>>> > I've been interested in this too - we don't per have a
>>> specific
>>> > > > >>>>> > OODT workflow language, but we specific workflows using
>>> XML,
>>> > and
>>> > > > >>>>> > other configuration (we are also thinking of moving to
>>> JSON for
>>> > > > >>>>> > this).
>>> > > > >>>>> >
>>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
>>> > complex
>>> > > > >>>>> > to me.
>>> > > > >>>>> >
>>> > > > >>>>> > I wonder at the end of the day if we should adopt something
>>> > more
>>> > > > >>>>> > modern like PIG or some other data flow type of language
>>> (PIG
>>> > > > >>>>> > is really neat).
>>> > > > >>>>> >
>>> > > > >>>>> > Cheers,
>>> > > > >>>>> > Chris
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > > > >>>>> > Chris Mattmann, Ph.D.
>>> > > > >>>>> > Chief Architect
>>> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
>>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
>>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
>>> > > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
>>> > > > >>>>> >
>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
>>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089
>>> USA
>>> > > > >>>>> >
>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> > -----Original Message-----
>>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
>>> > > > >>>>> > <ar...@airavata.apache.org>
>>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>> > > > >>>>> > To: "architecture@airavata.apache.org" <
>>> > > > >>>>> architecture@airavata.apache.org>,
>>> > > > >>>>> > dev <de...@airavata.apache.org>
>>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
>>> > > > >>>>>Airavata.
>>> > > > >>>>> >
>>> > > > >>>>> >> Hi All,
>>> > > > >>>>> >>
>>> > > > >>>>> >> As we all know Airavata has its own workflow language call
>>> > XWF.
>>> > > > >>>>>When
>>> > > > >>>>> XWF
>>> > > > >>>>> >> was introduced, main focus points are interoperability and
>>> > > > >>>>> convertibility.
>>> > > > >>>>> >> But with years of experience it is convinced that above
>>> > > > >>>>>requirements
>>> > > > >>>>> are
>>> > > > >>>>> >> not really useful when we come to real world use cases.
>>> And
>>> > XWF
>>> > > is
>>> > > > >>>>> XML
>>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
>>> image
>>> > > it
>>> > > > >>>>> self.
>>> > > > >>>>> >> But
>>> > > > >>>>> >> with the recent changes WSDL part is being removed from
>>> XWF.
>>> > > > >>>>> >>
>>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
>>> languages in
>>> > > > >>>>> industry
>>> > > > >>>>> >> and
>>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
>>> need
>>> > to
>>> > > > >>>>> come up
>>> > > > >>>>> >> with idea how we should improve Airavata workflow
>>> language,
>>> > > either
>>> > > > >>>>> we can
>>> > > > >>>>> >> improve existing XWF language, totally change to a new
>>> > language
>>> > > > >>>>> available
>>> > > > >>>>> >> in industry or write a new light weight language. Basic
>>> > > > >>>>>requirements
>>> > > > >>>>> that
>>> > > > >>>>> >> we expect from new improvement are, high usability,
>>> flexible,
>>> > > > >>>>>light
>>> > > > >>>>> weight
>>> > > > >>>>> >> and real time monitoring support. As you can see above
>>> > > > >>>>>requirements
>>> > > > >>>>> are
>>> > > > >>>>> >> not
>>> > > > >>>>> >> direct comes with workflow languages but we need workflow
>>> > > language
>>> > > > >>>>> which
>>> > > > >>>>> >> help to support above requirements.
>>> > > > >>>>> >>
>>> > > > >>>>> >> After reading few papers and googling, initially i have
>>> come
>>> > up
>>> > > > >>>>>with
>>> > > > >>>>> >> following three existing languages,
>>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>> > > > >>>>> >> 2. WS-BPEL
>>> > > > >>>>> >> 3. SIDL
>>> > > > >>>>> >> <
>>> > > http://computation.llnl.gov/casc/components/index.html#page=home
>>> > > > >
>>> > > > >>>>> >>
>>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific
>>> domain,
>>> > > > >>>>> Radical-SAGA
>>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
>>> above
>>> > > > >>>>>three
>>> > > > >>>>> >> languages we can come up with simple workflow language
>>> base on
>>> > > > >>>>> json(or
>>> > > > >>>>> >> yaml) which support all our requirements for some extends.
>>> > > > >>>>> >>
>>> > > > >>>>> >> It would be grate if I can get more input regarding the
>>> > $Subject
>>> > > > >>>>> form the
>>> > > > >>>>> >> airavata community. You all are more than welcome to
>>> provide
>>> > any
>>> > > > >>>>> type of
>>> > > > >>>>> >> suggestions.
>>> > > > >>>>> >>
>>> > > > >>>>> >> Thanks,
>>> > > > >>>>> >> Shameera.
>>> > > > >>>>> >>
>>> > > > >>>>> >>
>>> > > > >>>>> >>
>>> > > > >>>>> >> --
>>> > > > >>>>> >> Best Regards,
>>> > > > >>>>> >> Shameera Rathnayaka.
>>> > > > >>>>> >>
>>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > > >>>>
>>> > > > >>>>
>>> > > > >>>> --
>>> > > > >>>> The best way to predict the future is to invent it - Alan Kay
>>> > > > >>>>
>>> > > > >>>
>>> > > > >>>
>>> > > > >>>
>>> > > > >>> --
>>> > > > >>> Best Regards,
>>> > > > >>> Shameera Rathnayaka.
>>> > > > >>>
>>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
>>> > > > >>>
>>> > > > >>
>>> > > > >>
>>> > > > >>
>>> > > > >> --
>>> > > > >> Supun Kamburugamuva
>>> > > > >> Member, Apache Software Foundation; http://www.apache.org
>>> > > > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>> > > > >> Blog: http://supunk.blogspot.com
>>> > > > >>
>>> > > > >>
>>> > > > >
>>> > > > >
>>> > > > >--
>>> > > > >Supun Kamburugamuva
>>> > > > >Member, Apache Software Foundation; http://www.apache.org
>>> > > > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>> > > > >Blog: http://supunk.blogspot.com
>>> > > >
>>> > > >
>>> > > >
>>> > >
>>> > >
>>> > > --
>>> > > Best Regards,
>>> > > Shameera Rathnayaka.
>>> > >
>>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
>>> > > Blog : http://shameerarathnayaka.blogspot.com/
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Supun Kamburugamuva
>>> > Member, Apache Software Foundation; http://www.apache.org
>>> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>> > Blog: http://supunk.blogspot.com
>>> >
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>>
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>
--
Best Regards,
Shameera Rathnayaka.
email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi Amila,
Please see my comments inline.
On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara <th...@gmail.com>
wrote:
> Hello,
>
> I am sorry, I am bit late on this thread. But when reading through this
> thread I simply got lost, what this thread is discussing. I have few
> questions.
>
> 1. @Shameera : Is XWF actually a language to define workflow ? To my
> understanding it was an intermediate representation to convert workflow
> defined in UI to java object model. Was XWF ever used by any airavata user
> to define a workflow graph ?
>
Yes, XWF is the language defined and used by Airavata to explain the
workflows but it is not well documented. Both server and client sides read
this description language to generate runtime java representation. so
when you used XBaya to create a workflow with multiple applications, under
the hood XBaya generate xwf which describe that workflow to server.
>
> From initial description what I understood is we are looking for a
> improved "intermediate representation", not a language which describes
> workflows.
>
>
>
> 2. So what is the exact purpose of this proposed language ?
> - Is it to hide parallelism in workflows ?
> - Is it to replace the XBAYA functionalities ? (i hope not)
>
Actually initial idea was to identify good, well-defined and scientific
domain friendly language. Whole purpose of this effort is reduce the entry
barrier of the end user. But later it is understood that introducing a new
language won't fix our issue.
>
> 3. What are we trying to achieve by this proposed language which we cannot
> achieved through workflow UI tool ?
>
> 4. Who is going to use this language ?
>
As I explained, our direction has been changed.By introducing a new
language we are get nothing but nice description file. No functional
improvements etc ... The current language should use all airavata
client(Currently we have only XBaya) side applications to explain the
workflow to the server side.
>
> 5. Why would a user prefer (assuming intended audience of proposed
> language is end users) a language over a Work Flow UI tool such as XBAYA ?
> (In other words what are the things we can do with language which we cannot
> do with UI)
>
Let's say you are going to write a new web base client for Airavata which
generate workflows and launch it. What you need to do is do some magic
with UI and finally come up with description language and send it to server
side. Here you need to learn how to write a valid XWF file and write your
own JS code to generate it. But if airavata has a JavaScript API which can
be used to generate XWF for you by getting JSON objects as input then it
would be great help for client side developers.
Thank,
Shameera.
>
> Sorry, if above questions are in-appropriate, just wanted to understand
> what exactly needed.
>
> Thanks
> -Thejaka Amila
>
>
>
> On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
>> I think I'm not suggesting to create a Workflow interpreter using Python
>> etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
>> and move it to a more higher level component. The more I think about it,
>> the model I'm suggesting is similar to what Hadoop, Storm etc has done for
>> distributed system computations. This model is proven to be successful over
>> the years.
>>
>> Keeping what Airavata does at its core can help you to build a more
>> robust system. If we look at Airavata as middleware to execute applications
>> on computing resources we can simplify what Airavata does and focus on
>> improving the core functionality. All the successful systems have thrived
>> on defining what it does at its core and keeping it simple and being
>> excellent at what it does. In that regard keeping workflow aspect out of
>> Airavata can help you to focus on the core problem. That is to execute an
>> application on a remote computing resource in a fault tolerant and scalable
>> manner.
>>
>> What I'm suggesting is to give the Orchestrator the capability to execute
>> a Driver program that is specified by the user. (This program can be
>> written in Python, Java or any other language). This driver program is
>> similar to what you define in a Hadoop or Storm configuration. The driver
>> program specifies the flow of the computation. It specifies what are the
>> applications needs to be executed, how to manipulate input output. The
>> driver program is the workflow for the user. Because the user specifies the
>> program he can program it to handle workflow steering etc. Every time the
>> user wants to execute this program he can tell Airavata to execute the
>> Driver program.
>>
>> I'm also not 100% sure about all the details. But this can be a new way
>> to look at how systems like Airavata should behave. Your thoughts and
>> suggestions are more than welcome.
>>
>> Thanks,
>> Supun..
>>
>>
>>
>>
>>
>>
>> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>>
>>> Hi Supun,
>>>
>>> I think we were in two context, because I as suggesting a way to
>>> serialize
>>> and deserialize the workflow description while you are suggesting to
>>> implement
>>> some kind of workflow interpreter using Python, where Python client can
>>> send
>>> thrift calls to Airavata server to run the application. I can see with
>>> your
>>> suggested
>>> approach we can control the workflow execution process from client side
>>> which
>>> make it easy to implement workflow debugger.As you mentioned this is a
>>> major change
>>> to Airavata. So we should neatly think as this will change our existing
>>> architecture.
>>>
>>> Still if someone need to use different language java, php, JS etc ... to
>>> run the same
>>> workflow which generated by Python, we need a language independent
>>> workflow
>>> description.
>>> My initial question was what is the best language for this?. But as I
>>> have
>>> explained in
>>> one of my previous reply, It is not matter what language we used Either
>>> we
>>> can use
>>> XML or JSON to write this description, what matters is how easy to
>>> generate
>>> workflow with the provided API. Hence it would be great to have set of
>>> neat
>>> APIs in
>>> different languages.
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <supun06@gmail.com
>>> >
>>> wrote:
>>>
>>> > Hi Shameera,
>>> >
>>> > Using python is a radical approach for workflow I think. But I believe
>>> this
>>> > is a very beautiful and new approach.Here is a possible scenario for
>>> > implementation and running using a Python based library.
>>> >
>>> > The Python library facilitates the creation of Applications and
>>> submitting
>>> > them to Airavata for execution. The underlying functionality is exactly
>>> > similar to what Java clients provides.The only difference is that,
>>> Python
>>> > library should have a more fluent API than Java for easy creation of
>>> > workflows. We can generate the Python clients that talk to Airavata
>>> server
>>> > using Thrift.
>>> >
>>> > Here is an example off the top of my head to a Python script created by
>>> > user for a Workflow. This is a very crude example and we need to come
>>> up
>>> > with a much better API if we are going to go along this path. First we
>>> need
>>> > to write a Python script that can execute a workflow using Airavata.
>>> >
>>> > import airavata
>>> >
>>> > host = Host("localhost", ....)
>>> > app1 = Application(host, ....)
>>> > app2 = Application(host, ....)
>>> >
>>> > # we will connect these applications as a workflow using some topology
>>> > builder or other constructs
>>> >
>>> > wb = WorkFlowBuilder()
>>> > wb.setApp("name1", app1)
>>> > # you can do a simple output transformation here etc
>>> >
>>> > # connects the input of app1 to app2 etc
>>> > wb.setApp("name2", app2).connectInput("name1")
>>> >
>>> > wb.submit()
>>> >
>>> > Now we can load this Python script from XBaya. When XBaya loads this
>>> script
>>> > the Python script can output an XML configuration of the topology,
>>> XBaya
>>> > can render. There are other ways like directly executing the Python
>>> script
>>> > from command line and connecting XBaya indirectly as well. Now you can
>>> run
>>> > the workflow from XBaya. Running the Workflow means just executing the
>>> > Python script.
>>> >
>>> > XBaya gets the notifications through messaging and update the UI
>>> > accordingly.
>>> >
>>> > The users need to write the Python script by hand. XBaya cannot create
>>> the
>>> > script. Because workflow language is an actual python program the
>>> benefits
>>> > are immense. For example user can do workflow steering in the workflow
>>> > itself by subscribing to messages from Airavata.
>>> >
>>> > Thanks,
>>> > Supun..
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>>> > shameerainfo@gmail.com> wrote:
>>> >
>>> > > Hi Supun,
>>> > >
>>> > > I meant to say JS is a well-known client side scripting language i
>>> have
>>> > > messed scripting part. Even we use Python, ultimately we should
>>> convert
>>> > > this to java model in server side, somehow we need to serialized
>>> python
>>> > > representation to the language which java can parse and generate that
>>> > > model. In this case we need to parse python script in java isn't it?
>>> I am
>>> > > not exactly clear how you suggesting to use python here. More
>>> details on
>>> > > how end system works if we use Python would be great help to clearly
>>> > > understand your points.
>>> > >
>>> > > Thanks,
>>> > > Shameera.
>>> > >
>>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>>> > chris.mattmann@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > Have you guys considered using JCC [1] as a means
>>> > > > to expose the workflow API currently in Java as a
>>> > > > Python API?
>>> > > >
>>> > > > We are exploring its use in OODT, and we have already
>>> > > > created a Tika [2] JCC-based python API.
>>> > > >
>>> > > > Cheers,
>>> > > > Chris
>>> > > >
>>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>>> > > > [2] http://github.com/chrismattmann/tika-python/
>>> > > >
>>> > > > ------------------------
>>> > > > Chris Mattmann
>>> > > > chris.mattmann@gmail.com
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > -----Original Message-----
>>> > > > From: Supun Kamburugamuva <su...@gmail.com>
>>> > > > Reply-To: <de...@oodt.apache.org>
>>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>>> > > > To: dev <de...@airavata.apache.org>
>>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
>>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
>>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
>>> > > > <de...@oodt.apache.org>
>>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>>> > Airavata.
>>> > > >
>>> > > > >Once we had an offline discussion about the Airavata Workflow
>>> language
>>> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
>>> came
>>> > > out
>>> > > > >was why we need to invent a different language when a simple
>>> library
>>> > > like
>>> > > > >Python will full fill of Airavata requirements.
>>> > > > >
>>> > > > >There are many benefits in using a Python library as the API for
>>> > > > >controlling Airavata workflows.
>>> > > > >
>>> > > > >1. It is a library, gives the ultimate control over the execution
>>> and
>>> > it
>>> > > > >can be simpler than any domain specific language that we can come
>>> with
>>> > > > >like
>>> > > > >XML, JSON etc
>>> > > > >2. Most people use python and can learn it easily than any Domain
>>> > > specific
>>> > > > >language
>>> > > > >3. You can easily create a Python library for Airavata because
>>> all the
>>> > > > >APIs
>>> > > > >are thrift based.
>>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>>> > > > >
>>> > > > >Any thoughts?
>>> > > > >
>>> > > > >Thanks,
>>> > > > >Supun..
>>> > > > >
>>> > > > >
>>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
>>> > supun06@gmail.com
>>> > > >
>>> > > > >wrote:
>>> > > > >
>>> > > > >> Hi Shameera,
>>> > > > >>
>>> > > > >> Why you prefer JavaScript over a language like Python?
>>> > > > >>
>>> > > > >> Thanks,
>>> > > > >> Supun..
>>> > > > >>
>>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>>> > > > >> shameerainfo@gmail.com> wrote:
>>> > > > >>
>>> > > > >>> Hi,
>>> > > > >>>
>>> > > > >>> First of all thanks everyone for giving valuable inputs. After
>>> > doing
>>> > > > >>>some
>>> > > > >>> background search and talking to different people in the
>>> University
>>> > > > >>>who has
>>> > > > >>> used different workflow languages, I myself convinced that
>>> > > introducing
>>> > > > >>>an
>>> > > > >>> another workflow language is not what actually they need. By
>>> > changing
>>> > > > >>> exiting workflow language to another will not solve problems.
>>> What
>>> > > they
>>> > > > >>> asking is a easy way to construct the workflows. Indirectly
>>> what
>>> > they
>>> > > > >>> asking for a sort of API which they can use to generate the
>>> > workflows
>>> > > > >>>and
>>> > > > >>> run it. Correct me if i am wrong here.
>>> > > > >>>
>>> > > > >>> As most of above replies depict, if we can get a simple API,
>>> as an
>>> > > > >>> example, for a web based application, JavaScript API would be a
>>> > good
>>> > > > >>> solution, and probably JSON would be a good candidate for
>>> language,
>>> > > > >>>instead
>>> > > > >>> of XML.
>>> > > > >>>
>>> > > > >>> Airavata community already have started to implement web base
>>> GUI.
>>> > > > >>>Hence
>>> > > > >>> introducing a JSON base JavaScript API would be great help.
>>> WDYT?
>>> > > > >>>
>>> > > > >>> Thanks,
>>> > > > >>> Shameera.
>>> > > > >>>
>>> > > > >>>
>>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> > > > >>> alekny7@gmail.com> wrote:
>>> > > > >>>
>>> > > > >>>> Hi,
>>> > > > >>>>
>>> > > > >>>> it is not dataflow instead focused on orchestrating REST
>>> services
>>> > > but
>>> > > > >>>> you may find it useful datapoint - we created worfklow service
>>> > that
>>> > > > >>>>uses
>>> > > > >>>> natively JavaScript and JSON to describe what happens during
>>> > > workflow
>>> > > > >>>> execution:
>>> > > > >>>>
>>> > > >
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>> > > > >>>>
>>> > > > >>>> HTH,
>>> > > > >>>>
>>> > > > >>>> Alek
>>> > > > >>>>
>>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
>>> smarru@apache.org>
>>> > > > >>>>wrote:
>>> > > > >>>>
>>> > > > >>>>> Hi Chris,
>>> > > > >>>>>
>>> > > > >>>>> Great to hear OODT community will be interested in adopting a
>>> > JSON
>>> > > > >>>>> based workflow language and potentially a web based composer
>>> as
>>> > > well.
>>> > > > >>>>> Airavata previously had BPEL support initially through a home
>>> > grown
>>> > > > >>>>> implementation [1] by Alek Slominski and later through
>>> Apache ODE
>>> > > > >>>>>[2]. Also
>>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
>>> read.
>>> > > > >>>>>
>>> > > > >>>>> I am of the same opinion that we should adopt something more
>>> > modern
>>> > > > >>>>>as
>>> > > > >>>>> the challenges from scientific workflows seems to be
>>> converging
>>> > > with
>>> > > > >>>>>the
>>> > > > >>>>> data flow patterns in business workflows.
>>> > > > >>>>>
>>> > > > >>>>> It will be great if we can all compile a list of potential
>>> > > candidates
>>> > > > >>>>> and hack them through.
>>> > > > >>>>>
>>> > > > >>>>> Suresh
>>> > > > >>>>> [1] -
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > >
>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>> > > > >>>>> [2] -
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > >
>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>>> > > > >>>>>time_for_eScience_workflows
>>> > > > >>>>> [3] -
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > >
>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>>> > > > >>>>>6.pdf
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>> > > > >>>>>
>>> > > > >>>>> > Hi Guys,
>>> > > > >>>>> >
>>> > > > >>>>> > I've been interested in this too - we don't per have a
>>> specific
>>> > > > >>>>> > OODT workflow language, but we specific workflows using
>>> XML,
>>> > and
>>> > > > >>>>> > other configuration (we are also thinking of moving to
>>> JSON for
>>> > > > >>>>> > this).
>>> > > > >>>>> >
>>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
>>> > complex
>>> > > > >>>>> > to me.
>>> > > > >>>>> >
>>> > > > >>>>> > I wonder at the end of the day if we should adopt something
>>> > more
>>> > > > >>>>> > modern like PIG or some other data flow type of language
>>> (PIG
>>> > > > >>>>> > is really neat).
>>> > > > >>>>> >
>>> > > > >>>>> > Cheers,
>>> > > > >>>>> > Chris
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > > > >>>>> > Chris Mattmann, Ph.D.
>>> > > > >>>>> > Chief Architect
>>> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
>>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
>>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
>>> > > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
>>> > > > >>>>> >
>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
>>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089
>>> USA
>>> > > > >>>>> >
>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> > -----Original Message-----
>>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
>>> > > > >>>>> > <ar...@airavata.apache.org>
>>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>> > > > >>>>> > To: "architecture@airavata.apache.org" <
>>> > > > >>>>> architecture@airavata.apache.org>,
>>> > > > >>>>> > dev <de...@airavata.apache.org>
>>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
>>> > > > >>>>>Airavata.
>>> > > > >>>>> >
>>> > > > >>>>> >> Hi All,
>>> > > > >>>>> >>
>>> > > > >>>>> >> As we all know Airavata has its own workflow language call
>>> > XWF.
>>> > > > >>>>>When
>>> > > > >>>>> XWF
>>> > > > >>>>> >> was introduced, main focus points are interoperability and
>>> > > > >>>>> convertibility.
>>> > > > >>>>> >> But with years of experience it is convinced that above
>>> > > > >>>>>requirements
>>> > > > >>>>> are
>>> > > > >>>>> >> not really useful when we come to real world use cases.
>>> And
>>> > XWF
>>> > > is
>>> > > > >>>>> XML
>>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
>>> image
>>> > > it
>>> > > > >>>>> self.
>>> > > > >>>>> >> But
>>> > > > >>>>> >> with the recent changes WSDL part is being removed from
>>> XWF.
>>> > > > >>>>> >>
>>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
>>> languages in
>>> > > > >>>>> industry
>>> > > > >>>>> >> and
>>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
>>> need
>>> > to
>>> > > > >>>>> come up
>>> > > > >>>>> >> with idea how we should improve Airavata workflow
>>> language,
>>> > > either
>>> > > > >>>>> we can
>>> > > > >>>>> >> improve existing XWF language, totally change to a new
>>> > language
>>> > > > >>>>> available
>>> > > > >>>>> >> in industry or write a new light weight language. Basic
>>> > > > >>>>>requirements
>>> > > > >>>>> that
>>> > > > >>>>> >> we expect from new improvement are, high usability,
>>> flexible,
>>> > > > >>>>>light
>>> > > > >>>>> weight
>>> > > > >>>>> >> and real time monitoring support. As you can see above
>>> > > > >>>>>requirements
>>> > > > >>>>> are
>>> > > > >>>>> >> not
>>> > > > >>>>> >> direct comes with workflow languages but we need workflow
>>> > > language
>>> > > > >>>>> which
>>> > > > >>>>> >> help to support above requirements.
>>> > > > >>>>> >>
>>> > > > >>>>> >> After reading few papers and googling, initially i have
>>> come
>>> > up
>>> > > > >>>>>with
>>> > > > >>>>> >> following three existing languages,
>>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>> > > > >>>>> >> 2. WS-BPEL
>>> > > > >>>>> >> 3. SIDL
>>> > > > >>>>> >> <
>>> > > http://computation.llnl.gov/casc/components/index.html#page=home
>>> > > > >
>>> > > > >>>>> >>
>>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific
>>> domain,
>>> > > > >>>>> Radical-SAGA
>>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
>>> above
>>> > > > >>>>>three
>>> > > > >>>>> >> languages we can come up with simple workflow language
>>> base on
>>> > > > >>>>> json(or
>>> > > > >>>>> >> yaml) which support all our requirements for some extends.
>>> > > > >>>>> >>
>>> > > > >>>>> >> It would be grate if I can get more input regarding the
>>> > $Subject
>>> > > > >>>>> form the
>>> > > > >>>>> >> airavata community. You all are more than welcome to
>>> provide
>>> > any
>>> > > > >>>>> type of
>>> > > > >>>>> >> suggestions.
>>> > > > >>>>> >>
>>> > > > >>>>> >> Thanks,
>>> > > > >>>>> >> Shameera.
>>> > > > >>>>> >>
>>> > > > >>>>> >>
>>> > > > >>>>> >>
>>> > > > >>>>> >> --
>>> > > > >>>>> >> Best Regards,
>>> > > > >>>>> >> Shameera Rathnayaka.
>>> > > > >>>>> >>
>>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > > >>>>
>>> > > > >>>>
>>> > > > >>>> --
>>> > > > >>>> The best way to predict the future is to invent it - Alan Kay
>>> > > > >>>>
>>> > > > >>>
>>> > > > >>>
>>> > > > >>>
>>> > > > >>> --
>>> > > > >>> Best Regards,
>>> > > > >>> Shameera Rathnayaka.
>>> > > > >>>
>>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
>>> > > > >>>
>>> > > > >>
>>> > > > >>
>>> > > > >>
>>> > > > >> --
>>> > > > >> Supun Kamburugamuva
>>> > > > >> Member, Apache Software Foundation; http://www.apache.org
>>> > > > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>> > > > >> Blog: http://supunk.blogspot.com
>>> > > > >>
>>> > > > >>
>>> > > > >
>>> > > > >
>>> > > > >--
>>> > > > >Supun Kamburugamuva
>>> > > > >Member, Apache Software Foundation; http://www.apache.org
>>> > > > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>> > > > >Blog: http://supunk.blogspot.com
>>> > > >
>>> > > >
>>> > > >
>>> > >
>>> > >
>>> > > --
>>> > > Best Regards,
>>> > > Shameera Rathnayaka.
>>> > >
>>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
>>> > > Blog : http://shameerarathnayaka.blogspot.com/
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Supun Kamburugamuva
>>> > Member, Apache Software Foundation; http://www.apache.org
>>> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>>> > Blog: http://supunk.blogspot.com
>>> >
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>>
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>
--
Best Regards,
Shameera Rathnayaka.
email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Amila Jayasekara <th...@gmail.com>.
Hello,
I am sorry, I am bit late on this thread. But when reading through this
thread I simply got lost, what this thread is discussing. I have few
questions.
1. @Shameera : Is XWF actually a language to define workflow ? To my
understanding it was an intermediate representation to convert workflow
defined in UI to java object model. Was XWF ever used by any airavata user
to define a workflow graph ?
>From initial description what I understood is we are looking for a improved
"intermediate representation", not a language which describes workflows.
2. So what is the exact purpose of this proposed language ?
- Is it to hide parallelism in workflows ?
- Is it to replace the XBAYA functionalities ? (i hope not)
3. What are we trying to achieve by this proposed language which we cannot
achieved through workflow UI tool ?
4. Who is going to use this language ?
5. Why would a user prefer (assuming intended audience of proposed language
is end users) a language over a Work Flow UI tool such as XBAYA ? (In other
words what are the things we can do with language which we cannot do with
UI)
Sorry, if above questions are in-appropriate, just wanted to understand
what exactly needed.
Thanks
-Thejaka Amila
On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:
> I think I'm not suggesting to create a Workflow interpreter using Python
> etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
> and move it to a more higher level component. The more I think about it,
> the model I'm suggesting is similar to what Hadoop, Storm etc has done for
> distributed system computations. This model is proven to be successful over
> the years.
>
> Keeping what Airavata does at its core can help you to build a more robust
> system. If we look at Airavata as middleware to execute applications on
> computing resources we can simplify what Airavata does and focus on
> improving the core functionality. All the successful systems have thrived
> on defining what it does at its core and keeping it simple and being
> excellent at what it does. In that regard keeping workflow aspect out of
> Airavata can help you to focus on the core problem. That is to execute an
> application on a remote computing resource in a fault tolerant and scalable
> manner.
>
> What I'm suggesting is to give the Orchestrator the capability to execute
> a Driver program that is specified by the user. (This program can be
> written in Python, Java or any other language). This driver program is
> similar to what you define in a Hadoop or Storm configuration. The driver
> program specifies the flow of the computation. It specifies what are the
> applications needs to be executed, how to manipulate input output. The
> driver program is the workflow for the user. Because the user specifies the
> program he can program it to handle workflow steering etc. Every time the
> user wants to execute this program he can tell Airavata to execute the
> Driver program.
>
> I'm also not 100% sure about all the details. But this can be a new way to
> look at how systems like Airavata should behave. Your thoughts and
> suggestions are more than welcome.
>
> Thanks,
> Supun..
>
>
>
>
>
>
> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi Supun,
>>
>> I think we were in two context, because I as suggesting a way to serialize
>> and deserialize the workflow description while you are suggesting to
>> implement
>> some kind of workflow interpreter using Python, where Python client can
>> send
>> thrift calls to Airavata server to run the application. I can see with
>> your
>> suggested
>> approach we can control the workflow execution process from client side
>> which
>> make it easy to implement workflow debugger.As you mentioned this is a
>> major change
>> to Airavata. So we should neatly think as this will change our existing
>> architecture.
>>
>> Still if someone need to use different language java, php, JS etc ... to
>> run the same
>> workflow which generated by Python, we need a language independent
>> workflow
>> description.
>> My initial question was what is the best language for this?. But as I have
>> explained in
>> one of my previous reply, It is not matter what language we used Either we
>> can use
>> XML or JSON to write this description, what matters is how easy to
>> generate
>> workflow with the provided API. Hence it would be great to have set of
>> neat
>> APIs in
>> different languages.
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <su...@gmail.com>
>> wrote:
>>
>> > Hi Shameera,
>> >
>> > Using python is a radical approach for workflow I think. But I believe
>> this
>> > is a very beautiful and new approach.Here is a possible scenario for
>> > implementation and running using a Python based library.
>> >
>> > The Python library facilitates the creation of Applications and
>> submitting
>> > them to Airavata for execution. The underlying functionality is exactly
>> > similar to what Java clients provides.The only difference is that,
>> Python
>> > library should have a more fluent API than Java for easy creation of
>> > workflows. We can generate the Python clients that talk to Airavata
>> server
>> > using Thrift.
>> >
>> > Here is an example off the top of my head to a Python script created by
>> > user for a Workflow. This is a very crude example and we need to come up
>> > with a much better API if we are going to go along this path. First we
>> need
>> > to write a Python script that can execute a workflow using Airavata.
>> >
>> > import airavata
>> >
>> > host = Host("localhost", ....)
>> > app1 = Application(host, ....)
>> > app2 = Application(host, ....)
>> >
>> > # we will connect these applications as a workflow using some topology
>> > builder or other constructs
>> >
>> > wb = WorkFlowBuilder()
>> > wb.setApp("name1", app1)
>> > # you can do a simple output transformation here etc
>> >
>> > # connects the input of app1 to app2 etc
>> > wb.setApp("name2", app2).connectInput("name1")
>> >
>> > wb.submit()
>> >
>> > Now we can load this Python script from XBaya. When XBaya loads this
>> script
>> > the Python script can output an XML configuration of the topology, XBaya
>> > can render. There are other ways like directly executing the Python
>> script
>> > from command line and connecting XBaya indirectly as well. Now you can
>> run
>> > the workflow from XBaya. Running the Workflow means just executing the
>> > Python script.
>> >
>> > XBaya gets the notifications through messaging and update the UI
>> > accordingly.
>> >
>> > The users need to write the Python script by hand. XBaya cannot create
>> the
>> > script. Because workflow language is an actual python program the
>> benefits
>> > are immense. For example user can do workflow steering in the workflow
>> > itself by subscribing to messages from Airavata.
>> >
>> > Thanks,
>> > Supun..
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>> > shameerainfo@gmail.com> wrote:
>> >
>> > > Hi Supun,
>> > >
>> > > I meant to say JS is a well-known client side scripting language i
>> have
>> > > messed scripting part. Even we use Python, ultimately we should
>> convert
>> > > this to java model in server side, somehow we need to serialized
>> python
>> > > representation to the language which java can parse and generate that
>> > > model. In this case we need to parse python script in java isn't it?
>> I am
>> > > not exactly clear how you suggesting to use python here. More details
>> on
>> > > how end system works if we use Python would be great help to clearly
>> > > understand your points.
>> > >
>> > > Thanks,
>> > > Shameera.
>> > >
>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>> > chris.mattmann@gmail.com>
>> > > wrote:
>> > >
>> > > > Have you guys considered using JCC [1] as a means
>> > > > to expose the workflow API currently in Java as a
>> > > > Python API?
>> > > >
>> > > > We are exploring its use in OODT, and we have already
>> > > > created a Tika [2] JCC-based python API.
>> > > >
>> > > > Cheers,
>> > > > Chris
>> > > >
>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>> > > > [2] http://github.com/chrismattmann/tika-python/
>> > > >
>> > > > ------------------------
>> > > > Chris Mattmann
>> > > > chris.mattmann@gmail.com
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > -----Original Message-----
>> > > > From: Supun Kamburugamuva <su...@gmail.com>
>> > > > Reply-To: <de...@oodt.apache.org>
>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>> > > > To: dev <de...@airavata.apache.org>
>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
>> > > > <de...@oodt.apache.org>
>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>> > Airavata.
>> > > >
>> > > > >Once we had an offline discussion about the Airavata Workflow
>> language
>> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
>> came
>> > > out
>> > > > >was why we need to invent a different language when a simple
>> library
>> > > like
>> > > > >Python will full fill of Airavata requirements.
>> > > > >
>> > > > >There are many benefits in using a Python library as the API for
>> > > > >controlling Airavata workflows.
>> > > > >
>> > > > >1. It is a library, gives the ultimate control over the execution
>> and
>> > it
>> > > > >can be simpler than any domain specific language that we can come
>> with
>> > > > >like
>> > > > >XML, JSON etc
>> > > > >2. Most people use python and can learn it easily than any Domain
>> > > specific
>> > > > >language
>> > > > >3. You can easily create a Python library for Airavata because all
>> the
>> > > > >APIs
>> > > > >are thrift based.
>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>> > > > >
>> > > > >Any thoughts?
>> > > > >
>> > > > >Thanks,
>> > > > >Supun..
>> > > > >
>> > > > >
>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
>> > supun06@gmail.com
>> > > >
>> > > > >wrote:
>> > > > >
>> > > > >> Hi Shameera,
>> > > > >>
>> > > > >> Why you prefer JavaScript over a language like Python?
>> > > > >>
>> > > > >> Thanks,
>> > > > >> Supun..
>> > > > >>
>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> > > > >> shameerainfo@gmail.com> wrote:
>> > > > >>
>> > > > >>> Hi,
>> > > > >>>
>> > > > >>> First of all thanks everyone for giving valuable inputs. After
>> > doing
>> > > > >>>some
>> > > > >>> background search and talking to different people in the
>> University
>> > > > >>>who has
>> > > > >>> used different workflow languages, I myself convinced that
>> > > introducing
>> > > > >>>an
>> > > > >>> another workflow language is not what actually they need. By
>> > changing
>> > > > >>> exiting workflow language to another will not solve problems.
>> What
>> > > they
>> > > > >>> asking is a easy way to construct the workflows. Indirectly what
>> > they
>> > > > >>> asking for a sort of API which they can use to generate the
>> > workflows
>> > > > >>>and
>> > > > >>> run it. Correct me if i am wrong here.
>> > > > >>>
>> > > > >>> As most of above replies depict, if we can get a simple API,
>> as an
>> > > > >>> example, for a web based application, JavaScript API would be a
>> > good
>> > > > >>> solution, and probably JSON would be a good candidate for
>> language,
>> > > > >>>instead
>> > > > >>> of XML.
>> > > > >>>
>> > > > >>> Airavata community already have started to implement web base
>> GUI.
>> > > > >>>Hence
>> > > > >>> introducing a JSON base JavaScript API would be great help.
>> WDYT?
>> > > > >>>
>> > > > >>> Thanks,
>> > > > >>> Shameera.
>> > > > >>>
>> > > > >>>
>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> > > > >>> alekny7@gmail.com> wrote:
>> > > > >>>
>> > > > >>>> Hi,
>> > > > >>>>
>> > > > >>>> it is not dataflow instead focused on orchestrating REST
>> services
>> > > but
>> > > > >>>> you may find it useful datapoint - we created worfklow service
>> > that
>> > > > >>>>uses
>> > > > >>>> natively JavaScript and JSON to describe what happens during
>> > > workflow
>> > > > >>>> execution:
>> > > > >>>>
>> > > >
>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>> > > > >>>>
>> > > > >>>> HTH,
>> > > > >>>>
>> > > > >>>> Alek
>> > > > >>>>
>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
>> smarru@apache.org>
>> > > > >>>>wrote:
>> > > > >>>>
>> > > > >>>>> Hi Chris,
>> > > > >>>>>
>> > > > >>>>> Great to hear OODT community will be interested in adopting a
>> > JSON
>> > > > >>>>> based workflow language and potentially a web based composer
>> as
>> > > well.
>> > > > >>>>> Airavata previously had BPEL support initially through a home
>> > grown
>> > > > >>>>> implementation [1] by Alek Slominski and later through Apache
>> ODE
>> > > > >>>>>[2]. Also
>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
>> read.
>> > > > >>>>>
>> > > > >>>>> I am of the same opinion that we should adopt something more
>> > modern
>> > > > >>>>>as
>> > > > >>>>> the challenges from scientific workflows seems to be
>> converging
>> > > with
>> > > > >>>>>the
>> > > > >>>>> data flow patterns in business workflows.
>> > > > >>>>>
>> > > > >>>>> It will be great if we can all compile a list of potential
>> > > candidates
>> > > > >>>>> and hack them through.
>> > > > >>>>>
>> > > > >>>>> Suresh
>> > > > >>>>> [1] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>> > > > >>>>> [2] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>> > > > >>>>>time_for_eScience_workflows
>> > > > >>>>> [3] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>> > > > >>>>>6.pdf
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>> > > > >>>>>
>> > > > >>>>> > Hi Guys,
>> > > > >>>>> >
>> > > > >>>>> > I've been interested in this too - we don't per have a
>> specific
>> > > > >>>>> > OODT workflow language, but we specific workflows using XML,
>> > and
>> > > > >>>>> > other configuration (we are also thinking of moving to JSON
>> for
>> > > > >>>>> > this).
>> > > > >>>>> >
>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
>> > complex
>> > > > >>>>> > to me.
>> > > > >>>>> >
>> > > > >>>>> > I wonder at the end of the day if we should adopt something
>> > more
>> > > > >>>>> > modern like PIG or some other data flow type of language
>> (PIG
>> > > > >>>>> > is really neat).
>> > > > >>>>> >
>> > > > >>>>> > Cheers,
>> > > > >>>>> > Chris
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> > Chris Mattmann, Ph.D.
>> > > > >>>>> > Chief Architect
>> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
>> > > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> > -----Original Message-----
>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
>> > > > >>>>> > <ar...@airavata.apache.org>
>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
>> > > > >>>>> > To: "architecture@airavata.apache.org" <
>> > > > >>>>> architecture@airavata.apache.org>,
>> > > > >>>>> > dev <de...@airavata.apache.org>
>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
>> > > > >>>>>Airavata.
>> > > > >>>>> >
>> > > > >>>>> >> Hi All,
>> > > > >>>>> >>
>> > > > >>>>> >> As we all know Airavata has its own workflow language call
>> > XWF.
>> > > > >>>>>When
>> > > > >>>>> XWF
>> > > > >>>>> >> was introduced, main focus points are interoperability and
>> > > > >>>>> convertibility.
>> > > > >>>>> >> But with years of experience it is convinced that above
>> > > > >>>>>requirements
>> > > > >>>>> are
>> > > > >>>>> >> not really useful when we come to real world use cases. And
>> > XWF
>> > > is
>> > > > >>>>> XML
>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
>> image
>> > > it
>> > > > >>>>> self.
>> > > > >>>>> >> But
>> > > > >>>>> >> with the recent changes WSDL part is being removed from
>> XWF.
>> > > > >>>>> >>
>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
>> languages in
>> > > > >>>>> industry
>> > > > >>>>> >> and
>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
>> need
>> > to
>> > > > >>>>> come up
>> > > > >>>>> >> with idea how we should improve Airavata workflow language,
>> > > either
>> > > > >>>>> we can
>> > > > >>>>> >> improve existing XWF language, totally change to a new
>> > language
>> > > > >>>>> available
>> > > > >>>>> >> in industry or write a new light weight language. Basic
>> > > > >>>>>requirements
>> > > > >>>>> that
>> > > > >>>>> >> we expect from new improvement are, high usability,
>> flexible,
>> > > > >>>>>light
>> > > > >>>>> weight
>> > > > >>>>> >> and real time monitoring support. As you can see above
>> > > > >>>>>requirements
>> > > > >>>>> are
>> > > > >>>>> >> not
>> > > > >>>>> >> direct comes with workflow languages but we need workflow
>> > > language
>> > > > >>>>> which
>> > > > >>>>> >> help to support above requirements.
>> > > > >>>>> >>
>> > > > >>>>> >> After reading few papers and googling, initially i have
>> come
>> > up
>> > > > >>>>>with
>> > > > >>>>> >> following three existing languages,
>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>> > > > >>>>> >> 2. WS-BPEL
>> > > > >>>>> >> 3. SIDL
>> > > > >>>>> >> <
>> > > http://computation.llnl.gov/casc/components/index.html#page=home
>> > > > >
>> > > > >>>>> >>
>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
>> > > > >>>>> Radical-SAGA
>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
>> above
>> > > > >>>>>three
>> > > > >>>>> >> languages we can come up with simple workflow language
>> base on
>> > > > >>>>> json(or
>> > > > >>>>> >> yaml) which support all our requirements for some extends.
>> > > > >>>>> >>
>> > > > >>>>> >> It would be grate if I can get more input regarding the
>> > $Subject
>> > > > >>>>> form the
>> > > > >>>>> >> airavata community. You all are more than welcome to
>> provide
>> > any
>> > > > >>>>> type of
>> > > > >>>>> >> suggestions.
>> > > > >>>>> >>
>> > > > >>>>> >> Thanks,
>> > > > >>>>> >> Shameera.
>> > > > >>>>> >>
>> > > > >>>>> >>
>> > > > >>>>> >>
>> > > > >>>>> >> --
>> > > > >>>>> >> Best Regards,
>> > > > >>>>> >> Shameera Rathnayaka.
>> > > > >>>>> >>
>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> --
>> > > > >>>> The best way to predict the future is to invent it - Alan Kay
>> > > > >>>>
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> --
>> > > > >>> Best Regards,
>> > > > >>> Shameera Rathnayaka.
>> > > > >>>
>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
>> > > > >>>
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >> --
>> > > > >> Supun Kamburugamuva
>> > > > >> Member, Apache Software Foundation; http://www.apache.org
>> > > > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> > > > >> Blog: http://supunk.blogspot.com
>> > > > >>
>> > > > >>
>> > > > >
>> > > > >
>> > > > >--
>> > > > >Supun Kamburugamuva
>> > > > >Member, Apache Software Foundation; http://www.apache.org
>> > > > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> > > > >Blog: http://supunk.blogspot.com
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Best Regards,
>> > > Shameera Rathnayaka.
>> > >
>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > Blog : http://shameerarathnayaka.blogspot.com/
>> > >
>> >
>> >
>> >
>> > --
>> > Supun Kamburugamuva
>> > Member, Apache Software Foundation; http://www.apache.org
>> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> > Blog: http://supunk.blogspot.com
>> >
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Amila Jayasekara <th...@gmail.com>.
Hello,
I am sorry, I am bit late on this thread. But when reading through this
thread I simply got lost, what this thread is discussing. I have few
questions.
1. @Shameera : Is XWF actually a language to define workflow ? To my
understanding it was an intermediate representation to convert workflow
defined in UI to java object model. Was XWF ever used by any airavata user
to define a workflow graph ?
>From initial description what I understood is we are looking for a improved
"intermediate representation", not a language which describes workflows.
2. So what is the exact purpose of this proposed language ?
- Is it to hide parallelism in workflows ?
- Is it to replace the XBAYA functionalities ? (i hope not)
3. What are we trying to achieve by this proposed language which we cannot
achieved through workflow UI tool ?
4. Who is going to use this language ?
5. Why would a user prefer (assuming intended audience of proposed language
is end users) a language over a Work Flow UI tool such as XBAYA ? (In other
words what are the things we can do with language which we cannot do with
UI)
Sorry, if above questions are in-appropriate, just wanted to understand
what exactly needed.
Thanks
-Thejaka Amila
On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:
> I think I'm not suggesting to create a Workflow interpreter using Python
> etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
> and move it to a more higher level component. The more I think about it,
> the model I'm suggesting is similar to what Hadoop, Storm etc has done for
> distributed system computations. This model is proven to be successful over
> the years.
>
> Keeping what Airavata does at its core can help you to build a more robust
> system. If we look at Airavata as middleware to execute applications on
> computing resources we can simplify what Airavata does and focus on
> improving the core functionality. All the successful systems have thrived
> on defining what it does at its core and keeping it simple and being
> excellent at what it does. In that regard keeping workflow aspect out of
> Airavata can help you to focus on the core problem. That is to execute an
> application on a remote computing resource in a fault tolerant and scalable
> manner.
>
> What I'm suggesting is to give the Orchestrator the capability to execute
> a Driver program that is specified by the user. (This program can be
> written in Python, Java or any other language). This driver program is
> similar to what you define in a Hadoop or Storm configuration. The driver
> program specifies the flow of the computation. It specifies what are the
> applications needs to be executed, how to manipulate input output. The
> driver program is the workflow for the user. Because the user specifies the
> program he can program it to handle workflow steering etc. Every time the
> user wants to execute this program he can tell Airavata to execute the
> Driver program.
>
> I'm also not 100% sure about all the details. But this can be a new way to
> look at how systems like Airavata should behave. Your thoughts and
> suggestions are more than welcome.
>
> Thanks,
> Supun..
>
>
>
>
>
>
> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi Supun,
>>
>> I think we were in two context, because I as suggesting a way to serialize
>> and deserialize the workflow description while you are suggesting to
>> implement
>> some kind of workflow interpreter using Python, where Python client can
>> send
>> thrift calls to Airavata server to run the application. I can see with
>> your
>> suggested
>> approach we can control the workflow execution process from client side
>> which
>> make it easy to implement workflow debugger.As you mentioned this is a
>> major change
>> to Airavata. So we should neatly think as this will change our existing
>> architecture.
>>
>> Still if someone need to use different language java, php, JS etc ... to
>> run the same
>> workflow which generated by Python, we need a language independent
>> workflow
>> description.
>> My initial question was what is the best language for this?. But as I have
>> explained in
>> one of my previous reply, It is not matter what language we used Either we
>> can use
>> XML or JSON to write this description, what matters is how easy to
>> generate
>> workflow with the provided API. Hence it would be great to have set of
>> neat
>> APIs in
>> different languages.
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <su...@gmail.com>
>> wrote:
>>
>> > Hi Shameera,
>> >
>> > Using python is a radical approach for workflow I think. But I believe
>> this
>> > is a very beautiful and new approach.Here is a possible scenario for
>> > implementation and running using a Python based library.
>> >
>> > The Python library facilitates the creation of Applications and
>> submitting
>> > them to Airavata for execution. The underlying functionality is exactly
>> > similar to what Java clients provides.The only difference is that,
>> Python
>> > library should have a more fluent API than Java for easy creation of
>> > workflows. We can generate the Python clients that talk to Airavata
>> server
>> > using Thrift.
>> >
>> > Here is an example off the top of my head to a Python script created by
>> > user for a Workflow. This is a very crude example and we need to come up
>> > with a much better API if we are going to go along this path. First we
>> need
>> > to write a Python script that can execute a workflow using Airavata.
>> >
>> > import airavata
>> >
>> > host = Host("localhost", ....)
>> > app1 = Application(host, ....)
>> > app2 = Application(host, ....)
>> >
>> > # we will connect these applications as a workflow using some topology
>> > builder or other constructs
>> >
>> > wb = WorkFlowBuilder()
>> > wb.setApp("name1", app1)
>> > # you can do a simple output transformation here etc
>> >
>> > # connects the input of app1 to app2 etc
>> > wb.setApp("name2", app2).connectInput("name1")
>> >
>> > wb.submit()
>> >
>> > Now we can load this Python script from XBaya. When XBaya loads this
>> script
>> > the Python script can output an XML configuration of the topology, XBaya
>> > can render. There are other ways like directly executing the Python
>> script
>> > from command line and connecting XBaya indirectly as well. Now you can
>> run
>> > the workflow from XBaya. Running the Workflow means just executing the
>> > Python script.
>> >
>> > XBaya gets the notifications through messaging and update the UI
>> > accordingly.
>> >
>> > The users need to write the Python script by hand. XBaya cannot create
>> the
>> > script. Because workflow language is an actual python program the
>> benefits
>> > are immense. For example user can do workflow steering in the workflow
>> > itself by subscribing to messages from Airavata.
>> >
>> > Thanks,
>> > Supun..
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>> > shameerainfo@gmail.com> wrote:
>> >
>> > > Hi Supun,
>> > >
>> > > I meant to say JS is a well-known client side scripting language i
>> have
>> > > messed scripting part. Even we use Python, ultimately we should
>> convert
>> > > this to java model in server side, somehow we need to serialized
>> python
>> > > representation to the language which java can parse and generate that
>> > > model. In this case we need to parse python script in java isn't it?
>> I am
>> > > not exactly clear how you suggesting to use python here. More details
>> on
>> > > how end system works if we use Python would be great help to clearly
>> > > understand your points.
>> > >
>> > > Thanks,
>> > > Shameera.
>> > >
>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>> > chris.mattmann@gmail.com>
>> > > wrote:
>> > >
>> > > > Have you guys considered using JCC [1] as a means
>> > > > to expose the workflow API currently in Java as a
>> > > > Python API?
>> > > >
>> > > > We are exploring its use in OODT, and we have already
>> > > > created a Tika [2] JCC-based python API.
>> > > >
>> > > > Cheers,
>> > > > Chris
>> > > >
>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>> > > > [2] http://github.com/chrismattmann/tika-python/
>> > > >
>> > > > ------------------------
>> > > > Chris Mattmann
>> > > > chris.mattmann@gmail.com
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > -----Original Message-----
>> > > > From: Supun Kamburugamuva <su...@gmail.com>
>> > > > Reply-To: <de...@oodt.apache.org>
>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>> > > > To: dev <de...@airavata.apache.org>
>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
>> > > > <de...@oodt.apache.org>
>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>> > Airavata.
>> > > >
>> > > > >Once we had an offline discussion about the Airavata Workflow
>> language
>> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
>> came
>> > > out
>> > > > >was why we need to invent a different language when a simple
>> library
>> > > like
>> > > > >Python will full fill of Airavata requirements.
>> > > > >
>> > > > >There are many benefits in using a Python library as the API for
>> > > > >controlling Airavata workflows.
>> > > > >
>> > > > >1. It is a library, gives the ultimate control over the execution
>> and
>> > it
>> > > > >can be simpler than any domain specific language that we can come
>> with
>> > > > >like
>> > > > >XML, JSON etc
>> > > > >2. Most people use python and can learn it easily than any Domain
>> > > specific
>> > > > >language
>> > > > >3. You can easily create a Python library for Airavata because all
>> the
>> > > > >APIs
>> > > > >are thrift based.
>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>> > > > >
>> > > > >Any thoughts?
>> > > > >
>> > > > >Thanks,
>> > > > >Supun..
>> > > > >
>> > > > >
>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
>> > supun06@gmail.com
>> > > >
>> > > > >wrote:
>> > > > >
>> > > > >> Hi Shameera,
>> > > > >>
>> > > > >> Why you prefer JavaScript over a language like Python?
>> > > > >>
>> > > > >> Thanks,
>> > > > >> Supun..
>> > > > >>
>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> > > > >> shameerainfo@gmail.com> wrote:
>> > > > >>
>> > > > >>> Hi,
>> > > > >>>
>> > > > >>> First of all thanks everyone for giving valuable inputs. After
>> > doing
>> > > > >>>some
>> > > > >>> background search and talking to different people in the
>> University
>> > > > >>>who has
>> > > > >>> used different workflow languages, I myself convinced that
>> > > introducing
>> > > > >>>an
>> > > > >>> another workflow language is not what actually they need. By
>> > changing
>> > > > >>> exiting workflow language to another will not solve problems.
>> What
>> > > they
>> > > > >>> asking is a easy way to construct the workflows. Indirectly what
>> > they
>> > > > >>> asking for a sort of API which they can use to generate the
>> > workflows
>> > > > >>>and
>> > > > >>> run it. Correct me if i am wrong here.
>> > > > >>>
>> > > > >>> As most of above replies depict, if we can get a simple API,
>> as an
>> > > > >>> example, for a web based application, JavaScript API would be a
>> > good
>> > > > >>> solution, and probably JSON would be a good candidate for
>> language,
>> > > > >>>instead
>> > > > >>> of XML.
>> > > > >>>
>> > > > >>> Airavata community already have started to implement web base
>> GUI.
>> > > > >>>Hence
>> > > > >>> introducing a JSON base JavaScript API would be great help.
>> WDYT?
>> > > > >>>
>> > > > >>> Thanks,
>> > > > >>> Shameera.
>> > > > >>>
>> > > > >>>
>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> > > > >>> alekny7@gmail.com> wrote:
>> > > > >>>
>> > > > >>>> Hi,
>> > > > >>>>
>> > > > >>>> it is not dataflow instead focused on orchestrating REST
>> services
>> > > but
>> > > > >>>> you may find it useful datapoint - we created worfklow service
>> > that
>> > > > >>>>uses
>> > > > >>>> natively JavaScript and JSON to describe what happens during
>> > > workflow
>> > > > >>>> execution:
>> > > > >>>>
>> > > >
>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>> > > > >>>>
>> > > > >>>> HTH,
>> > > > >>>>
>> > > > >>>> Alek
>> > > > >>>>
>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
>> smarru@apache.org>
>> > > > >>>>wrote:
>> > > > >>>>
>> > > > >>>>> Hi Chris,
>> > > > >>>>>
>> > > > >>>>> Great to hear OODT community will be interested in adopting a
>> > JSON
>> > > > >>>>> based workflow language and potentially a web based composer
>> as
>> > > well.
>> > > > >>>>> Airavata previously had BPEL support initially through a home
>> > grown
>> > > > >>>>> implementation [1] by Alek Slominski and later through Apache
>> ODE
>> > > > >>>>>[2]. Also
>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
>> read.
>> > > > >>>>>
>> > > > >>>>> I am of the same opinion that we should adopt something more
>> > modern
>> > > > >>>>>as
>> > > > >>>>> the challenges from scientific workflows seems to be
>> converging
>> > > with
>> > > > >>>>>the
>> > > > >>>>> data flow patterns in business workflows.
>> > > > >>>>>
>> > > > >>>>> It will be great if we can all compile a list of potential
>> > > candidates
>> > > > >>>>> and hack them through.
>> > > > >>>>>
>> > > > >>>>> Suresh
>> > > > >>>>> [1] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>> > > > >>>>> [2] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>> > > > >>>>>time_for_eScience_workflows
>> > > > >>>>> [3] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>> > > > >>>>>6.pdf
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>> > > > >>>>>
>> > > > >>>>> > Hi Guys,
>> > > > >>>>> >
>> > > > >>>>> > I've been interested in this too - we don't per have a
>> specific
>> > > > >>>>> > OODT workflow language, but we specific workflows using XML,
>> > and
>> > > > >>>>> > other configuration (we are also thinking of moving to JSON
>> for
>> > > > >>>>> > this).
>> > > > >>>>> >
>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
>> > complex
>> > > > >>>>> > to me.
>> > > > >>>>> >
>> > > > >>>>> > I wonder at the end of the day if we should adopt something
>> > more
>> > > > >>>>> > modern like PIG or some other data flow type of language
>> (PIG
>> > > > >>>>> > is really neat).
>> > > > >>>>> >
>> > > > >>>>> > Cheers,
>> > > > >>>>> > Chris
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> > Chris Mattmann, Ph.D.
>> > > > >>>>> > Chief Architect
>> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
>> > > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> > -----Original Message-----
>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
>> > > > >>>>> > <ar...@airavata.apache.org>
>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
>> > > > >>>>> > To: "architecture@airavata.apache.org" <
>> > > > >>>>> architecture@airavata.apache.org>,
>> > > > >>>>> > dev <de...@airavata.apache.org>
>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
>> > > > >>>>>Airavata.
>> > > > >>>>> >
>> > > > >>>>> >> Hi All,
>> > > > >>>>> >>
>> > > > >>>>> >> As we all know Airavata has its own workflow language call
>> > XWF.
>> > > > >>>>>When
>> > > > >>>>> XWF
>> > > > >>>>> >> was introduced, main focus points are interoperability and
>> > > > >>>>> convertibility.
>> > > > >>>>> >> But with years of experience it is convinced that above
>> > > > >>>>>requirements
>> > > > >>>>> are
>> > > > >>>>> >> not really useful when we come to real world use cases. And
>> > XWF
>> > > is
>> > > > >>>>> XML
>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
>> image
>> > > it
>> > > > >>>>> self.
>> > > > >>>>> >> But
>> > > > >>>>> >> with the recent changes WSDL part is being removed from
>> XWF.
>> > > > >>>>> >>
>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
>> languages in
>> > > > >>>>> industry
>> > > > >>>>> >> and
>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
>> need
>> > to
>> > > > >>>>> come up
>> > > > >>>>> >> with idea how we should improve Airavata workflow language,
>> > > either
>> > > > >>>>> we can
>> > > > >>>>> >> improve existing XWF language, totally change to a new
>> > language
>> > > > >>>>> available
>> > > > >>>>> >> in industry or write a new light weight language. Basic
>> > > > >>>>>requirements
>> > > > >>>>> that
>> > > > >>>>> >> we expect from new improvement are, high usability,
>> flexible,
>> > > > >>>>>light
>> > > > >>>>> weight
>> > > > >>>>> >> and real time monitoring support. As you can see above
>> > > > >>>>>requirements
>> > > > >>>>> are
>> > > > >>>>> >> not
>> > > > >>>>> >> direct comes with workflow languages but we need workflow
>> > > language
>> > > > >>>>> which
>> > > > >>>>> >> help to support above requirements.
>> > > > >>>>> >>
>> > > > >>>>> >> After reading few papers and googling, initially i have
>> come
>> > up
>> > > > >>>>>with
>> > > > >>>>> >> following three existing languages,
>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>> > > > >>>>> >> 2. WS-BPEL
>> > > > >>>>> >> 3. SIDL
>> > > > >>>>> >> <
>> > > http://computation.llnl.gov/casc/components/index.html#page=home
>> > > > >
>> > > > >>>>> >>
>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
>> > > > >>>>> Radical-SAGA
>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
>> above
>> > > > >>>>>three
>> > > > >>>>> >> languages we can come up with simple workflow language
>> base on
>> > > > >>>>> json(or
>> > > > >>>>> >> yaml) which support all our requirements for some extends.
>> > > > >>>>> >>
>> > > > >>>>> >> It would be grate if I can get more input regarding the
>> > $Subject
>> > > > >>>>> form the
>> > > > >>>>> >> airavata community. You all are more than welcome to
>> provide
>> > any
>> > > > >>>>> type of
>> > > > >>>>> >> suggestions.
>> > > > >>>>> >>
>> > > > >>>>> >> Thanks,
>> > > > >>>>> >> Shameera.
>> > > > >>>>> >>
>> > > > >>>>> >>
>> > > > >>>>> >>
>> > > > >>>>> >> --
>> > > > >>>>> >> Best Regards,
>> > > > >>>>> >> Shameera Rathnayaka.
>> > > > >>>>> >>
>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> --
>> > > > >>>> The best way to predict the future is to invent it - Alan Kay
>> > > > >>>>
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> --
>> > > > >>> Best Regards,
>> > > > >>> Shameera Rathnayaka.
>> > > > >>>
>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
>> > > > >>>
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >> --
>> > > > >> Supun Kamburugamuva
>> > > > >> Member, Apache Software Foundation; http://www.apache.org
>> > > > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> > > > >> Blog: http://supunk.blogspot.com
>> > > > >>
>> > > > >>
>> > > > >
>> > > > >
>> > > > >--
>> > > > >Supun Kamburugamuva
>> > > > >Member, Apache Software Foundation; http://www.apache.org
>> > > > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> > > > >Blog: http://supunk.blogspot.com
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Best Regards,
>> > > Shameera Rathnayaka.
>> > >
>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > Blog : http://shameerarathnayaka.blogspot.com/
>> > >
>> >
>> >
>> >
>> > --
>> > Supun Kamburugamuva
>> > Member, Apache Software Foundation; http://www.apache.org
>> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> > Blog: http://supunk.blogspot.com
>> >
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Amila Jayasekara <th...@gmail.com>.
Hello,
I am sorry, I am bit late on this thread. But when reading through this
thread I simply got lost, what this thread is discussing. I have few
questions.
1. @Shameera : Is XWF actually a language to define workflow ? To my
understanding it was an intermediate representation to convert workflow
defined in UI to java object model. Was XWF ever used by any airavata user
to define a workflow graph ?
>From initial description what I understood is we are looking for a improved
"intermediate representation", not a language which describes workflows.
2. So what is the exact purpose of this proposed language ?
- Is it to hide parallelism in workflows ?
- Is it to replace the XBAYA functionalities ? (i hope not)
3. What are we trying to achieve by this proposed language which we cannot
achieved through workflow UI tool ?
4. Who is going to use this language ?
5. Why would a user prefer (assuming intended audience of proposed language
is end users) a language over a Work Flow UI tool such as XBAYA ? (In other
words what are the things we can do with language which we cannot do with
UI)
Sorry, if above questions are in-appropriate, just wanted to understand
what exactly needed.
Thanks
-Thejaka Amila
On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:
> I think I'm not suggesting to create a Workflow interpreter using Python
> etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
> and move it to a more higher level component. The more I think about it,
> the model I'm suggesting is similar to what Hadoop, Storm etc has done for
> distributed system computations. This model is proven to be successful over
> the years.
>
> Keeping what Airavata does at its core can help you to build a more robust
> system. If we look at Airavata as middleware to execute applications on
> computing resources we can simplify what Airavata does and focus on
> improving the core functionality. All the successful systems have thrived
> on defining what it does at its core and keeping it simple and being
> excellent at what it does. In that regard keeping workflow aspect out of
> Airavata can help you to focus on the core problem. That is to execute an
> application on a remote computing resource in a fault tolerant and scalable
> manner.
>
> What I'm suggesting is to give the Orchestrator the capability to execute
> a Driver program that is specified by the user. (This program can be
> written in Python, Java or any other language). This driver program is
> similar to what you define in a Hadoop or Storm configuration. The driver
> program specifies the flow of the computation. It specifies what are the
> applications needs to be executed, how to manipulate input output. The
> driver program is the workflow for the user. Because the user specifies the
> program he can program it to handle workflow steering etc. Every time the
> user wants to execute this program he can tell Airavata to execute the
> Driver program.
>
> I'm also not 100% sure about all the details. But this can be a new way to
> look at how systems like Airavata should behave. Your thoughts and
> suggestions are more than welcome.
>
> Thanks,
> Supun..
>
>
>
>
>
>
> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi Supun,
>>
>> I think we were in two context, because I as suggesting a way to serialize
>> and deserialize the workflow description while you are suggesting to
>> implement
>> some kind of workflow interpreter using Python, where Python client can
>> send
>> thrift calls to Airavata server to run the application. I can see with
>> your
>> suggested
>> approach we can control the workflow execution process from client side
>> which
>> make it easy to implement workflow debugger.As you mentioned this is a
>> major change
>> to Airavata. So we should neatly think as this will change our existing
>> architecture.
>>
>> Still if someone need to use different language java, php, JS etc ... to
>> run the same
>> workflow which generated by Python, we need a language independent
>> workflow
>> description.
>> My initial question was what is the best language for this?. But as I have
>> explained in
>> one of my previous reply, It is not matter what language we used Either we
>> can use
>> XML or JSON to write this description, what matters is how easy to
>> generate
>> workflow with the provided API. Hence it would be great to have set of
>> neat
>> APIs in
>> different languages.
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <su...@gmail.com>
>> wrote:
>>
>> > Hi Shameera,
>> >
>> > Using python is a radical approach for workflow I think. But I believe
>> this
>> > is a very beautiful and new approach.Here is a possible scenario for
>> > implementation and running using a Python based library.
>> >
>> > The Python library facilitates the creation of Applications and
>> submitting
>> > them to Airavata for execution. The underlying functionality is exactly
>> > similar to what Java clients provides.The only difference is that,
>> Python
>> > library should have a more fluent API than Java for easy creation of
>> > workflows. We can generate the Python clients that talk to Airavata
>> server
>> > using Thrift.
>> >
>> > Here is an example off the top of my head to a Python script created by
>> > user for a Workflow. This is a very crude example and we need to come up
>> > with a much better API if we are going to go along this path. First we
>> need
>> > to write a Python script that can execute a workflow using Airavata.
>> >
>> > import airavata
>> >
>> > host = Host("localhost", ....)
>> > app1 = Application(host, ....)
>> > app2 = Application(host, ....)
>> >
>> > # we will connect these applications as a workflow using some topology
>> > builder or other constructs
>> >
>> > wb = WorkFlowBuilder()
>> > wb.setApp("name1", app1)
>> > # you can do a simple output transformation here etc
>> >
>> > # connects the input of app1 to app2 etc
>> > wb.setApp("name2", app2).connectInput("name1")
>> >
>> > wb.submit()
>> >
>> > Now we can load this Python script from XBaya. When XBaya loads this
>> script
>> > the Python script can output an XML configuration of the topology, XBaya
>> > can render. There are other ways like directly executing the Python
>> script
>> > from command line and connecting XBaya indirectly as well. Now you can
>> run
>> > the workflow from XBaya. Running the Workflow means just executing the
>> > Python script.
>> >
>> > XBaya gets the notifications through messaging and update the UI
>> > accordingly.
>> >
>> > The users need to write the Python script by hand. XBaya cannot create
>> the
>> > script. Because workflow language is an actual python program the
>> benefits
>> > are immense. For example user can do workflow steering in the workflow
>> > itself by subscribing to messages from Airavata.
>> >
>> > Thanks,
>> > Supun..
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>> > shameerainfo@gmail.com> wrote:
>> >
>> > > Hi Supun,
>> > >
>> > > I meant to say JS is a well-known client side scripting language i
>> have
>> > > messed scripting part. Even we use Python, ultimately we should
>> convert
>> > > this to java model in server side, somehow we need to serialized
>> python
>> > > representation to the language which java can parse and generate that
>> > > model. In this case we need to parse python script in java isn't it?
>> I am
>> > > not exactly clear how you suggesting to use python here. More details
>> on
>> > > how end system works if we use Python would be great help to clearly
>> > > understand your points.
>> > >
>> > > Thanks,
>> > > Shameera.
>> > >
>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>> > chris.mattmann@gmail.com>
>> > > wrote:
>> > >
>> > > > Have you guys considered using JCC [1] as a means
>> > > > to expose the workflow API currently in Java as a
>> > > > Python API?
>> > > >
>> > > > We are exploring its use in OODT, and we have already
>> > > > created a Tika [2] JCC-based python API.
>> > > >
>> > > > Cheers,
>> > > > Chris
>> > > >
>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>> > > > [2] http://github.com/chrismattmann/tika-python/
>> > > >
>> > > > ------------------------
>> > > > Chris Mattmann
>> > > > chris.mattmann@gmail.com
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > -----Original Message-----
>> > > > From: Supun Kamburugamuva <su...@gmail.com>
>> > > > Reply-To: <de...@oodt.apache.org>
>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>> > > > To: dev <de...@airavata.apache.org>
>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
>> > > > <de...@oodt.apache.org>
>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>> > Airavata.
>> > > >
>> > > > >Once we had an offline discussion about the Airavata Workflow
>> language
>> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
>> came
>> > > out
>> > > > >was why we need to invent a different language when a simple
>> library
>> > > like
>> > > > >Python will full fill of Airavata requirements.
>> > > > >
>> > > > >There are many benefits in using a Python library as the API for
>> > > > >controlling Airavata workflows.
>> > > > >
>> > > > >1. It is a library, gives the ultimate control over the execution
>> and
>> > it
>> > > > >can be simpler than any domain specific language that we can come
>> with
>> > > > >like
>> > > > >XML, JSON etc
>> > > > >2. Most people use python and can learn it easily than any Domain
>> > > specific
>> > > > >language
>> > > > >3. You can easily create a Python library for Airavata because all
>> the
>> > > > >APIs
>> > > > >are thrift based.
>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>> > > > >
>> > > > >Any thoughts?
>> > > > >
>> > > > >Thanks,
>> > > > >Supun..
>> > > > >
>> > > > >
>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
>> > supun06@gmail.com
>> > > >
>> > > > >wrote:
>> > > > >
>> > > > >> Hi Shameera,
>> > > > >>
>> > > > >> Why you prefer JavaScript over a language like Python?
>> > > > >>
>> > > > >> Thanks,
>> > > > >> Supun..
>> > > > >>
>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> > > > >> shameerainfo@gmail.com> wrote:
>> > > > >>
>> > > > >>> Hi,
>> > > > >>>
>> > > > >>> First of all thanks everyone for giving valuable inputs. After
>> > doing
>> > > > >>>some
>> > > > >>> background search and talking to different people in the
>> University
>> > > > >>>who has
>> > > > >>> used different workflow languages, I myself convinced that
>> > > introducing
>> > > > >>>an
>> > > > >>> another workflow language is not what actually they need. By
>> > changing
>> > > > >>> exiting workflow language to another will not solve problems.
>> What
>> > > they
>> > > > >>> asking is a easy way to construct the workflows. Indirectly what
>> > they
>> > > > >>> asking for a sort of API which they can use to generate the
>> > workflows
>> > > > >>>and
>> > > > >>> run it. Correct me if i am wrong here.
>> > > > >>>
>> > > > >>> As most of above replies depict, if we can get a simple API,
>> as an
>> > > > >>> example, for a web based application, JavaScript API would be a
>> > good
>> > > > >>> solution, and probably JSON would be a good candidate for
>> language,
>> > > > >>>instead
>> > > > >>> of XML.
>> > > > >>>
>> > > > >>> Airavata community already have started to implement web base
>> GUI.
>> > > > >>>Hence
>> > > > >>> introducing a JSON base JavaScript API would be great help.
>> WDYT?
>> > > > >>>
>> > > > >>> Thanks,
>> > > > >>> Shameera.
>> > > > >>>
>> > > > >>>
>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> > > > >>> alekny7@gmail.com> wrote:
>> > > > >>>
>> > > > >>>> Hi,
>> > > > >>>>
>> > > > >>>> it is not dataflow instead focused on orchestrating REST
>> services
>> > > but
>> > > > >>>> you may find it useful datapoint - we created worfklow service
>> > that
>> > > > >>>>uses
>> > > > >>>> natively JavaScript and JSON to describe what happens during
>> > > workflow
>> > > > >>>> execution:
>> > > > >>>>
>> > > >
>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>> > > > >>>>
>> > > > >>>> HTH,
>> > > > >>>>
>> > > > >>>> Alek
>> > > > >>>>
>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
>> smarru@apache.org>
>> > > > >>>>wrote:
>> > > > >>>>
>> > > > >>>>> Hi Chris,
>> > > > >>>>>
>> > > > >>>>> Great to hear OODT community will be interested in adopting a
>> > JSON
>> > > > >>>>> based workflow language and potentially a web based composer
>> as
>> > > well.
>> > > > >>>>> Airavata previously had BPEL support initially through a home
>> > grown
>> > > > >>>>> implementation [1] by Alek Slominski and later through Apache
>> ODE
>> > > > >>>>>[2]. Also
>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
>> read.
>> > > > >>>>>
>> > > > >>>>> I am of the same opinion that we should adopt something more
>> > modern
>> > > > >>>>>as
>> > > > >>>>> the challenges from scientific workflows seems to be
>> converging
>> > > with
>> > > > >>>>>the
>> > > > >>>>> data flow patterns in business workflows.
>> > > > >>>>>
>> > > > >>>>> It will be great if we can all compile a list of potential
>> > > candidates
>> > > > >>>>> and hack them through.
>> > > > >>>>>
>> > > > >>>>> Suresh
>> > > > >>>>> [1] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>> > > > >>>>> [2] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>> > > > >>>>>time_for_eScience_workflows
>> > > > >>>>> [3] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>> > > > >>>>>6.pdf
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>> > > > >>>>>
>> > > > >>>>> > Hi Guys,
>> > > > >>>>> >
>> > > > >>>>> > I've been interested in this too - we don't per have a
>> specific
>> > > > >>>>> > OODT workflow language, but we specific workflows using XML,
>> > and
>> > > > >>>>> > other configuration (we are also thinking of moving to JSON
>> for
>> > > > >>>>> > this).
>> > > > >>>>> >
>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
>> > complex
>> > > > >>>>> > to me.
>> > > > >>>>> >
>> > > > >>>>> > I wonder at the end of the day if we should adopt something
>> > more
>> > > > >>>>> > modern like PIG or some other data flow type of language
>> (PIG
>> > > > >>>>> > is really neat).
>> > > > >>>>> >
>> > > > >>>>> > Cheers,
>> > > > >>>>> > Chris
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> > Chris Mattmann, Ph.D.
>> > > > >>>>> > Chief Architect
>> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
>> > > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> > -----Original Message-----
>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
>> > > > >>>>> > <ar...@airavata.apache.org>
>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
>> > > > >>>>> > To: "architecture@airavata.apache.org" <
>> > > > >>>>> architecture@airavata.apache.org>,
>> > > > >>>>> > dev <de...@airavata.apache.org>
>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
>> > > > >>>>>Airavata.
>> > > > >>>>> >
>> > > > >>>>> >> Hi All,
>> > > > >>>>> >>
>> > > > >>>>> >> As we all know Airavata has its own workflow language call
>> > XWF.
>> > > > >>>>>When
>> > > > >>>>> XWF
>> > > > >>>>> >> was introduced, main focus points are interoperability and
>> > > > >>>>> convertibility.
>> > > > >>>>> >> But with years of experience it is convinced that above
>> > > > >>>>>requirements
>> > > > >>>>> are
>> > > > >>>>> >> not really useful when we come to real world use cases. And
>> > XWF
>> > > is
>> > > > >>>>> XML
>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
>> image
>> > > it
>> > > > >>>>> self.
>> > > > >>>>> >> But
>> > > > >>>>> >> with the recent changes WSDL part is being removed from
>> XWF.
>> > > > >>>>> >>
>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
>> languages in
>> > > > >>>>> industry
>> > > > >>>>> >> and
>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
>> need
>> > to
>> > > > >>>>> come up
>> > > > >>>>> >> with idea how we should improve Airavata workflow language,
>> > > either
>> > > > >>>>> we can
>> > > > >>>>> >> improve existing XWF language, totally change to a new
>> > language
>> > > > >>>>> available
>> > > > >>>>> >> in industry or write a new light weight language. Basic
>> > > > >>>>>requirements
>> > > > >>>>> that
>> > > > >>>>> >> we expect from new improvement are, high usability,
>> flexible,
>> > > > >>>>>light
>> > > > >>>>> weight
>> > > > >>>>> >> and real time monitoring support. As you can see above
>> > > > >>>>>requirements
>> > > > >>>>> are
>> > > > >>>>> >> not
>> > > > >>>>> >> direct comes with workflow languages but we need workflow
>> > > language
>> > > > >>>>> which
>> > > > >>>>> >> help to support above requirements.
>> > > > >>>>> >>
>> > > > >>>>> >> After reading few papers and googling, initially i have
>> come
>> > up
>> > > > >>>>>with
>> > > > >>>>> >> following three existing languages,
>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>> > > > >>>>> >> 2. WS-BPEL
>> > > > >>>>> >> 3. SIDL
>> > > > >>>>> >> <
>> > > http://computation.llnl.gov/casc/components/index.html#page=home
>> > > > >
>> > > > >>>>> >>
>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
>> > > > >>>>> Radical-SAGA
>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
>> above
>> > > > >>>>>three
>> > > > >>>>> >> languages we can come up with simple workflow language
>> base on
>> > > > >>>>> json(or
>> > > > >>>>> >> yaml) which support all our requirements for some extends.
>> > > > >>>>> >>
>> > > > >>>>> >> It would be grate if I can get more input regarding the
>> > $Subject
>> > > > >>>>> form the
>> > > > >>>>> >> airavata community. You all are more than welcome to
>> provide
>> > any
>> > > > >>>>> type of
>> > > > >>>>> >> suggestions.
>> > > > >>>>> >>
>> > > > >>>>> >> Thanks,
>> > > > >>>>> >> Shameera.
>> > > > >>>>> >>
>> > > > >>>>> >>
>> > > > >>>>> >>
>> > > > >>>>> >> --
>> > > > >>>>> >> Best Regards,
>> > > > >>>>> >> Shameera Rathnayaka.
>> > > > >>>>> >>
>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> --
>> > > > >>>> The best way to predict the future is to invent it - Alan Kay
>> > > > >>>>
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> --
>> > > > >>> Best Regards,
>> > > > >>> Shameera Rathnayaka.
>> > > > >>>
>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
>> > > > >>>
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >> --
>> > > > >> Supun Kamburugamuva
>> > > > >> Member, Apache Software Foundation; http://www.apache.org
>> > > > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> > > > >> Blog: http://supunk.blogspot.com
>> > > > >>
>> > > > >>
>> > > > >
>> > > > >
>> > > > >--
>> > > > >Supun Kamburugamuva
>> > > > >Member, Apache Software Foundation; http://www.apache.org
>> > > > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> > > > >Blog: http://supunk.blogspot.com
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Best Regards,
>> > > Shameera Rathnayaka.
>> > >
>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > Blog : http://shameerarathnayaka.blogspot.com/
>> > >
>> >
>> >
>> >
>> > --
>> > Supun Kamburugamuva
>> > Member, Apache Software Foundation; http://www.apache.org
>> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> > Blog: http://supunk.blogspot.com
>> >
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Supun Kamburugamuva <su...@gmail.com>.
I think I'm not suggesting to create a Workflow interpreter using Python
etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
and move it to a more higher level component. The more I think about it,
the model I'm suggesting is similar to what Hadoop, Storm etc has done for
distributed system computations. This model is proven to be successful over
the years.
Keeping what Airavata does at its core can help you to build a more robust
system. If we look at Airavata as middleware to execute applications on
computing resources we can simplify what Airavata does and focus on
improving the core functionality. All the successful systems have thrived
on defining what it does at its core and keeping it simple and being
excellent at what it does. In that regard keeping workflow aspect out of
Airavata can help you to focus on the core problem. That is to execute an
application on a remote computing resource in a fault tolerant and scalable
manner.
What I'm suggesting is to give the Orchestrator the capability to execute a
Driver program that is specified by the user. (This program can be written
in Python, Java or any other language). This driver program is similar to
what you define in a Hadoop or Storm configuration. The driver program
specifies the flow of the computation. It specifies what are the
applications needs to be executed, how to manipulate input output. The
driver program is the workflow for the user. Because the user specifies the
program he can program it to handle workflow steering etc. Every time the
user wants to execute this program he can tell Airavata to execute the
Driver program.
I'm also not 100% sure about all the details. But this can be a new way to
look at how systems like Airavata should behave. Your thoughts and
suggestions are more than welcome.
Thanks,
Supun..
On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:
> Hi Supun,
>
> I think we were in two context, because I as suggesting a way to serialize
> and deserialize the workflow description while you are suggesting to
> implement
> some kind of workflow interpreter using Python, where Python client can
> send
> thrift calls to Airavata server to run the application. I can see with your
> suggested
> approach we can control the workflow execution process from client side
> which
> make it easy to implement workflow debugger.As you mentioned this is a
> major change
> to Airavata. So we should neatly think as this will change our existing
> architecture.
>
> Still if someone need to use different language java, php, JS etc ... to
> run the same
> workflow which generated by Python, we need a language independent workflow
> description.
> My initial question was what is the best language for this?. But as I have
> explained in
> one of my previous reply, It is not matter what language we used Either we
> can use
> XML or JSON to write this description, what matters is how easy to generate
> workflow with the provided API. Hence it would be great to have set of neat
> APIs in
> different languages.
>
> Thanks,
> Shameera.
>
>
> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
> > Hi Shameera,
> >
> > Using python is a radical approach for workflow I think. But I believe
> this
> > is a very beautiful and new approach.Here is a possible scenario for
> > implementation and running using a Python based library.
> >
> > The Python library facilitates the creation of Applications and
> submitting
> > them to Airavata for execution. The underlying functionality is exactly
> > similar to what Java clients provides.The only difference is that, Python
> > library should have a more fluent API than Java for easy creation of
> > workflows. We can generate the Python clients that talk to Airavata
> server
> > using Thrift.
> >
> > Here is an example off the top of my head to a Python script created by
> > user for a Workflow. This is a very crude example and we need to come up
> > with a much better API if we are going to go along this path. First we
> need
> > to write a Python script that can execute a workflow using Airavata.
> >
> > import airavata
> >
> > host = Host("localhost", ....)
> > app1 = Application(host, ....)
> > app2 = Application(host, ....)
> >
> > # we will connect these applications as a workflow using some topology
> > builder or other constructs
> >
> > wb = WorkFlowBuilder()
> > wb.setApp("name1", app1)
> > # you can do a simple output transformation here etc
> >
> > # connects the input of app1 to app2 etc
> > wb.setApp("name2", app2).connectInput("name1")
> >
> > wb.submit()
> >
> > Now we can load this Python script from XBaya. When XBaya loads this
> script
> > the Python script can output an XML configuration of the topology, XBaya
> > can render. There are other ways like directly executing the Python
> script
> > from command line and connecting XBaya indirectly as well. Now you can
> run
> > the workflow from XBaya. Running the Workflow means just executing the
> > Python script.
> >
> > XBaya gets the notifications through messaging and update the UI
> > accordingly.
> >
> > The users need to write the Python script by hand. XBaya cannot create
> the
> > script. Because workflow language is an actual python program the
> benefits
> > are immense. For example user can do workflow steering in the workflow
> > itself by subscribing to messages from Airavata.
> >
> > Thanks,
> > Supun..
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> > shameerainfo@gmail.com> wrote:
> >
> > > Hi Supun,
> > >
> > > I meant to say JS is a well-known client side scripting language i have
> > > messed scripting part. Even we use Python, ultimately we should convert
> > > this to java model in server side, somehow we need to serialized python
> > > representation to the language which java can parse and generate that
> > > model. In this case we need to parse python script in java isn't it? I
> am
> > > not exactly clear how you suggesting to use python here. More details
> on
> > > how end system works if we use Python would be great help to clearly
> > > understand your points.
> > >
> > > Thanks,
> > > Shameera.
> > >
> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> > chris.mattmann@gmail.com>
> > > wrote:
> > >
> > > > Have you guys considered using JCC [1] as a means
> > > > to expose the workflow API currently in Java as a
> > > > Python API?
> > > >
> > > > We are exploring its use in OODT, and we have already
> > > > created a Tika [2] JCC-based python API.
> > > >
> > > > Cheers,
> > > > Chris
> > > >
> > > > [1] http://lucene.apache.org/pylucene/jcc/
> > > > [2] http://github.com/chrismattmann/tika-python/
> > > >
> > > > ------------------------
> > > > Chris Mattmann
> > > > chris.mattmann@gmail.com
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Supun Kamburugamuva <su...@gmail.com>
> > > > Reply-To: <de...@oodt.apache.org>
> > > > Date: Thursday, October 16, 2014 at 3:43 PM
> > > > To: dev <de...@airavata.apache.org>
> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> > > > <de...@oodt.apache.org>
> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> > Airavata.
> > > >
> > > > >Once we had an offline discussion about the Airavata Workflow
> language
> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
> came
> > > out
> > > > >was why we need to invent a different language when a simple library
> > > like
> > > > >Python will full fill of Airavata requirements.
> > > > >
> > > > >There are many benefits in using a Python library as the API for
> > > > >controlling Airavata workflows.
> > > > >
> > > > >1. It is a library, gives the ultimate control over the execution
> and
> > it
> > > > >can be simpler than any domain specific language that we can come
> with
> > > > >like
> > > > >XML, JSON etc
> > > > >2. Most people use python and can learn it easily than any Domain
> > > specific
> > > > >language
> > > > >3. You can easily create a Python library for Airavata because all
> the
> > > > >APIs
> > > > >are thrift based.
> > > > >4. If you design the constructs correctly you can plug an XBaya.
> > > > >
> > > > >Any thoughts?
> > > > >
> > > > >Thanks,
> > > > >Supun..
> > > > >
> > > > >
> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> > supun06@gmail.com
> > > >
> > > > >wrote:
> > > > >
> > > > >> Hi Shameera,
> > > > >>
> > > > >> Why you prefer JavaScript over a language like Python?
> > > > >>
> > > > >> Thanks,
> > > > >> Supun..
> > > > >>
> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > > > >> shameerainfo@gmail.com> wrote:
> > > > >>
> > > > >>> Hi,
> > > > >>>
> > > > >>> First of all thanks everyone for giving valuable inputs. After
> > doing
> > > > >>>some
> > > > >>> background search and talking to different people in the
> University
> > > > >>>who has
> > > > >>> used different workflow languages, I myself convinced that
> > > introducing
> > > > >>>an
> > > > >>> another workflow language is not what actually they need. By
> > changing
> > > > >>> exiting workflow language to another will not solve problems.
> What
> > > they
> > > > >>> asking is a easy way to construct the workflows. Indirectly what
> > they
> > > > >>> asking for a sort of API which they can use to generate the
> > workflows
> > > > >>>and
> > > > >>> run it. Correct me if i am wrong here.
> > > > >>>
> > > > >>> As most of above replies depict, if we can get a simple API, as
> an
> > > > >>> example, for a web based application, JavaScript API would be a
> > good
> > > > >>> solution, and probably JSON would be a good candidate for
> language,
> > > > >>>instead
> > > > >>> of XML.
> > > > >>>
> > > > >>> Airavata community already have started to implement web base
> GUI.
> > > > >>>Hence
> > > > >>> introducing a JSON base JavaScript API would be great help. WDYT?
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Shameera.
> > > > >>>
> > > > >>>
> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> > > > >>> alekny7@gmail.com> wrote:
> > > > >>>
> > > > >>>> Hi,
> > > > >>>>
> > > > >>>> it is not dataflow instead focused on orchestrating REST
> services
> > > but
> > > > >>>> you may find it useful datapoint - we created worfklow service
> > that
> > > > >>>>uses
> > > > >>>> natively JavaScript and JSON to describe what happens during
> > > workflow
> > > > >>>> execution:
> > > > >>>>
> > > >
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> > > > >>>>
> > > > >>>> HTH,
> > > > >>>>
> > > > >>>> Alek
> > > > >>>>
> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
> smarru@apache.org>
> > > > >>>>wrote:
> > > > >>>>
> > > > >>>>> Hi Chris,
> > > > >>>>>
> > > > >>>>> Great to hear OODT community will be interested in adopting a
> > JSON
> > > > >>>>> based workflow language and potentially a web based composer as
> > > well.
> > > > >>>>> Airavata previously had BPEL support initially through a home
> > grown
> > > > >>>>> implementation [1] by Alek Slominski and later through Apache
> ODE
> > > > >>>>>[2]. Also
> > > > >>>>> a white paper [3] by Alek on this topic is an interesting read.
> > > > >>>>>
> > > > >>>>> I am of the same opinion that we should adopt something more
> > modern
> > > > >>>>>as
> > > > >>>>> the challenges from scientific workflows seems to be converging
> > > with
> > > > >>>>>the
> > > > >>>>> data flow patterns in business workflows.
> > > > >>>>>
> > > > >>>>> It will be great if we can all compile a list of potential
> > > candidates
> > > > >>>>> and hack them through.
> > > > >>>>>
> > > > >>>>> Suresh
> > > > >>>>> [1] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > > > >>>>> [2] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> > > > >>>>>time_for_eScience_workflows
> > > > >>>>> [3] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> > > > >>>>>6.pdf
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> > > > >>>>>
> > > > >>>>> > Hi Guys,
> > > > >>>>> >
> > > > >>>>> > I've been interested in this too - we don't per have a
> specific
> > > > >>>>> > OODT workflow language, but we specific workflows using XML,
> > and
> > > > >>>>> > other configuration (we are also thinking of moving to JSON
> for
> > > > >>>>> > this).
> > > > >>>>> >
> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
> > complex
> > > > >>>>> > to me.
> > > > >>>>> >
> > > > >>>>> > I wonder at the end of the day if we should adopt something
> > more
> > > > >>>>> > modern like PIG or some other data flow type of language (PIG
> > > > >>>>> > is really neat).
> > > > >>>>> >
> > > > >>>>> > Cheers,
> > > > >>>>> > Chris
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> > Chris Mattmann, Ph.D.
> > > > >>>>> > Chief Architect
> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > > >>>>> > Office: 168-519, Mailstop: 168-527
> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
> > > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
> > > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> > -----Original Message-----
> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
> > > > >>>>> > <ar...@airavata.apache.org>
> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> > > > >>>>> > To: "architecture@airavata.apache.org" <
> > > > >>>>> architecture@airavata.apache.org>,
> > > > >>>>> > dev <de...@airavata.apache.org>
> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> > > > >>>>>Airavata.
> > > > >>>>> >
> > > > >>>>> >> Hi All,
> > > > >>>>> >>
> > > > >>>>> >> As we all know Airavata has its own workflow language call
> > XWF.
> > > > >>>>>When
> > > > >>>>> XWF
> > > > >>>>> >> was introduced, main focus points are interoperability and
> > > > >>>>> convertibility.
> > > > >>>>> >> But with years of experience it is convinced that above
> > > > >>>>>requirements
> > > > >>>>> are
> > > > >>>>> >> not really useful when we come to real world use cases. And
> > XWF
> > > is
> > > > >>>>> XML
> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
> image
> > > it
> > > > >>>>> self.
> > > > >>>>> >> But
> > > > >>>>> >> with the recent changes WSDL part is being removed from XWF.
> > > > >>>>> >>
> > > > >>>>> >> It is worth to evaluate handy Scientific workflow languages
> in
> > > > >>>>> industry
> > > > >>>>> >> and
> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
> need
> > to
> > > > >>>>> come up
> > > > >>>>> >> with idea how we should improve Airavata workflow language,
> > > either
> > > > >>>>> we can
> > > > >>>>> >> improve existing XWF language, totally change to a new
> > language
> > > > >>>>> available
> > > > >>>>> >> in industry or write a new light weight language. Basic
> > > > >>>>>requirements
> > > > >>>>> that
> > > > >>>>> >> we expect from new improvement are, high usability,
> flexible,
> > > > >>>>>light
> > > > >>>>> weight
> > > > >>>>> >> and real time monitoring support. As you can see above
> > > > >>>>>requirements
> > > > >>>>> are
> > > > >>>>> >> not
> > > > >>>>> >> direct comes with workflow languages but we need workflow
> > > language
> > > > >>>>> which
> > > > >>>>> >> help to support above requirements.
> > > > >>>>> >>
> > > > >>>>> >> After reading few papers and googling, initially i have come
> > up
> > > > >>>>>with
> > > > >>>>> >> following three existing languages,
> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> > > > >>>>> >> 2. WS-BPEL
> > > > >>>>> >> 3. SIDL
> > > > >>>>> >> <
> > > http://computation.llnl.gov/casc/components/index.html#page=home
> > > > >
> > > > >>>>> >>
> > > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> > > > >>>>> Radical-SAGA
> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
> above
> > > > >>>>>three
> > > > >>>>> >> languages we can come up with simple workflow language base
> on
> > > > >>>>> json(or
> > > > >>>>> >> yaml) which support all our requirements for some extends.
> > > > >>>>> >>
> > > > >>>>> >> It would be grate if I can get more input regarding the
> > $Subject
> > > > >>>>> form the
> > > > >>>>> >> airavata community. You all are more than welcome to provide
> > any
> > > > >>>>> type of
> > > > >>>>> >> suggestions.
> > > > >>>>> >>
> > > > >>>>> >> Thanks,
> > > > >>>>> >> Shameera.
> > > > >>>>> >>
> > > > >>>>> >>
> > > > >>>>> >>
> > > > >>>>> >> --
> > > > >>>>> >> Best Regards,
> > > > >>>>> >> Shameera Rathnayaka.
> > > > >>>>> >>
> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> The best way to predict the future is to invent it - Alan Kay
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Best Regards,
> > > > >>> Shameera Rathnayaka.
> > > > >>>
> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
> > > > >>>
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Supun Kamburugamuva
> > > > >> Member, Apache Software Foundation; http://www.apache.org
> > > > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > > > >> Blog: http://supunk.blogspot.com
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > >--
> > > > >Supun Kamburugamuva
> > > > >Member, Apache Software Foundation; http://www.apache.org
> > > > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > > > >Blog: http://supunk.blogspot.com
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Best Regards,
> > > Shameera Rathnayaka.
> > >
> > > email: shameera AT apache.org , shameerainfo AT gmail.com
> > > Blog : http://shameerarathnayaka.blogspot.com/
> > >
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Supun Kamburugamuva <su...@gmail.com>.
I think I'm not suggesting to create a Workflow interpreter using Python
etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
and move it to a more higher level component. The more I think about it,
the model I'm suggesting is similar to what Hadoop, Storm etc has done for
distributed system computations. This model is proven to be successful over
the years.
Keeping what Airavata does at its core can help you to build a more robust
system. If we look at Airavata as middleware to execute applications on
computing resources we can simplify what Airavata does and focus on
improving the core functionality. All the successful systems have thrived
on defining what it does at its core and keeping it simple and being
excellent at what it does. In that regard keeping workflow aspect out of
Airavata can help you to focus on the core problem. That is to execute an
application on a remote computing resource in a fault tolerant and scalable
manner.
What I'm suggesting is to give the Orchestrator the capability to execute a
Driver program that is specified by the user. (This program can be written
in Python, Java or any other language). This driver program is similar to
what you define in a Hadoop or Storm configuration. The driver program
specifies the flow of the computation. It specifies what are the
applications needs to be executed, how to manipulate input output. The
driver program is the workflow for the user. Because the user specifies the
program he can program it to handle workflow steering etc. Every time the
user wants to execute this program he can tell Airavata to execute the
Driver program.
I'm also not 100% sure about all the details. But this can be a new way to
look at how systems like Airavata should behave. Your thoughts and
suggestions are more than welcome.
Thanks,
Supun..
On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:
> Hi Supun,
>
> I think we were in two context, because I as suggesting a way to serialize
> and deserialize the workflow description while you are suggesting to
> implement
> some kind of workflow interpreter using Python, where Python client can
> send
> thrift calls to Airavata server to run the application. I can see with your
> suggested
> approach we can control the workflow execution process from client side
> which
> make it easy to implement workflow debugger.As you mentioned this is a
> major change
> to Airavata. So we should neatly think as this will change our existing
> architecture.
>
> Still if someone need to use different language java, php, JS etc ... to
> run the same
> workflow which generated by Python, we need a language independent workflow
> description.
> My initial question was what is the best language for this?. But as I have
> explained in
> one of my previous reply, It is not matter what language we used Either we
> can use
> XML or JSON to write this description, what matters is how easy to generate
> workflow with the provided API. Hence it would be great to have set of neat
> APIs in
> different languages.
>
> Thanks,
> Shameera.
>
>
> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
> > Hi Shameera,
> >
> > Using python is a radical approach for workflow I think. But I believe
> this
> > is a very beautiful and new approach.Here is a possible scenario for
> > implementation and running using a Python based library.
> >
> > The Python library facilitates the creation of Applications and
> submitting
> > them to Airavata for execution. The underlying functionality is exactly
> > similar to what Java clients provides.The only difference is that, Python
> > library should have a more fluent API than Java for easy creation of
> > workflows. We can generate the Python clients that talk to Airavata
> server
> > using Thrift.
> >
> > Here is an example off the top of my head to a Python script created by
> > user for a Workflow. This is a very crude example and we need to come up
> > with a much better API if we are going to go along this path. First we
> need
> > to write a Python script that can execute a workflow using Airavata.
> >
> > import airavata
> >
> > host = Host("localhost", ....)
> > app1 = Application(host, ....)
> > app2 = Application(host, ....)
> >
> > # we will connect these applications as a workflow using some topology
> > builder or other constructs
> >
> > wb = WorkFlowBuilder()
> > wb.setApp("name1", app1)
> > # you can do a simple output transformation here etc
> >
> > # connects the input of app1 to app2 etc
> > wb.setApp("name2", app2).connectInput("name1")
> >
> > wb.submit()
> >
> > Now we can load this Python script from XBaya. When XBaya loads this
> script
> > the Python script can output an XML configuration of the topology, XBaya
> > can render. There are other ways like directly executing the Python
> script
> > from command line and connecting XBaya indirectly as well. Now you can
> run
> > the workflow from XBaya. Running the Workflow means just executing the
> > Python script.
> >
> > XBaya gets the notifications through messaging and update the UI
> > accordingly.
> >
> > The users need to write the Python script by hand. XBaya cannot create
> the
> > script. Because workflow language is an actual python program the
> benefits
> > are immense. For example user can do workflow steering in the workflow
> > itself by subscribing to messages from Airavata.
> >
> > Thanks,
> > Supun..
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> > shameerainfo@gmail.com> wrote:
> >
> > > Hi Supun,
> > >
> > > I meant to say JS is a well-known client side scripting language i have
> > > messed scripting part. Even we use Python, ultimately we should convert
> > > this to java model in server side, somehow we need to serialized python
> > > representation to the language which java can parse and generate that
> > > model. In this case we need to parse python script in java isn't it? I
> am
> > > not exactly clear how you suggesting to use python here. More details
> on
> > > how end system works if we use Python would be great help to clearly
> > > understand your points.
> > >
> > > Thanks,
> > > Shameera.
> > >
> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> > chris.mattmann@gmail.com>
> > > wrote:
> > >
> > > > Have you guys considered using JCC [1] as a means
> > > > to expose the workflow API currently in Java as a
> > > > Python API?
> > > >
> > > > We are exploring its use in OODT, and we have already
> > > > created a Tika [2] JCC-based python API.
> > > >
> > > > Cheers,
> > > > Chris
> > > >
> > > > [1] http://lucene.apache.org/pylucene/jcc/
> > > > [2] http://github.com/chrismattmann/tika-python/
> > > >
> > > > ------------------------
> > > > Chris Mattmann
> > > > chris.mattmann@gmail.com
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Supun Kamburugamuva <su...@gmail.com>
> > > > Reply-To: <de...@oodt.apache.org>
> > > > Date: Thursday, October 16, 2014 at 3:43 PM
> > > > To: dev <de...@airavata.apache.org>
> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> > > > <de...@oodt.apache.org>
> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> > Airavata.
> > > >
> > > > >Once we had an offline discussion about the Airavata Workflow
> language
> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
> came
> > > out
> > > > >was why we need to invent a different language when a simple library
> > > like
> > > > >Python will full fill of Airavata requirements.
> > > > >
> > > > >There are many benefits in using a Python library as the API for
> > > > >controlling Airavata workflows.
> > > > >
> > > > >1. It is a library, gives the ultimate control over the execution
> and
> > it
> > > > >can be simpler than any domain specific language that we can come
> with
> > > > >like
> > > > >XML, JSON etc
> > > > >2. Most people use python and can learn it easily than any Domain
> > > specific
> > > > >language
> > > > >3. You can easily create a Python library for Airavata because all
> the
> > > > >APIs
> > > > >are thrift based.
> > > > >4. If you design the constructs correctly you can plug an XBaya.
> > > > >
> > > > >Any thoughts?
> > > > >
> > > > >Thanks,
> > > > >Supun..
> > > > >
> > > > >
> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> > supun06@gmail.com
> > > >
> > > > >wrote:
> > > > >
> > > > >> Hi Shameera,
> > > > >>
> > > > >> Why you prefer JavaScript over a language like Python?
> > > > >>
> > > > >> Thanks,
> > > > >> Supun..
> > > > >>
> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > > > >> shameerainfo@gmail.com> wrote:
> > > > >>
> > > > >>> Hi,
> > > > >>>
> > > > >>> First of all thanks everyone for giving valuable inputs. After
> > doing
> > > > >>>some
> > > > >>> background search and talking to different people in the
> University
> > > > >>>who has
> > > > >>> used different workflow languages, I myself convinced that
> > > introducing
> > > > >>>an
> > > > >>> another workflow language is not what actually they need. By
> > changing
> > > > >>> exiting workflow language to another will not solve problems.
> What
> > > they
> > > > >>> asking is a easy way to construct the workflows. Indirectly what
> > they
> > > > >>> asking for a sort of API which they can use to generate the
> > workflows
> > > > >>>and
> > > > >>> run it. Correct me if i am wrong here.
> > > > >>>
> > > > >>> As most of above replies depict, if we can get a simple API, as
> an
> > > > >>> example, for a web based application, JavaScript API would be a
> > good
> > > > >>> solution, and probably JSON would be a good candidate for
> language,
> > > > >>>instead
> > > > >>> of XML.
> > > > >>>
> > > > >>> Airavata community already have started to implement web base
> GUI.
> > > > >>>Hence
> > > > >>> introducing a JSON base JavaScript API would be great help. WDYT?
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Shameera.
> > > > >>>
> > > > >>>
> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> > > > >>> alekny7@gmail.com> wrote:
> > > > >>>
> > > > >>>> Hi,
> > > > >>>>
> > > > >>>> it is not dataflow instead focused on orchestrating REST
> services
> > > but
> > > > >>>> you may find it useful datapoint - we created worfklow service
> > that
> > > > >>>>uses
> > > > >>>> natively JavaScript and JSON to describe what happens during
> > > workflow
> > > > >>>> execution:
> > > > >>>>
> > > >
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> > > > >>>>
> > > > >>>> HTH,
> > > > >>>>
> > > > >>>> Alek
> > > > >>>>
> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
> smarru@apache.org>
> > > > >>>>wrote:
> > > > >>>>
> > > > >>>>> Hi Chris,
> > > > >>>>>
> > > > >>>>> Great to hear OODT community will be interested in adopting a
> > JSON
> > > > >>>>> based workflow language and potentially a web based composer as
> > > well.
> > > > >>>>> Airavata previously had BPEL support initially through a home
> > grown
> > > > >>>>> implementation [1] by Alek Slominski and later through Apache
> ODE
> > > > >>>>>[2]. Also
> > > > >>>>> a white paper [3] by Alek on this topic is an interesting read.
> > > > >>>>>
> > > > >>>>> I am of the same opinion that we should adopt something more
> > modern
> > > > >>>>>as
> > > > >>>>> the challenges from scientific workflows seems to be converging
> > > with
> > > > >>>>>the
> > > > >>>>> data flow patterns in business workflows.
> > > > >>>>>
> > > > >>>>> It will be great if we can all compile a list of potential
> > > candidates
> > > > >>>>> and hack them through.
> > > > >>>>>
> > > > >>>>> Suresh
> > > > >>>>> [1] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > > > >>>>> [2] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> > > > >>>>>time_for_eScience_workflows
> > > > >>>>> [3] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> > > > >>>>>6.pdf
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> > > > >>>>>
> > > > >>>>> > Hi Guys,
> > > > >>>>> >
> > > > >>>>> > I've been interested in this too - we don't per have a
> specific
> > > > >>>>> > OODT workflow language, but we specific workflows using XML,
> > and
> > > > >>>>> > other configuration (we are also thinking of moving to JSON
> for
> > > > >>>>> > this).
> > > > >>>>> >
> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
> > complex
> > > > >>>>> > to me.
> > > > >>>>> >
> > > > >>>>> > I wonder at the end of the day if we should adopt something
> > more
> > > > >>>>> > modern like PIG or some other data flow type of language (PIG
> > > > >>>>> > is really neat).
> > > > >>>>> >
> > > > >>>>> > Cheers,
> > > > >>>>> > Chris
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> > Chris Mattmann, Ph.D.
> > > > >>>>> > Chief Architect
> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > > >>>>> > Office: 168-519, Mailstop: 168-527
> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
> > > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
> > > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> > -----Original Message-----
> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
> > > > >>>>> > <ar...@airavata.apache.org>
> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> > > > >>>>> > To: "architecture@airavata.apache.org" <
> > > > >>>>> architecture@airavata.apache.org>,
> > > > >>>>> > dev <de...@airavata.apache.org>
> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> > > > >>>>>Airavata.
> > > > >>>>> >
> > > > >>>>> >> Hi All,
> > > > >>>>> >>
> > > > >>>>> >> As we all know Airavata has its own workflow language call
> > XWF.
> > > > >>>>>When
> > > > >>>>> XWF
> > > > >>>>> >> was introduced, main focus points are interoperability and
> > > > >>>>> convertibility.
> > > > >>>>> >> But with years of experience it is convinced that above
> > > > >>>>>requirements
> > > > >>>>> are
> > > > >>>>> >> not really useful when we come to real world use cases. And
> > XWF
> > > is
> > > > >>>>> XML
> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
> image
> > > it
> > > > >>>>> self.
> > > > >>>>> >> But
> > > > >>>>> >> with the recent changes WSDL part is being removed from XWF.
> > > > >>>>> >>
> > > > >>>>> >> It is worth to evaluate handy Scientific workflow languages
> in
> > > > >>>>> industry
> > > > >>>>> >> and
> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
> need
> > to
> > > > >>>>> come up
> > > > >>>>> >> with idea how we should improve Airavata workflow language,
> > > either
> > > > >>>>> we can
> > > > >>>>> >> improve existing XWF language, totally change to a new
> > language
> > > > >>>>> available
> > > > >>>>> >> in industry or write a new light weight language. Basic
> > > > >>>>>requirements
> > > > >>>>> that
> > > > >>>>> >> we expect from new improvement are, high usability,
> flexible,
> > > > >>>>>light
> > > > >>>>> weight
> > > > >>>>> >> and real time monitoring support. As you can see above
> > > > >>>>>requirements
> > > > >>>>> are
> > > > >>>>> >> not
> > > > >>>>> >> direct comes with workflow languages but we need workflow
> > > language
> > > > >>>>> which
> > > > >>>>> >> help to support above requirements.
> > > > >>>>> >>
> > > > >>>>> >> After reading few papers and googling, initially i have come
> > up
> > > > >>>>>with
> > > > >>>>> >> following three existing languages,
> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> > > > >>>>> >> 2. WS-BPEL
> > > > >>>>> >> 3. SIDL
> > > > >>>>> >> <
> > > http://computation.llnl.gov/casc/components/index.html#page=home
> > > > >
> > > > >>>>> >>
> > > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> > > > >>>>> Radical-SAGA
> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
> above
> > > > >>>>>three
> > > > >>>>> >> languages we can come up with simple workflow language base
> on
> > > > >>>>> json(or
> > > > >>>>> >> yaml) which support all our requirements for some extends.
> > > > >>>>> >>
> > > > >>>>> >> It would be grate if I can get more input regarding the
> > $Subject
> > > > >>>>> form the
> > > > >>>>> >> airavata community. You all are more than welcome to provide
> > any
> > > > >>>>> type of
> > > > >>>>> >> suggestions.
> > > > >>>>> >>
> > > > >>>>> >> Thanks,
> > > > >>>>> >> Shameera.
> > > > >>>>> >>
> > > > >>>>> >>
> > > > >>>>> >>
> > > > >>>>> >> --
> > > > >>>>> >> Best Regards,
> > > > >>>>> >> Shameera Rathnayaka.
> > > > >>>>> >>
> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> The best way to predict the future is to invent it - Alan Kay
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Best Regards,
> > > > >>> Shameera Rathnayaka.
> > > > >>>
> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
> > > > >>>
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Supun Kamburugamuva
> > > > >> Member, Apache Software Foundation; http://www.apache.org
> > > > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > > > >> Blog: http://supunk.blogspot.com
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > >--
> > > > >Supun Kamburugamuva
> > > > >Member, Apache Software Foundation; http://www.apache.org
> > > > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > > > >Blog: http://supunk.blogspot.com
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Best Regards,
> > > Shameera Rathnayaka.
> > >
> > > email: shameera AT apache.org , shameerainfo AT gmail.com
> > > Blog : http://shameerarathnayaka.blogspot.com/
> > >
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Supun Kamburugamuva <su...@gmail.com>.
I think I'm not suggesting to create a Workflow interpreter using Python
etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
and move it to a more higher level component. The more I think about it,
the model I'm suggesting is similar to what Hadoop, Storm etc has done for
distributed system computations. This model is proven to be successful over
the years.
Keeping what Airavata does at its core can help you to build a more robust
system. If we look at Airavata as middleware to execute applications on
computing resources we can simplify what Airavata does and focus on
improving the core functionality. All the successful systems have thrived
on defining what it does at its core and keeping it simple and being
excellent at what it does. In that regard keeping workflow aspect out of
Airavata can help you to focus on the core problem. That is to execute an
application on a remote computing resource in a fault tolerant and scalable
manner.
What I'm suggesting is to give the Orchestrator the capability to execute a
Driver program that is specified by the user. (This program can be written
in Python, Java or any other language). This driver program is similar to
what you define in a Hadoop or Storm configuration. The driver program
specifies the flow of the computation. It specifies what are the
applications needs to be executed, how to manipulate input output. The
driver program is the workflow for the user. Because the user specifies the
program he can program it to handle workflow steering etc. Every time the
user wants to execute this program he can tell Airavata to execute the
Driver program.
I'm also not 100% sure about all the details. But this can be a new way to
look at how systems like Airavata should behave. Your thoughts and
suggestions are more than welcome.
Thanks,
Supun..
On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:
> Hi Supun,
>
> I think we were in two context, because I as suggesting a way to serialize
> and deserialize the workflow description while you are suggesting to
> implement
> some kind of workflow interpreter using Python, where Python client can
> send
> thrift calls to Airavata server to run the application. I can see with your
> suggested
> approach we can control the workflow execution process from client side
> which
> make it easy to implement workflow debugger.As you mentioned this is a
> major change
> to Airavata. So we should neatly think as this will change our existing
> architecture.
>
> Still if someone need to use different language java, php, JS etc ... to
> run the same
> workflow which generated by Python, we need a language independent workflow
> description.
> My initial question was what is the best language for this?. But as I have
> explained in
> one of my previous reply, It is not matter what language we used Either we
> can use
> XML or JSON to write this description, what matters is how easy to generate
> workflow with the provided API. Hence it would be great to have set of neat
> APIs in
> different languages.
>
> Thanks,
> Shameera.
>
>
> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
> > Hi Shameera,
> >
> > Using python is a radical approach for workflow I think. But I believe
> this
> > is a very beautiful and new approach.Here is a possible scenario for
> > implementation and running using a Python based library.
> >
> > The Python library facilitates the creation of Applications and
> submitting
> > them to Airavata for execution. The underlying functionality is exactly
> > similar to what Java clients provides.The only difference is that, Python
> > library should have a more fluent API than Java for easy creation of
> > workflows. We can generate the Python clients that talk to Airavata
> server
> > using Thrift.
> >
> > Here is an example off the top of my head to a Python script created by
> > user for a Workflow. This is a very crude example and we need to come up
> > with a much better API if we are going to go along this path. First we
> need
> > to write a Python script that can execute a workflow using Airavata.
> >
> > import airavata
> >
> > host = Host("localhost", ....)
> > app1 = Application(host, ....)
> > app2 = Application(host, ....)
> >
> > # we will connect these applications as a workflow using some topology
> > builder or other constructs
> >
> > wb = WorkFlowBuilder()
> > wb.setApp("name1", app1)
> > # you can do a simple output transformation here etc
> >
> > # connects the input of app1 to app2 etc
> > wb.setApp("name2", app2).connectInput("name1")
> >
> > wb.submit()
> >
> > Now we can load this Python script from XBaya. When XBaya loads this
> script
> > the Python script can output an XML configuration of the topology, XBaya
> > can render. There are other ways like directly executing the Python
> script
> > from command line and connecting XBaya indirectly as well. Now you can
> run
> > the workflow from XBaya. Running the Workflow means just executing the
> > Python script.
> >
> > XBaya gets the notifications through messaging and update the UI
> > accordingly.
> >
> > The users need to write the Python script by hand. XBaya cannot create
> the
> > script. Because workflow language is an actual python program the
> benefits
> > are immense. For example user can do workflow steering in the workflow
> > itself by subscribing to messages from Airavata.
> >
> > Thanks,
> > Supun..
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> > shameerainfo@gmail.com> wrote:
> >
> > > Hi Supun,
> > >
> > > I meant to say JS is a well-known client side scripting language i have
> > > messed scripting part. Even we use Python, ultimately we should convert
> > > this to java model in server side, somehow we need to serialized python
> > > representation to the language which java can parse and generate that
> > > model. In this case we need to parse python script in java isn't it? I
> am
> > > not exactly clear how you suggesting to use python here. More details
> on
> > > how end system works if we use Python would be great help to clearly
> > > understand your points.
> > >
> > > Thanks,
> > > Shameera.
> > >
> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> > chris.mattmann@gmail.com>
> > > wrote:
> > >
> > > > Have you guys considered using JCC [1] as a means
> > > > to expose the workflow API currently in Java as a
> > > > Python API?
> > > >
> > > > We are exploring its use in OODT, and we have already
> > > > created a Tika [2] JCC-based python API.
> > > >
> > > > Cheers,
> > > > Chris
> > > >
> > > > [1] http://lucene.apache.org/pylucene/jcc/
> > > > [2] http://github.com/chrismattmann/tika-python/
> > > >
> > > > ------------------------
> > > > Chris Mattmann
> > > > chris.mattmann@gmail.com
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Supun Kamburugamuva <su...@gmail.com>
> > > > Reply-To: <de...@oodt.apache.org>
> > > > Date: Thursday, October 16, 2014 at 3:43 PM
> > > > To: dev <de...@airavata.apache.org>
> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> > > > <de...@oodt.apache.org>
> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> > Airavata.
> > > >
> > > > >Once we had an offline discussion about the Airavata Workflow
> language
> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
> came
> > > out
> > > > >was why we need to invent a different language when a simple library
> > > like
> > > > >Python will full fill of Airavata requirements.
> > > > >
> > > > >There are many benefits in using a Python library as the API for
> > > > >controlling Airavata workflows.
> > > > >
> > > > >1. It is a library, gives the ultimate control over the execution
> and
> > it
> > > > >can be simpler than any domain specific language that we can come
> with
> > > > >like
> > > > >XML, JSON etc
> > > > >2. Most people use python and can learn it easily than any Domain
> > > specific
> > > > >language
> > > > >3. You can easily create a Python library for Airavata because all
> the
> > > > >APIs
> > > > >are thrift based.
> > > > >4. If you design the constructs correctly you can plug an XBaya.
> > > > >
> > > > >Any thoughts?
> > > > >
> > > > >Thanks,
> > > > >Supun..
> > > > >
> > > > >
> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> > supun06@gmail.com
> > > >
> > > > >wrote:
> > > > >
> > > > >> Hi Shameera,
> > > > >>
> > > > >> Why you prefer JavaScript over a language like Python?
> > > > >>
> > > > >> Thanks,
> > > > >> Supun..
> > > > >>
> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > > > >> shameerainfo@gmail.com> wrote:
> > > > >>
> > > > >>> Hi,
> > > > >>>
> > > > >>> First of all thanks everyone for giving valuable inputs. After
> > doing
> > > > >>>some
> > > > >>> background search and talking to different people in the
> University
> > > > >>>who has
> > > > >>> used different workflow languages, I myself convinced that
> > > introducing
> > > > >>>an
> > > > >>> another workflow language is not what actually they need. By
> > changing
> > > > >>> exiting workflow language to another will not solve problems.
> What
> > > they
> > > > >>> asking is a easy way to construct the workflows. Indirectly what
> > they
> > > > >>> asking for a sort of API which they can use to generate the
> > workflows
> > > > >>>and
> > > > >>> run it. Correct me if i am wrong here.
> > > > >>>
> > > > >>> As most of above replies depict, if we can get a simple API, as
> an
> > > > >>> example, for a web based application, JavaScript API would be a
> > good
> > > > >>> solution, and probably JSON would be a good candidate for
> language,
> > > > >>>instead
> > > > >>> of XML.
> > > > >>>
> > > > >>> Airavata community already have started to implement web base
> GUI.
> > > > >>>Hence
> > > > >>> introducing a JSON base JavaScript API would be great help. WDYT?
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Shameera.
> > > > >>>
> > > > >>>
> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> > > > >>> alekny7@gmail.com> wrote:
> > > > >>>
> > > > >>>> Hi,
> > > > >>>>
> > > > >>>> it is not dataflow instead focused on orchestrating REST
> services
> > > but
> > > > >>>> you may find it useful datapoint - we created worfklow service
> > that
> > > > >>>>uses
> > > > >>>> natively JavaScript and JSON to describe what happens during
> > > workflow
> > > > >>>> execution:
> > > > >>>>
> > > >
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> > > > >>>>
> > > > >>>> HTH,
> > > > >>>>
> > > > >>>> Alek
> > > > >>>>
> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
> smarru@apache.org>
> > > > >>>>wrote:
> > > > >>>>
> > > > >>>>> Hi Chris,
> > > > >>>>>
> > > > >>>>> Great to hear OODT community will be interested in adopting a
> > JSON
> > > > >>>>> based workflow language and potentially a web based composer as
> > > well.
> > > > >>>>> Airavata previously had BPEL support initially through a home
> > grown
> > > > >>>>> implementation [1] by Alek Slominski and later through Apache
> ODE
> > > > >>>>>[2]. Also
> > > > >>>>> a white paper [3] by Alek on this topic is an interesting read.
> > > > >>>>>
> > > > >>>>> I am of the same opinion that we should adopt something more
> > modern
> > > > >>>>>as
> > > > >>>>> the challenges from scientific workflows seems to be converging
> > > with
> > > > >>>>>the
> > > > >>>>> data flow patterns in business workflows.
> > > > >>>>>
> > > > >>>>> It will be great if we can all compile a list of potential
> > > candidates
> > > > >>>>> and hack them through.
> > > > >>>>>
> > > > >>>>> Suresh
> > > > >>>>> [1] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > > > >>>>> [2] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> > > > >>>>>time_for_eScience_workflows
> > > > >>>>> [3] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> > > > >>>>>6.pdf
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> > > > >>>>>
> > > > >>>>> > Hi Guys,
> > > > >>>>> >
> > > > >>>>> > I've been interested in this too - we don't per have a
> specific
> > > > >>>>> > OODT workflow language, but we specific workflows using XML,
> > and
> > > > >>>>> > other configuration (we are also thinking of moving to JSON
> for
> > > > >>>>> > this).
> > > > >>>>> >
> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
> > complex
> > > > >>>>> > to me.
> > > > >>>>> >
> > > > >>>>> > I wonder at the end of the day if we should adopt something
> > more
> > > > >>>>> > modern like PIG or some other data flow type of language (PIG
> > > > >>>>> > is really neat).
> > > > >>>>> >
> > > > >>>>> > Cheers,
> > > > >>>>> > Chris
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> > Chris Mattmann, Ph.D.
> > > > >>>>> > Chief Architect
> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > > >>>>> > Office: 168-519, Mailstop: 168-527
> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
> > > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
> > > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> > -----Original Message-----
> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
> > > > >>>>> > <ar...@airavata.apache.org>
> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> > > > >>>>> > To: "architecture@airavata.apache.org" <
> > > > >>>>> architecture@airavata.apache.org>,
> > > > >>>>> > dev <de...@airavata.apache.org>
> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> > > > >>>>>Airavata.
> > > > >>>>> >
> > > > >>>>> >> Hi All,
> > > > >>>>> >>
> > > > >>>>> >> As we all know Airavata has its own workflow language call
> > XWF.
> > > > >>>>>When
> > > > >>>>> XWF
> > > > >>>>> >> was introduced, main focus points are interoperability and
> > > > >>>>> convertibility.
> > > > >>>>> >> But with years of experience it is convinced that above
> > > > >>>>>requirements
> > > > >>>>> are
> > > > >>>>> >> not really useful when we come to real world use cases. And
> > XWF
> > > is
> > > > >>>>> XML
> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
> image
> > > it
> > > > >>>>> self.
> > > > >>>>> >> But
> > > > >>>>> >> with the recent changes WSDL part is being removed from XWF.
> > > > >>>>> >>
> > > > >>>>> >> It is worth to evaluate handy Scientific workflow languages
> in
> > > > >>>>> industry
> > > > >>>>> >> and
> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
> need
> > to
> > > > >>>>> come up
> > > > >>>>> >> with idea how we should improve Airavata workflow language,
> > > either
> > > > >>>>> we can
> > > > >>>>> >> improve existing XWF language, totally change to a new
> > language
> > > > >>>>> available
> > > > >>>>> >> in industry or write a new light weight language. Basic
> > > > >>>>>requirements
> > > > >>>>> that
> > > > >>>>> >> we expect from new improvement are, high usability,
> flexible,
> > > > >>>>>light
> > > > >>>>> weight
> > > > >>>>> >> and real time monitoring support. As you can see above
> > > > >>>>>requirements
> > > > >>>>> are
> > > > >>>>> >> not
> > > > >>>>> >> direct comes with workflow languages but we need workflow
> > > language
> > > > >>>>> which
> > > > >>>>> >> help to support above requirements.
> > > > >>>>> >>
> > > > >>>>> >> After reading few papers and googling, initially i have come
> > up
> > > > >>>>>with
> > > > >>>>> >> following three existing languages,
> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> > > > >>>>> >> 2. WS-BPEL
> > > > >>>>> >> 3. SIDL
> > > > >>>>> >> <
> > > http://computation.llnl.gov/casc/components/index.html#page=home
> > > > >
> > > > >>>>> >>
> > > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> > > > >>>>> Radical-SAGA
> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
> above
> > > > >>>>>three
> > > > >>>>> >> languages we can come up with simple workflow language base
> on
> > > > >>>>> json(or
> > > > >>>>> >> yaml) which support all our requirements for some extends.
> > > > >>>>> >>
> > > > >>>>> >> It would be grate if I can get more input regarding the
> > $Subject
> > > > >>>>> form the
> > > > >>>>> >> airavata community. You all are more than welcome to provide
> > any
> > > > >>>>> type of
> > > > >>>>> >> suggestions.
> > > > >>>>> >>
> > > > >>>>> >> Thanks,
> > > > >>>>> >> Shameera.
> > > > >>>>> >>
> > > > >>>>> >>
> > > > >>>>> >>
> > > > >>>>> >> --
> > > > >>>>> >> Best Regards,
> > > > >>>>> >> Shameera Rathnayaka.
> > > > >>>>> >>
> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> The best way to predict the future is to invent it - Alan Kay
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Best Regards,
> > > > >>> Shameera Rathnayaka.
> > > > >>>
> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
> > > > >>>
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Supun Kamburugamuva
> > > > >> Member, Apache Software Foundation; http://www.apache.org
> > > > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > > > >> Blog: http://supunk.blogspot.com
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > >--
> > > > >Supun Kamburugamuva
> > > > >Member, Apache Software Foundation; http://www.apache.org
> > > > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > > > >Blog: http://supunk.blogspot.com
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Best Regards,
> > > Shameera Rathnayaka.
> > >
> > > email: shameera AT apache.org , shameerainfo AT gmail.com
> > > Blog : http://shameerarathnayaka.blogspot.com/
> > >
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi Supun,
I think we were in two context, because I as suggesting a way to serialize
and deserialize the workflow description while you are suggesting to
implement
some kind of workflow interpreter using Python, where Python client can
send
thrift calls to Airavata server to run the application. I can see with your
suggested
approach we can control the workflow execution process from client side
which
make it easy to implement workflow debugger.As you mentioned this is a
major change
to Airavata. So we should neatly think as this will change our existing
architecture.
Still if someone need to use different language java, php, JS etc ... to
run the same
workflow which generated by Python, we need a language independent workflow
description.
My initial question was what is the best language for this?. But as I have
explained in
one of my previous reply, It is not matter what language we used Either we
can use
XML or JSON to write this description, what matters is how easy to generate
workflow with the provided API. Hence it would be great to have set of neat
APIs in
different languages.
Thanks,
Shameera.
On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:
> Hi Shameera,
>
> Using python is a radical approach for workflow I think. But I believe this
> is a very beautiful and new approach.Here is a possible scenario for
> implementation and running using a Python based library.
>
> The Python library facilitates the creation of Applications and submitting
> them to Airavata for execution. The underlying functionality is exactly
> similar to what Java clients provides.The only difference is that, Python
> library should have a more fluent API than Java for easy creation of
> workflows. We can generate the Python clients that talk to Airavata server
> using Thrift.
>
> Here is an example off the top of my head to a Python script created by
> user for a Workflow. This is a very crude example and we need to come up
> with a much better API if we are going to go along this path. First we need
> to write a Python script that can execute a workflow using Airavata.
>
> import airavata
>
> host = Host("localhost", ....)
> app1 = Application(host, ....)
> app2 = Application(host, ....)
>
> # we will connect these applications as a workflow using some topology
> builder or other constructs
>
> wb = WorkFlowBuilder()
> wb.setApp("name1", app1)
> # you can do a simple output transformation here etc
>
> # connects the input of app1 to app2 etc
> wb.setApp("name2", app2).connectInput("name1")
>
> wb.submit()
>
> Now we can load this Python script from XBaya. When XBaya loads this script
> the Python script can output an XML configuration of the topology, XBaya
> can render. There are other ways like directly executing the Python script
> from command line and connecting XBaya indirectly as well. Now you can run
> the workflow from XBaya. Running the Workflow means just executing the
> Python script.
>
> XBaya gets the notifications through messaging and update the UI
> accordingly.
>
> The users need to write the Python script by hand. XBaya cannot create the
> script. Because workflow language is an actual python program the benefits
> are immense. For example user can do workflow steering in the workflow
> itself by subscribing to messages from Airavata.
>
> Thanks,
> Supun..
>
>
>
>
>
>
>
>
> On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
> > Hi Supun,
> >
> > I meant to say JS is a well-known client side scripting language i have
> > messed scripting part. Even we use Python, ultimately we should convert
> > this to java model in server side, somehow we need to serialized python
> > representation to the language which java can parse and generate that
> > model. In this case we need to parse python script in java isn't it? I am
> > not exactly clear how you suggesting to use python here. More details on
> > how end system works if we use Python would be great help to clearly
> > understand your points.
> >
> > Thanks,
> > Shameera.
> >
> > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> chris.mattmann@gmail.com>
> > wrote:
> >
> > > Have you guys considered using JCC [1] as a means
> > > to expose the workflow API currently in Java as a
> > > Python API?
> > >
> > > We are exploring its use in OODT, and we have already
> > > created a Tika [2] JCC-based python API.
> > >
> > > Cheers,
> > > Chris
> > >
> > > [1] http://lucene.apache.org/pylucene/jcc/
> > > [2] http://github.com/chrismattmann/tika-python/
> > >
> > > ------------------------
> > > Chris Mattmann
> > > chris.mattmann@gmail.com
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Supun Kamburugamuva <su...@gmail.com>
> > > Reply-To: <de...@oodt.apache.org>
> > > Date: Thursday, October 16, 2014 at 3:43 PM
> > > To: dev <de...@airavata.apache.org>
> > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> > > <sm...@apache.org>, "architecture@airavata.apache.org"
> > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> > > <de...@oodt.apache.org>
> > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> Airavata.
> > >
> > > >Once we had an offline discussion about the Airavata Workflow language
> > > >(with Milinda, Saliya and Shameera). In that discussion one thing came
> > out
> > > >was why we need to invent a different language when a simple library
> > like
> > > >Python will full fill of Airavata requirements.
> > > >
> > > >There are many benefits in using a Python library as the API for
> > > >controlling Airavata workflows.
> > > >
> > > >1. It is a library, gives the ultimate control over the execution and
> it
> > > >can be simpler than any domain specific language that we can come with
> > > >like
> > > >XML, JSON etc
> > > >2. Most people use python and can learn it easily than any Domain
> > specific
> > > >language
> > > >3. You can easily create a Python library for Airavata because all the
> > > >APIs
> > > >are thrift based.
> > > >4. If you design the constructs correctly you can plug an XBaya.
> > > >
> > > >Any thoughts?
> > > >
> > > >Thanks,
> > > >Supun..
> > > >
> > > >
> > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> supun06@gmail.com
> > >
> > > >wrote:
> > > >
> > > >> Hi Shameera,
> > > >>
> > > >> Why you prefer JavaScript over a language like Python?
> > > >>
> > > >> Thanks,
> > > >> Supun..
> > > >>
> > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > > >> shameerainfo@gmail.com> wrote:
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> First of all thanks everyone for giving valuable inputs. After
> doing
> > > >>>some
> > > >>> background search and talking to different people in the University
> > > >>>who has
> > > >>> used different workflow languages, I myself convinced that
> > introducing
> > > >>>an
> > > >>> another workflow language is not what actually they need. By
> changing
> > > >>> exiting workflow language to another will not solve problems. What
> > they
> > > >>> asking is a easy way to construct the workflows. Indirectly what
> they
> > > >>> asking for a sort of API which they can use to generate the
> workflows
> > > >>>and
> > > >>> run it. Correct me if i am wrong here.
> > > >>>
> > > >>> As most of above replies depict, if we can get a simple API, as an
> > > >>> example, for a web based application, JavaScript API would be a
> good
> > > >>> solution, and probably JSON would be a good candidate for language,
> > > >>>instead
> > > >>> of XML.
> > > >>>
> > > >>> Airavata community already have started to implement web base GUI.
> > > >>>Hence
> > > >>> introducing a JSON base JavaScript API would be great help. WDYT?
> > > >>>
> > > >>> Thanks,
> > > >>> Shameera.
> > > >>>
> > > >>>
> > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> > > >>> alekny7@gmail.com> wrote:
> > > >>>
> > > >>>> Hi,
> > > >>>>
> > > >>>> it is not dataflow instead focused on orchestrating REST services
> > but
> > > >>>> you may find it useful datapoint - we created worfklow service
> that
> > > >>>>uses
> > > >>>> natively JavaScript and JSON to describe what happens during
> > workflow
> > > >>>> execution:
> > > >>>>
> > > https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> > > >>>>
> > > >>>> HTH,
> > > >>>>
> > > >>>> Alek
> > > >>>>
> > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> > > >>>>wrote:
> > > >>>>
> > > >>>>> Hi Chris,
> > > >>>>>
> > > >>>>> Great to hear OODT community will be interested in adopting a
> JSON
> > > >>>>> based workflow language and potentially a web based composer as
> > well.
> > > >>>>> Airavata previously had BPEL support initially through a home
> grown
> > > >>>>> implementation [1] by Alek Slominski and later through Apache ODE
> > > >>>>>[2]. Also
> > > >>>>> a white paper [3] by Alek on this topic is an interesting read.
> > > >>>>>
> > > >>>>> I am of the same opinion that we should adopt something more
> modern
> > > >>>>>as
> > > >>>>> the challenges from scientific workflows seems to be converging
> > with
> > > >>>>>the
> > > >>>>> data flow patterns in business workflows.
> > > >>>>>
> > > >>>>> It will be great if we can all compile a list of potential
> > candidates
> > > >>>>> and hack them through.
> > > >>>>>
> > > >>>>> Suresh
> > > >>>>> [1] -
> > > >>>>>
> > > >>>>>
> > > http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > > >>>>> [2] -
> > > >>>>>
> > > >>>>>
> > > http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> > > >>>>>time_for_eScience_workflows
> > > >>>>> [3] -
> > > >>>>>
> > > >>>>>
> > > http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> > > >>>>>6.pdf
> > > >>>>>
> > > >>>>>
> > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> > > >>>>>
> > > >>>>> > Hi Guys,
> > > >>>>> >
> > > >>>>> > I've been interested in this too - we don't per have a specific
> > > >>>>> > OODT workflow language, but we specific workflows using XML,
> and
> > > >>>>> > other configuration (we are also thinking of moving to JSON for
> > > >>>>> > this).
> > > >>>>> >
> > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
> complex
> > > >>>>> > to me.
> > > >>>>> >
> > > >>>>> > I wonder at the end of the day if we should adopt something
> more
> > > >>>>> > modern like PIG or some other data flow type of language (PIG
> > > >>>>> > is really neat).
> > > >>>>> >
> > > >>>>> > Cheers,
> > > >>>>> > Chris
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>>>> > Chris Mattmann, Ph.D.
> > > >>>>> > Chief Architect
> > > >>>>> > Instrument Software and Science Data Systems Section (398)
> > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > >>>>> > Office: 168-519, Mailstop: 168-527
> > > >>>>> > Email: chris.a.mattmann@nasa.gov
> > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
> > > >>>>> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>>>> > Adjunct Associate Professor, Computer Science Department
> > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> > > >>>>> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> > -----Original Message-----
> > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> > > >>>>> > Reply-To: "architecture@airavata.apache.org"
> > > >>>>> > <ar...@airavata.apache.org>
> > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> > > >>>>> > To: "architecture@airavata.apache.org" <
> > > >>>>> architecture@airavata.apache.org>,
> > > >>>>> > dev <de...@airavata.apache.org>
> > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> > > >>>>>Airavata.
> > > >>>>> >
> > > >>>>> >> Hi All,
> > > >>>>> >>
> > > >>>>> >> As we all know Airavata has its own workflow language call
> XWF.
> > > >>>>>When
> > > >>>>> XWF
> > > >>>>> >> was introduced, main focus points are interoperability and
> > > >>>>> convertibility.
> > > >>>>> >> But with years of experience it is convinced that above
> > > >>>>>requirements
> > > >>>>> are
> > > >>>>> >> not really useful when we come to real world use cases. And
> XWF
> > is
> > > >>>>> XML
> > > >>>>> >> based bulky language where we attache WSDLs and Workflow image
> > it
> > > >>>>> self.
> > > >>>>> >> But
> > > >>>>> >> with the recent changes WSDL part is being removed from XWF.
> > > >>>>> >>
> > > >>>>> >> It is worth to evaluate handy Scientific workflow languages in
> > > >>>>> industry
> > > >>>>> >> and
> > > >>>>> >> find out pros and cons, at the end of this evaluation we need
> to
> > > >>>>> come up
> > > >>>>> >> with idea how we should improve Airavata workflow language,
> > either
> > > >>>>> we can
> > > >>>>> >> improve existing XWF language, totally change to a new
> language
> > > >>>>> available
> > > >>>>> >> in industry or write a new light weight language. Basic
> > > >>>>>requirements
> > > >>>>> that
> > > >>>>> >> we expect from new improvement are, high usability, flexible,
> > > >>>>>light
> > > >>>>> weight
> > > >>>>> >> and real time monitoring support. As you can see above
> > > >>>>>requirements
> > > >>>>> are
> > > >>>>> >> not
> > > >>>>> >> direct comes with workflow languages but we need workflow
> > language
> > > >>>>> which
> > > >>>>> >> help to support above requirements.
> > > >>>>> >>
> > > >>>>> >> After reading few papers and googling, initially i have come
> up
> > > >>>>>with
> > > >>>>> >> following three existing languages,
> > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> > > >>>>> >> 2. WS-BPEL
> > > >>>>> >> 3. SIDL
> > > >>>>> >> <
> > http://computation.llnl.gov/casc/components/index.html#page=home
> > > >
> > > >>>>> >>
> > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> > > >>>>> Radical-SAGA
> > > >>>>> >> also uses slightly modified version of SIDL. Other than above
> > > >>>>>three
> > > >>>>> >> languages we can come up with simple workflow language base on
> > > >>>>> json(or
> > > >>>>> >> yaml) which support all our requirements for some extends.
> > > >>>>> >>
> > > >>>>> >> It would be grate if I can get more input regarding the
> $Subject
> > > >>>>> form the
> > > >>>>> >> airavata community. You all are more than welcome to provide
> any
> > > >>>>> type of
> > > >>>>> >> suggestions.
> > > >>>>> >>
> > > >>>>> >> Thanks,
> > > >>>>> >> Shameera.
> > > >>>>> >>
> > > >>>>> >>
> > > >>>>> >>
> > > >>>>> >> --
> > > >>>>> >> Best Regards,
> > > >>>>> >> Shameera Rathnayaka.
> > > >>>>> >>
> > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> The best way to predict the future is to invent it - Alan Kay
> > > >>>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Best Regards,
> > > >>> Shameera Rathnayaka.
> > > >>>
> > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > >>> Blog : http://shameerarathnayaka.blogspot.com/
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Supun Kamburugamuva
> > > >> Member, Apache Software Foundation; http://www.apache.org
> > > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > > >> Blog: http://supunk.blogspot.com
> > > >>
> > > >>
> > > >
> > > >
> > > >--
> > > >Supun Kamburugamuva
> > > >Member, Apache Software Foundation; http://www.apache.org
> > > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > > >Blog: http://supunk.blogspot.com
> > >
> > >
> > >
> >
> >
> > --
> > Best Regards,
> > Shameera Rathnayaka.
> >
> > email: shameera AT apache.org , shameerainfo AT gmail.com
> > Blog : http://shameerarathnayaka.blogspot.com/
> >
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
--
Best Regards,
Shameera Rathnayaka.
email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi Supun,
I think we were in two context, because I as suggesting a way to serialize
and deserialize the workflow description while you are suggesting to
implement
some kind of workflow interpreter using Python, where Python client can
send
thrift calls to Airavata server to run the application. I can see with your
suggested
approach we can control the workflow execution process from client side
which
make it easy to implement workflow debugger.As you mentioned this is a
major change
to Airavata. So we should neatly think as this will change our existing
architecture.
Still if someone need to use different language java, php, JS etc ... to
run the same
workflow which generated by Python, we need a language independent workflow
description.
My initial question was what is the best language for this?. But as I have
explained in
one of my previous reply, It is not matter what language we used Either we
can use
XML or JSON to write this description, what matters is how easy to generate
workflow with the provided API. Hence it would be great to have set of neat
APIs in
different languages.
Thanks,
Shameera.
On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:
> Hi Shameera,
>
> Using python is a radical approach for workflow I think. But I believe this
> is a very beautiful and new approach.Here is a possible scenario for
> implementation and running using a Python based library.
>
> The Python library facilitates the creation of Applications and submitting
> them to Airavata for execution. The underlying functionality is exactly
> similar to what Java clients provides.The only difference is that, Python
> library should have a more fluent API than Java for easy creation of
> workflows. We can generate the Python clients that talk to Airavata server
> using Thrift.
>
> Here is an example off the top of my head to a Python script created by
> user for a Workflow. This is a very crude example and we need to come up
> with a much better API if we are going to go along this path. First we need
> to write a Python script that can execute a workflow using Airavata.
>
> import airavata
>
> host = Host("localhost", ....)
> app1 = Application(host, ....)
> app2 = Application(host, ....)
>
> # we will connect these applications as a workflow using some topology
> builder or other constructs
>
> wb = WorkFlowBuilder()
> wb.setApp("name1", app1)
> # you can do a simple output transformation here etc
>
> # connects the input of app1 to app2 etc
> wb.setApp("name2", app2).connectInput("name1")
>
> wb.submit()
>
> Now we can load this Python script from XBaya. When XBaya loads this script
> the Python script can output an XML configuration of the topology, XBaya
> can render. There are other ways like directly executing the Python script
> from command line and connecting XBaya indirectly as well. Now you can run
> the workflow from XBaya. Running the Workflow means just executing the
> Python script.
>
> XBaya gets the notifications through messaging and update the UI
> accordingly.
>
> The users need to write the Python script by hand. XBaya cannot create the
> script. Because workflow language is an actual python program the benefits
> are immense. For example user can do workflow steering in the workflow
> itself by subscribing to messages from Airavata.
>
> Thanks,
> Supun..
>
>
>
>
>
>
>
>
> On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
> > Hi Supun,
> >
> > I meant to say JS is a well-known client side scripting language i have
> > messed scripting part. Even we use Python, ultimately we should convert
> > this to java model in server side, somehow we need to serialized python
> > representation to the language which java can parse and generate that
> > model. In this case we need to parse python script in java isn't it? I am
> > not exactly clear how you suggesting to use python here. More details on
> > how end system works if we use Python would be great help to clearly
> > understand your points.
> >
> > Thanks,
> > Shameera.
> >
> > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> chris.mattmann@gmail.com>
> > wrote:
> >
> > > Have you guys considered using JCC [1] as a means
> > > to expose the workflow API currently in Java as a
> > > Python API?
> > >
> > > We are exploring its use in OODT, and we have already
> > > created a Tika [2] JCC-based python API.
> > >
> > > Cheers,
> > > Chris
> > >
> > > [1] http://lucene.apache.org/pylucene/jcc/
> > > [2] http://github.com/chrismattmann/tika-python/
> > >
> > > ------------------------
> > > Chris Mattmann
> > > chris.mattmann@gmail.com
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Supun Kamburugamuva <su...@gmail.com>
> > > Reply-To: <de...@oodt.apache.org>
> > > Date: Thursday, October 16, 2014 at 3:43 PM
> > > To: dev <de...@airavata.apache.org>
> > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> > > <sm...@apache.org>, "architecture@airavata.apache.org"
> > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> > > <de...@oodt.apache.org>
> > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> Airavata.
> > >
> > > >Once we had an offline discussion about the Airavata Workflow language
> > > >(with Milinda, Saliya and Shameera). In that discussion one thing came
> > out
> > > >was why we need to invent a different language when a simple library
> > like
> > > >Python will full fill of Airavata requirements.
> > > >
> > > >There are many benefits in using a Python library as the API for
> > > >controlling Airavata workflows.
> > > >
> > > >1. It is a library, gives the ultimate control over the execution and
> it
> > > >can be simpler than any domain specific language that we can come with
> > > >like
> > > >XML, JSON etc
> > > >2. Most people use python and can learn it easily than any Domain
> > specific
> > > >language
> > > >3. You can easily create a Python library for Airavata because all the
> > > >APIs
> > > >are thrift based.
> > > >4. If you design the constructs correctly you can plug an XBaya.
> > > >
> > > >Any thoughts?
> > > >
> > > >Thanks,
> > > >Supun..
> > > >
> > > >
> > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> supun06@gmail.com
> > >
> > > >wrote:
> > > >
> > > >> Hi Shameera,
> > > >>
> > > >> Why you prefer JavaScript over a language like Python?
> > > >>
> > > >> Thanks,
> > > >> Supun..
> > > >>
> > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > > >> shameerainfo@gmail.com> wrote:
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> First of all thanks everyone for giving valuable inputs. After
> doing
> > > >>>some
> > > >>> background search and talking to different people in the University
> > > >>>who has
> > > >>> used different workflow languages, I myself convinced that
> > introducing
> > > >>>an
> > > >>> another workflow language is not what actually they need. By
> changing
> > > >>> exiting workflow language to another will not solve problems. What
> > they
> > > >>> asking is a easy way to construct the workflows. Indirectly what
> they
> > > >>> asking for a sort of API which they can use to generate the
> workflows
> > > >>>and
> > > >>> run it. Correct me if i am wrong here.
> > > >>>
> > > >>> As most of above replies depict, if we can get a simple API, as an
> > > >>> example, for a web based application, JavaScript API would be a
> good
> > > >>> solution, and probably JSON would be a good candidate for language,
> > > >>>instead
> > > >>> of XML.
> > > >>>
> > > >>> Airavata community already have started to implement web base GUI.
> > > >>>Hence
> > > >>> introducing a JSON base JavaScript API would be great help. WDYT?
> > > >>>
> > > >>> Thanks,
> > > >>> Shameera.
> > > >>>
> > > >>>
> > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> > > >>> alekny7@gmail.com> wrote:
> > > >>>
> > > >>>> Hi,
> > > >>>>
> > > >>>> it is not dataflow instead focused on orchestrating REST services
> > but
> > > >>>> you may find it useful datapoint - we created worfklow service
> that
> > > >>>>uses
> > > >>>> natively JavaScript and JSON to describe what happens during
> > workflow
> > > >>>> execution:
> > > >>>>
> > > https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> > > >>>>
> > > >>>> HTH,
> > > >>>>
> > > >>>> Alek
> > > >>>>
> > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> > > >>>>wrote:
> > > >>>>
> > > >>>>> Hi Chris,
> > > >>>>>
> > > >>>>> Great to hear OODT community will be interested in adopting a
> JSON
> > > >>>>> based workflow language and potentially a web based composer as
> > well.
> > > >>>>> Airavata previously had BPEL support initially through a home
> grown
> > > >>>>> implementation [1] by Alek Slominski and later through Apache ODE
> > > >>>>>[2]. Also
> > > >>>>> a white paper [3] by Alek on this topic is an interesting read.
> > > >>>>>
> > > >>>>> I am of the same opinion that we should adopt something more
> modern
> > > >>>>>as
> > > >>>>> the challenges from scientific workflows seems to be converging
> > with
> > > >>>>>the
> > > >>>>> data flow patterns in business workflows.
> > > >>>>>
> > > >>>>> It will be great if we can all compile a list of potential
> > candidates
> > > >>>>> and hack them through.
> > > >>>>>
> > > >>>>> Suresh
> > > >>>>> [1] -
> > > >>>>>
> > > >>>>>
> > > http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > > >>>>> [2] -
> > > >>>>>
> > > >>>>>
> > > http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> > > >>>>>time_for_eScience_workflows
> > > >>>>> [3] -
> > > >>>>>
> > > >>>>>
> > > http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> > > >>>>>6.pdf
> > > >>>>>
> > > >>>>>
> > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> > > >>>>>
> > > >>>>> > Hi Guys,
> > > >>>>> >
> > > >>>>> > I've been interested in this too - we don't per have a specific
> > > >>>>> > OODT workflow language, but we specific workflows using XML,
> and
> > > >>>>> > other configuration (we are also thinking of moving to JSON for
> > > >>>>> > this).
> > > >>>>> >
> > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
> complex
> > > >>>>> > to me.
> > > >>>>> >
> > > >>>>> > I wonder at the end of the day if we should adopt something
> more
> > > >>>>> > modern like PIG or some other data flow type of language (PIG
> > > >>>>> > is really neat).
> > > >>>>> >
> > > >>>>> > Cheers,
> > > >>>>> > Chris
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>>>> > Chris Mattmann, Ph.D.
> > > >>>>> > Chief Architect
> > > >>>>> > Instrument Software and Science Data Systems Section (398)
> > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > >>>>> > Office: 168-519, Mailstop: 168-527
> > > >>>>> > Email: chris.a.mattmann@nasa.gov
> > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
> > > >>>>> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>>>> > Adjunct Associate Professor, Computer Science Department
> > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> > > >>>>> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> > -----Original Message-----
> > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> > > >>>>> > Reply-To: "architecture@airavata.apache.org"
> > > >>>>> > <ar...@airavata.apache.org>
> > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> > > >>>>> > To: "architecture@airavata.apache.org" <
> > > >>>>> architecture@airavata.apache.org>,
> > > >>>>> > dev <de...@airavata.apache.org>
> > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> > > >>>>>Airavata.
> > > >>>>> >
> > > >>>>> >> Hi All,
> > > >>>>> >>
> > > >>>>> >> As we all know Airavata has its own workflow language call
> XWF.
> > > >>>>>When
> > > >>>>> XWF
> > > >>>>> >> was introduced, main focus points are interoperability and
> > > >>>>> convertibility.
> > > >>>>> >> But with years of experience it is convinced that above
> > > >>>>>requirements
> > > >>>>> are
> > > >>>>> >> not really useful when we come to real world use cases. And
> XWF
> > is
> > > >>>>> XML
> > > >>>>> >> based bulky language where we attache WSDLs and Workflow image
> > it
> > > >>>>> self.
> > > >>>>> >> But
> > > >>>>> >> with the recent changes WSDL part is being removed from XWF.
> > > >>>>> >>
> > > >>>>> >> It is worth to evaluate handy Scientific workflow languages in
> > > >>>>> industry
> > > >>>>> >> and
> > > >>>>> >> find out pros and cons, at the end of this evaluation we need
> to
> > > >>>>> come up
> > > >>>>> >> with idea how we should improve Airavata workflow language,
> > either
> > > >>>>> we can
> > > >>>>> >> improve existing XWF language, totally change to a new
> language
> > > >>>>> available
> > > >>>>> >> in industry or write a new light weight language. Basic
> > > >>>>>requirements
> > > >>>>> that
> > > >>>>> >> we expect from new improvement are, high usability, flexible,
> > > >>>>>light
> > > >>>>> weight
> > > >>>>> >> and real time monitoring support. As you can see above
> > > >>>>>requirements
> > > >>>>> are
> > > >>>>> >> not
> > > >>>>> >> direct comes with workflow languages but we need workflow
> > language
> > > >>>>> which
> > > >>>>> >> help to support above requirements.
> > > >>>>> >>
> > > >>>>> >> After reading few papers and googling, initially i have come
> up
> > > >>>>>with
> > > >>>>> >> following three existing languages,
> > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> > > >>>>> >> 2. WS-BPEL
> > > >>>>> >> 3. SIDL
> > > >>>>> >> <
> > http://computation.llnl.gov/casc/components/index.html#page=home
> > > >
> > > >>>>> >>
> > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> > > >>>>> Radical-SAGA
> > > >>>>> >> also uses slightly modified version of SIDL. Other than above
> > > >>>>>three
> > > >>>>> >> languages we can come up with simple workflow language base on
> > > >>>>> json(or
> > > >>>>> >> yaml) which support all our requirements for some extends.
> > > >>>>> >>
> > > >>>>> >> It would be grate if I can get more input regarding the
> $Subject
> > > >>>>> form the
> > > >>>>> >> airavata community. You all are more than welcome to provide
> any
> > > >>>>> type of
> > > >>>>> >> suggestions.
> > > >>>>> >>
> > > >>>>> >> Thanks,
> > > >>>>> >> Shameera.
> > > >>>>> >>
> > > >>>>> >>
> > > >>>>> >>
> > > >>>>> >> --
> > > >>>>> >> Best Regards,
> > > >>>>> >> Shameera Rathnayaka.
> > > >>>>> >>
> > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> The best way to predict the future is to invent it - Alan Kay
> > > >>>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Best Regards,
> > > >>> Shameera Rathnayaka.
> > > >>>
> > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > >>> Blog : http://shameerarathnayaka.blogspot.com/
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Supun Kamburugamuva
> > > >> Member, Apache Software Foundation; http://www.apache.org
> > > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > > >> Blog: http://supunk.blogspot.com
> > > >>
> > > >>
> > > >
> > > >
> > > >--
> > > >Supun Kamburugamuva
> > > >Member, Apache Software Foundation; http://www.apache.org
> > > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > > >Blog: http://supunk.blogspot.com
> > >
> > >
> > >
> >
> >
> > --
> > Best Regards,
> > Shameera Rathnayaka.
> >
> > email: shameera AT apache.org , shameerainfo AT gmail.com
> > Blog : http://shameerarathnayaka.blogspot.com/
> >
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
--
Best Regards,
Shameera Rathnayaka.
email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi Supun,
I think we were in two context, because I as suggesting a way to serialize
and deserialize the workflow description while you are suggesting to
implement
some kind of workflow interpreter using Python, where Python client can
send
thrift calls to Airavata server to run the application. I can see with your
suggested
approach we can control the workflow execution process from client side
which
make it easy to implement workflow debugger.As you mentioned this is a
major change
to Airavata. So we should neatly think as this will change our existing
architecture.
Still if someone need to use different language java, php, JS etc ... to
run the same
workflow which generated by Python, we need a language independent workflow
description.
My initial question was what is the best language for this?. But as I have
explained in
one of my previous reply, It is not matter what language we used Either we
can use
XML or JSON to write this description, what matters is how easy to generate
workflow with the provided API. Hence it would be great to have set of neat
APIs in
different languages.
Thanks,
Shameera.
On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:
> Hi Shameera,
>
> Using python is a radical approach for workflow I think. But I believe this
> is a very beautiful and new approach.Here is a possible scenario for
> implementation and running using a Python based library.
>
> The Python library facilitates the creation of Applications and submitting
> them to Airavata for execution. The underlying functionality is exactly
> similar to what Java clients provides.The only difference is that, Python
> library should have a more fluent API than Java for easy creation of
> workflows. We can generate the Python clients that talk to Airavata server
> using Thrift.
>
> Here is an example off the top of my head to a Python script created by
> user for a Workflow. This is a very crude example and we need to come up
> with a much better API if we are going to go along this path. First we need
> to write a Python script that can execute a workflow using Airavata.
>
> import airavata
>
> host = Host("localhost", ....)
> app1 = Application(host, ....)
> app2 = Application(host, ....)
>
> # we will connect these applications as a workflow using some topology
> builder or other constructs
>
> wb = WorkFlowBuilder()
> wb.setApp("name1", app1)
> # you can do a simple output transformation here etc
>
> # connects the input of app1 to app2 etc
> wb.setApp("name2", app2).connectInput("name1")
>
> wb.submit()
>
> Now we can load this Python script from XBaya. When XBaya loads this script
> the Python script can output an XML configuration of the topology, XBaya
> can render. There are other ways like directly executing the Python script
> from command line and connecting XBaya indirectly as well. Now you can run
> the workflow from XBaya. Running the Workflow means just executing the
> Python script.
>
> XBaya gets the notifications through messaging and update the UI
> accordingly.
>
> The users need to write the Python script by hand. XBaya cannot create the
> script. Because workflow language is an actual python program the benefits
> are immense. For example user can do workflow steering in the workflow
> itself by subscribing to messages from Airavata.
>
> Thanks,
> Supun..
>
>
>
>
>
>
>
>
> On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
> > Hi Supun,
> >
> > I meant to say JS is a well-known client side scripting language i have
> > messed scripting part. Even we use Python, ultimately we should convert
> > this to java model in server side, somehow we need to serialized python
> > representation to the language which java can parse and generate that
> > model. In this case we need to parse python script in java isn't it? I am
> > not exactly clear how you suggesting to use python here. More details on
> > how end system works if we use Python would be great help to clearly
> > understand your points.
> >
> > Thanks,
> > Shameera.
> >
> > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> chris.mattmann@gmail.com>
> > wrote:
> >
> > > Have you guys considered using JCC [1] as a means
> > > to expose the workflow API currently in Java as a
> > > Python API?
> > >
> > > We are exploring its use in OODT, and we have already
> > > created a Tika [2] JCC-based python API.
> > >
> > > Cheers,
> > > Chris
> > >
> > > [1] http://lucene.apache.org/pylucene/jcc/
> > > [2] http://github.com/chrismattmann/tika-python/
> > >
> > > ------------------------
> > > Chris Mattmann
> > > chris.mattmann@gmail.com
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Supun Kamburugamuva <su...@gmail.com>
> > > Reply-To: <de...@oodt.apache.org>
> > > Date: Thursday, October 16, 2014 at 3:43 PM
> > > To: dev <de...@airavata.apache.org>
> > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> > > <sm...@apache.org>, "architecture@airavata.apache.org"
> > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> > > <de...@oodt.apache.org>
> > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> Airavata.
> > >
> > > >Once we had an offline discussion about the Airavata Workflow language
> > > >(with Milinda, Saliya and Shameera). In that discussion one thing came
> > out
> > > >was why we need to invent a different language when a simple library
> > like
> > > >Python will full fill of Airavata requirements.
> > > >
> > > >There are many benefits in using a Python library as the API for
> > > >controlling Airavata workflows.
> > > >
> > > >1. It is a library, gives the ultimate control over the execution and
> it
> > > >can be simpler than any domain specific language that we can come with
> > > >like
> > > >XML, JSON etc
> > > >2. Most people use python and can learn it easily than any Domain
> > specific
> > > >language
> > > >3. You can easily create a Python library for Airavata because all the
> > > >APIs
> > > >are thrift based.
> > > >4. If you design the constructs correctly you can plug an XBaya.
> > > >
> > > >Any thoughts?
> > > >
> > > >Thanks,
> > > >Supun..
> > > >
> > > >
> > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> supun06@gmail.com
> > >
> > > >wrote:
> > > >
> > > >> Hi Shameera,
> > > >>
> > > >> Why you prefer JavaScript over a language like Python?
> > > >>
> > > >> Thanks,
> > > >> Supun..
> > > >>
> > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > > >> shameerainfo@gmail.com> wrote:
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> First of all thanks everyone for giving valuable inputs. After
> doing
> > > >>>some
> > > >>> background search and talking to different people in the University
> > > >>>who has
> > > >>> used different workflow languages, I myself convinced that
> > introducing
> > > >>>an
> > > >>> another workflow language is not what actually they need. By
> changing
> > > >>> exiting workflow language to another will not solve problems. What
> > they
> > > >>> asking is a easy way to construct the workflows. Indirectly what
> they
> > > >>> asking for a sort of API which they can use to generate the
> workflows
> > > >>>and
> > > >>> run it. Correct me if i am wrong here.
> > > >>>
> > > >>> As most of above replies depict, if we can get a simple API, as an
> > > >>> example, for a web based application, JavaScript API would be a
> good
> > > >>> solution, and probably JSON would be a good candidate for language,
> > > >>>instead
> > > >>> of XML.
> > > >>>
> > > >>> Airavata community already have started to implement web base GUI.
> > > >>>Hence
> > > >>> introducing a JSON base JavaScript API would be great help. WDYT?
> > > >>>
> > > >>> Thanks,
> > > >>> Shameera.
> > > >>>
> > > >>>
> > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> > > >>> alekny7@gmail.com> wrote:
> > > >>>
> > > >>>> Hi,
> > > >>>>
> > > >>>> it is not dataflow instead focused on orchestrating REST services
> > but
> > > >>>> you may find it useful datapoint - we created worfklow service
> that
> > > >>>>uses
> > > >>>> natively JavaScript and JSON to describe what happens during
> > workflow
> > > >>>> execution:
> > > >>>>
> > > https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> > > >>>>
> > > >>>> HTH,
> > > >>>>
> > > >>>> Alek
> > > >>>>
> > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> > > >>>>wrote:
> > > >>>>
> > > >>>>> Hi Chris,
> > > >>>>>
> > > >>>>> Great to hear OODT community will be interested in adopting a
> JSON
> > > >>>>> based workflow language and potentially a web based composer as
> > well.
> > > >>>>> Airavata previously had BPEL support initially through a home
> grown
> > > >>>>> implementation [1] by Alek Slominski and later through Apache ODE
> > > >>>>>[2]. Also
> > > >>>>> a white paper [3] by Alek on this topic is an interesting read.
> > > >>>>>
> > > >>>>> I am of the same opinion that we should adopt something more
> modern
> > > >>>>>as
> > > >>>>> the challenges from scientific workflows seems to be converging
> > with
> > > >>>>>the
> > > >>>>> data flow patterns in business workflows.
> > > >>>>>
> > > >>>>> It will be great if we can all compile a list of potential
> > candidates
> > > >>>>> and hack them through.
> > > >>>>>
> > > >>>>> Suresh
> > > >>>>> [1] -
> > > >>>>>
> > > >>>>>
> > > http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > > >>>>> [2] -
> > > >>>>>
> > > >>>>>
> > > http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> > > >>>>>time_for_eScience_workflows
> > > >>>>> [3] -
> > > >>>>>
> > > >>>>>
> > > http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> > > >>>>>6.pdf
> > > >>>>>
> > > >>>>>
> > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> > > >>>>>
> > > >>>>> > Hi Guys,
> > > >>>>> >
> > > >>>>> > I've been interested in this too - we don't per have a specific
> > > >>>>> > OODT workflow language, but we specific workflows using XML,
> and
> > > >>>>> > other configuration (we are also thinking of moving to JSON for
> > > >>>>> > this).
> > > >>>>> >
> > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
> complex
> > > >>>>> > to me.
> > > >>>>> >
> > > >>>>> > I wonder at the end of the day if we should adopt something
> more
> > > >>>>> > modern like PIG or some other data flow type of language (PIG
> > > >>>>> > is really neat).
> > > >>>>> >
> > > >>>>> > Cheers,
> > > >>>>> > Chris
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>>>> > Chris Mattmann, Ph.D.
> > > >>>>> > Chief Architect
> > > >>>>> > Instrument Software and Science Data Systems Section (398)
> > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > >>>>> > Office: 168-519, Mailstop: 168-527
> > > >>>>> > Email: chris.a.mattmann@nasa.gov
> > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
> > > >>>>> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>>>> > Adjunct Associate Professor, Computer Science Department
> > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> > > >>>>> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> > -----Original Message-----
> > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> > > >>>>> > Reply-To: "architecture@airavata.apache.org"
> > > >>>>> > <ar...@airavata.apache.org>
> > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> > > >>>>> > To: "architecture@airavata.apache.org" <
> > > >>>>> architecture@airavata.apache.org>,
> > > >>>>> > dev <de...@airavata.apache.org>
> > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> > > >>>>>Airavata.
> > > >>>>> >
> > > >>>>> >> Hi All,
> > > >>>>> >>
> > > >>>>> >> As we all know Airavata has its own workflow language call
> XWF.
> > > >>>>>When
> > > >>>>> XWF
> > > >>>>> >> was introduced, main focus points are interoperability and
> > > >>>>> convertibility.
> > > >>>>> >> But with years of experience it is convinced that above
> > > >>>>>requirements
> > > >>>>> are
> > > >>>>> >> not really useful when we come to real world use cases. And
> XWF
> > is
> > > >>>>> XML
> > > >>>>> >> based bulky language where we attache WSDLs and Workflow image
> > it
> > > >>>>> self.
> > > >>>>> >> But
> > > >>>>> >> with the recent changes WSDL part is being removed from XWF.
> > > >>>>> >>
> > > >>>>> >> It is worth to evaluate handy Scientific workflow languages in
> > > >>>>> industry
> > > >>>>> >> and
> > > >>>>> >> find out pros and cons, at the end of this evaluation we need
> to
> > > >>>>> come up
> > > >>>>> >> with idea how we should improve Airavata workflow language,
> > either
> > > >>>>> we can
> > > >>>>> >> improve existing XWF language, totally change to a new
> language
> > > >>>>> available
> > > >>>>> >> in industry or write a new light weight language. Basic
> > > >>>>>requirements
> > > >>>>> that
> > > >>>>> >> we expect from new improvement are, high usability, flexible,
> > > >>>>>light
> > > >>>>> weight
> > > >>>>> >> and real time monitoring support. As you can see above
> > > >>>>>requirements
> > > >>>>> are
> > > >>>>> >> not
> > > >>>>> >> direct comes with workflow languages but we need workflow
> > language
> > > >>>>> which
> > > >>>>> >> help to support above requirements.
> > > >>>>> >>
> > > >>>>> >> After reading few papers and googling, initially i have come
> up
> > > >>>>>with
> > > >>>>> >> following three existing languages,
> > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> > > >>>>> >> 2. WS-BPEL
> > > >>>>> >> 3. SIDL
> > > >>>>> >> <
> > http://computation.llnl.gov/casc/components/index.html#page=home
> > > >
> > > >>>>> >>
> > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> > > >>>>> Radical-SAGA
> > > >>>>> >> also uses slightly modified version of SIDL. Other than above
> > > >>>>>three
> > > >>>>> >> languages we can come up with simple workflow language base on
> > > >>>>> json(or
> > > >>>>> >> yaml) which support all our requirements for some extends.
> > > >>>>> >>
> > > >>>>> >> It would be grate if I can get more input regarding the
> $Subject
> > > >>>>> form the
> > > >>>>> >> airavata community. You all are more than welcome to provide
> any
> > > >>>>> type of
> > > >>>>> >> suggestions.
> > > >>>>> >>
> > > >>>>> >> Thanks,
> > > >>>>> >> Shameera.
> > > >>>>> >>
> > > >>>>> >>
> > > >>>>> >>
> > > >>>>> >> --
> > > >>>>> >> Best Regards,
> > > >>>>> >> Shameera Rathnayaka.
> > > >>>>> >>
> > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> The best way to predict the future is to invent it - Alan Kay
> > > >>>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Best Regards,
> > > >>> Shameera Rathnayaka.
> > > >>>
> > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > >>> Blog : http://shameerarathnayaka.blogspot.com/
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Supun Kamburugamuva
> > > >> Member, Apache Software Foundation; http://www.apache.org
> > > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > > >> Blog: http://supunk.blogspot.com
> > > >>
> > > >>
> > > >
> > > >
> > > >--
> > > >Supun Kamburugamuva
> > > >Member, Apache Software Foundation; http://www.apache.org
> > > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > > >Blog: http://supunk.blogspot.com
> > >
> > >
> > >
> >
> >
> > --
> > Best Regards,
> > Shameera Rathnayaka.
> >
> > email: shameera AT apache.org , shameerainfo AT gmail.com
> > Blog : http://shameerarathnayaka.blogspot.com/
> >
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
--
Best Regards,
Shameera Rathnayaka.
email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Supun Kamburugamuva <su...@gmail.com>.
Hi Shameera,
Using python is a radical approach for workflow I think. But I believe this
is a very beautiful and new approach.Here is a possible scenario for
implementation and running using a Python based library.
The Python library facilitates the creation of Applications and submitting
them to Airavata for execution. The underlying functionality is exactly
similar to what Java clients provides.The only difference is that, Python
library should have a more fluent API than Java for easy creation of
workflows. We can generate the Python clients that talk to Airavata server
using Thrift.
Here is an example off the top of my head to a Python script created by
user for a Workflow. This is a very crude example and we need to come up
with a much better API if we are going to go along this path. First we need
to write a Python script that can execute a workflow using Airavata.
import airavata
host = Host("localhost", ....)
app1 = Application(host, ....)
app2 = Application(host, ....)
# we will connect these applications as a workflow using some topology
builder or other constructs
wb = WorkFlowBuilder()
wb.setApp("name1", app1)
# you can do a simple output transformation here etc
# connects the input of app1 to app2 etc
wb.setApp("name2", app2).connectInput("name1")
wb.submit()
Now we can load this Python script from XBaya. When XBaya loads this script
the Python script can output an XML configuration of the topology, XBaya
can render. There are other ways like directly executing the Python script
from command line and connecting XBaya indirectly as well. Now you can run
the workflow from XBaya. Running the Workflow means just executing the
Python script.
XBaya gets the notifications through messaging and update the UI
accordingly.
The users need to write the Python script by hand. XBaya cannot create the
script. Because workflow language is an actual python program the benefits
are immense. For example user can do workflow steering in the workflow
itself by subscribing to messages from Airavata.
Thanks,
Supun..
On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:
> Hi Supun,
>
> I meant to say JS is a well-known client side scripting language i have
> messed scripting part. Even we use Python, ultimately we should convert
> this to java model in server side, somehow we need to serialized python
> representation to the language which java can parse and generate that
> model. In this case we need to parse python script in java isn't it? I am
> not exactly clear how you suggesting to use python here. More details on
> how end system works if we use Python would be great help to clearly
> understand your points.
>
> Thanks,
> Shameera.
>
> On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <ch...@gmail.com>
> wrote:
>
> > Have you guys considered using JCC [1] as a means
> > to expose the workflow API currently in Java as a
> > Python API?
> >
> > We are exploring its use in OODT, and we have already
> > created a Tika [2] JCC-based python API.
> >
> > Cheers,
> > Chris
> >
> > [1] http://lucene.apache.org/pylucene/jcc/
> > [2] http://github.com/chrismattmann/tika-python/
> >
> > ------------------------
> > Chris Mattmann
> > chris.mattmann@gmail.com
> >
> >
> >
> >
> > -----Original Message-----
> > From: Supun Kamburugamuva <su...@gmail.com>
> > Reply-To: <de...@oodt.apache.org>
> > Date: Thursday, October 16, 2014 at 3:43 PM
> > To: dev <de...@airavata.apache.org>
> > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> > <sm...@apache.org>, "architecture@airavata.apache.org"
> > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> > <de...@oodt.apache.org>
> > Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
> >
> > >Once we had an offline discussion about the Airavata Workflow language
> > >(with Milinda, Saliya and Shameera). In that discussion one thing came
> out
> > >was why we need to invent a different language when a simple library
> like
> > >Python will full fill of Airavata requirements.
> > >
> > >There are many benefits in using a Python library as the API for
> > >controlling Airavata workflows.
> > >
> > >1. It is a library, gives the ultimate control over the execution and it
> > >can be simpler than any domain specific language that we can come with
> > >like
> > >XML, JSON etc
> > >2. Most people use python and can learn it easily than any Domain
> specific
> > >language
> > >3. You can easily create a Python library for Airavata because all the
> > >APIs
> > >are thrift based.
> > >4. If you design the constructs correctly you can plug an XBaya.
> > >
> > >Any thoughts?
> > >
> > >Thanks,
> > >Supun..
> > >
> > >
> > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <supun06@gmail.com
> >
> > >wrote:
> > >
> > >> Hi Shameera,
> > >>
> > >> Why you prefer JavaScript over a language like Python?
> > >>
> > >> Thanks,
> > >> Supun..
> > >>
> > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > >> shameerainfo@gmail.com> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> First of all thanks everyone for giving valuable inputs. After doing
> > >>>some
> > >>> background search and talking to different people in the University
> > >>>who has
> > >>> used different workflow languages, I myself convinced that
> introducing
> > >>>an
> > >>> another workflow language is not what actually they need. By changing
> > >>> exiting workflow language to another will not solve problems. What
> they
> > >>> asking is a easy way to construct the workflows. Indirectly what they
> > >>> asking for a sort of API which they can use to generate the workflows
> > >>>and
> > >>> run it. Correct me if i am wrong here.
> > >>>
> > >>> As most of above replies depict, if we can get a simple API, as an
> > >>> example, for a web based application, JavaScript API would be a good
> > >>> solution, and probably JSON would be a good candidate for language,
> > >>>instead
> > >>> of XML.
> > >>>
> > >>> Airavata community already have started to implement web base GUI.
> > >>>Hence
> > >>> introducing a JSON base JavaScript API would be great help. WDYT?
> > >>>
> > >>> Thanks,
> > >>> Shameera.
> > >>>
> > >>>
> > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> > >>> alekny7@gmail.com> wrote:
> > >>>
> > >>>> Hi,
> > >>>>
> > >>>> it is not dataflow instead focused on orchestrating REST services
> but
> > >>>> you may find it useful datapoint - we created worfklow service that
> > >>>>uses
> > >>>> natively JavaScript and JSON to describe what happens during
> workflow
> > >>>> execution:
> > >>>>
> > https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> > >>>>
> > >>>> HTH,
> > >>>>
> > >>>> Alek
> > >>>>
> > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> > >>>>wrote:
> > >>>>
> > >>>>> Hi Chris,
> > >>>>>
> > >>>>> Great to hear OODT community will be interested in adopting a JSON
> > >>>>> based workflow language and potentially a web based composer as
> well.
> > >>>>> Airavata previously had BPEL support initially through a home grown
> > >>>>> implementation [1] by Alek Slominski and later through Apache ODE
> > >>>>>[2]. Also
> > >>>>> a white paper [3] by Alek on this topic is an interesting read.
> > >>>>>
> > >>>>> I am of the same opinion that we should adopt something more modern
> > >>>>>as
> > >>>>> the challenges from scientific workflows seems to be converging
> with
> > >>>>>the
> > >>>>> data flow patterns in business workflows.
> > >>>>>
> > >>>>> It will be great if we can all compile a list of potential
> candidates
> > >>>>> and hack them through.
> > >>>>>
> > >>>>> Suresh
> > >>>>> [1] -
> > >>>>>
> > >>>>>
> > http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > >>>>> [2] -
> > >>>>>
> > >>>>>
> > http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> > >>>>>time_for_eScience_workflows
> > >>>>> [3] -
> > >>>>>
> > >>>>>
> > http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> > >>>>>6.pdf
> > >>>>>
> > >>>>>
> > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> > >>>>>
> > >>>>> > Hi Guys,
> > >>>>> >
> > >>>>> > I've been interested in this too - we don't per have a specific
> > >>>>> > OODT workflow language, but we specific workflows using XML, and
> > >>>>> > other configuration (we are also thinking of moving to JSON for
> > >>>>> > this).
> > >>>>> >
> > >>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> > >>>>> > to me.
> > >>>>> >
> > >>>>> > I wonder at the end of the day if we should adopt something more
> > >>>>> > modern like PIG or some other data flow type of language (PIG
> > >>>>> > is really neat).
> > >>>>> >
> > >>>>> > Cheers,
> > >>>>> > Chris
> > >>>>> >
> > >>>>> >
> > >>>>> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>>>> > Chris Mattmann, Ph.D.
> > >>>>> > Chief Architect
> > >>>>> > Instrument Software and Science Data Systems Section (398)
> > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > >>>>> > Office: 168-519, Mailstop: 168-527
> > >>>>> > Email: chris.a.mattmann@nasa.gov
> > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
> > >>>>> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>>>> > Adjunct Associate Professor, Computer Science Department
> > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> > >>>>> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> > -----Original Message-----
> > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> > >>>>> > Reply-To: "architecture@airavata.apache.org"
> > >>>>> > <ar...@airavata.apache.org>
> > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> > >>>>> > To: "architecture@airavata.apache.org" <
> > >>>>> architecture@airavata.apache.org>,
> > >>>>> > dev <de...@airavata.apache.org>
> > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> > >>>>>Airavata.
> > >>>>> >
> > >>>>> >> Hi All,
> > >>>>> >>
> > >>>>> >> As we all know Airavata has its own workflow language call XWF.
> > >>>>>When
> > >>>>> XWF
> > >>>>> >> was introduced, main focus points are interoperability and
> > >>>>> convertibility.
> > >>>>> >> But with years of experience it is convinced that above
> > >>>>>requirements
> > >>>>> are
> > >>>>> >> not really useful when we come to real world use cases. And XWF
> is
> > >>>>> XML
> > >>>>> >> based bulky language where we attache WSDLs and Workflow image
> it
> > >>>>> self.
> > >>>>> >> But
> > >>>>> >> with the recent changes WSDL part is being removed from XWF.
> > >>>>> >>
> > >>>>> >> It is worth to evaluate handy Scientific workflow languages in
> > >>>>> industry
> > >>>>> >> and
> > >>>>> >> find out pros and cons, at the end of this evaluation we need to
> > >>>>> come up
> > >>>>> >> with idea how we should improve Airavata workflow language,
> either
> > >>>>> we can
> > >>>>> >> improve existing XWF language, totally change to a new language
> > >>>>> available
> > >>>>> >> in industry or write a new light weight language. Basic
> > >>>>>requirements
> > >>>>> that
> > >>>>> >> we expect from new improvement are, high usability, flexible,
> > >>>>>light
> > >>>>> weight
> > >>>>> >> and real time monitoring support. As you can see above
> > >>>>>requirements
> > >>>>> are
> > >>>>> >> not
> > >>>>> >> direct comes with workflow languages but we need workflow
> language
> > >>>>> which
> > >>>>> >> help to support above requirements.
> > >>>>> >>
> > >>>>> >> After reading few papers and googling, initially i have come up
> > >>>>>with
> > >>>>> >> following three existing languages,
> > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> > >>>>> >> 2. WS-BPEL
> > >>>>> >> 3. SIDL
> > >>>>> >> <
> http://computation.llnl.gov/casc/components/index.html#page=home
> > >
> > >>>>> >>
> > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> > >>>>> Radical-SAGA
> > >>>>> >> also uses slightly modified version of SIDL. Other than above
> > >>>>>three
> > >>>>> >> languages we can come up with simple workflow language base on
> > >>>>> json(or
> > >>>>> >> yaml) which support all our requirements for some extends.
> > >>>>> >>
> > >>>>> >> It would be grate if I can get more input regarding the $Subject
> > >>>>> form the
> > >>>>> >> airavata community. You all are more than welcome to provide any
> > >>>>> type of
> > >>>>> >> suggestions.
> > >>>>> >>
> > >>>>> >> Thanks,
> > >>>>> >> Shameera.
> > >>>>> >>
> > >>>>> >>
> > >>>>> >>
> > >>>>> >> --
> > >>>>> >> Best Regards,
> > >>>>> >> Shameera Rathnayaka.
> > >>>>> >>
> > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> > >>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> The best way to predict the future is to invent it - Alan Kay
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Best Regards,
> > >>> Shameera Rathnayaka.
> > >>>
> > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > >>> Blog : http://shameerarathnayaka.blogspot.com/
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Supun Kamburugamuva
> > >> Member, Apache Software Foundation; http://www.apache.org
> > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > >> Blog: http://supunk.blogspot.com
> > >>
> > >>
> > >
> > >
> > >--
> > >Supun Kamburugamuva
> > >Member, Apache Software Foundation; http://www.apache.org
> > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > >Blog: http://supunk.blogspot.com
> >
> >
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Supun Kamburugamuva <su...@gmail.com>.
Hi Shameera,
Using python is a radical approach for workflow I think. But I believe this
is a very beautiful and new approach.Here is a possible scenario for
implementation and running using a Python based library.
The Python library facilitates the creation of Applications and submitting
them to Airavata for execution. The underlying functionality is exactly
similar to what Java clients provides.The only difference is that, Python
library should have a more fluent API than Java for easy creation of
workflows. We can generate the Python clients that talk to Airavata server
using Thrift.
Here is an example off the top of my head to a Python script created by
user for a Workflow. This is a very crude example and we need to come up
with a much better API if we are going to go along this path. First we need
to write a Python script that can execute a workflow using Airavata.
import airavata
host = Host("localhost", ....)
app1 = Application(host, ....)
app2 = Application(host, ....)
# we will connect these applications as a workflow using some topology
builder or other constructs
wb = WorkFlowBuilder()
wb.setApp("name1", app1)
# you can do a simple output transformation here etc
# connects the input of app1 to app2 etc
wb.setApp("name2", app2).connectInput("name1")
wb.submit()
Now we can load this Python script from XBaya. When XBaya loads this script
the Python script can output an XML configuration of the topology, XBaya
can render. There are other ways like directly executing the Python script
from command line and connecting XBaya indirectly as well. Now you can run
the workflow from XBaya. Running the Workflow means just executing the
Python script.
XBaya gets the notifications through messaging and update the UI
accordingly.
The users need to write the Python script by hand. XBaya cannot create the
script. Because workflow language is an actual python program the benefits
are immense. For example user can do workflow steering in the workflow
itself by subscribing to messages from Airavata.
Thanks,
Supun..
On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:
> Hi Supun,
>
> I meant to say JS is a well-known client side scripting language i have
> messed scripting part. Even we use Python, ultimately we should convert
> this to java model in server side, somehow we need to serialized python
> representation to the language which java can parse and generate that
> model. In this case we need to parse python script in java isn't it? I am
> not exactly clear how you suggesting to use python here. More details on
> how end system works if we use Python would be great help to clearly
> understand your points.
>
> Thanks,
> Shameera.
>
> On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <ch...@gmail.com>
> wrote:
>
> > Have you guys considered using JCC [1] as a means
> > to expose the workflow API currently in Java as a
> > Python API?
> >
> > We are exploring its use in OODT, and we have already
> > created a Tika [2] JCC-based python API.
> >
> > Cheers,
> > Chris
> >
> > [1] http://lucene.apache.org/pylucene/jcc/
> > [2] http://github.com/chrismattmann/tika-python/
> >
> > ------------------------
> > Chris Mattmann
> > chris.mattmann@gmail.com
> >
> >
> >
> >
> > -----Original Message-----
> > From: Supun Kamburugamuva <su...@gmail.com>
> > Reply-To: <de...@oodt.apache.org>
> > Date: Thursday, October 16, 2014 at 3:43 PM
> > To: dev <de...@airavata.apache.org>
> > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> > <sm...@apache.org>, "architecture@airavata.apache.org"
> > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> > <de...@oodt.apache.org>
> > Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
> >
> > >Once we had an offline discussion about the Airavata Workflow language
> > >(with Milinda, Saliya and Shameera). In that discussion one thing came
> out
> > >was why we need to invent a different language when a simple library
> like
> > >Python will full fill of Airavata requirements.
> > >
> > >There are many benefits in using a Python library as the API for
> > >controlling Airavata workflows.
> > >
> > >1. It is a library, gives the ultimate control over the execution and it
> > >can be simpler than any domain specific language that we can come with
> > >like
> > >XML, JSON etc
> > >2. Most people use python and can learn it easily than any Domain
> specific
> > >language
> > >3. You can easily create a Python library for Airavata because all the
> > >APIs
> > >are thrift based.
> > >4. If you design the constructs correctly you can plug an XBaya.
> > >
> > >Any thoughts?
> > >
> > >Thanks,
> > >Supun..
> > >
> > >
> > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <supun06@gmail.com
> >
> > >wrote:
> > >
> > >> Hi Shameera,
> > >>
> > >> Why you prefer JavaScript over a language like Python?
> > >>
> > >> Thanks,
> > >> Supun..
> > >>
> > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > >> shameerainfo@gmail.com> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> First of all thanks everyone for giving valuable inputs. After doing
> > >>>some
> > >>> background search and talking to different people in the University
> > >>>who has
> > >>> used different workflow languages, I myself convinced that
> introducing
> > >>>an
> > >>> another workflow language is not what actually they need. By changing
> > >>> exiting workflow language to another will not solve problems. What
> they
> > >>> asking is a easy way to construct the workflows. Indirectly what they
> > >>> asking for a sort of API which they can use to generate the workflows
> > >>>and
> > >>> run it. Correct me if i am wrong here.
> > >>>
> > >>> As most of above replies depict, if we can get a simple API, as an
> > >>> example, for a web based application, JavaScript API would be a good
> > >>> solution, and probably JSON would be a good candidate for language,
> > >>>instead
> > >>> of XML.
> > >>>
> > >>> Airavata community already have started to implement web base GUI.
> > >>>Hence
> > >>> introducing a JSON base JavaScript API would be great help. WDYT?
> > >>>
> > >>> Thanks,
> > >>> Shameera.
> > >>>
> > >>>
> > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> > >>> alekny7@gmail.com> wrote:
> > >>>
> > >>>> Hi,
> > >>>>
> > >>>> it is not dataflow instead focused on orchestrating REST services
> but
> > >>>> you may find it useful datapoint - we created worfklow service that
> > >>>>uses
> > >>>> natively JavaScript and JSON to describe what happens during
> workflow
> > >>>> execution:
> > >>>>
> > https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> > >>>>
> > >>>> HTH,
> > >>>>
> > >>>> Alek
> > >>>>
> > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> > >>>>wrote:
> > >>>>
> > >>>>> Hi Chris,
> > >>>>>
> > >>>>> Great to hear OODT community will be interested in adopting a JSON
> > >>>>> based workflow language and potentially a web based composer as
> well.
> > >>>>> Airavata previously had BPEL support initially through a home grown
> > >>>>> implementation [1] by Alek Slominski and later through Apache ODE
> > >>>>>[2]. Also
> > >>>>> a white paper [3] by Alek on this topic is an interesting read.
> > >>>>>
> > >>>>> I am of the same opinion that we should adopt something more modern
> > >>>>>as
> > >>>>> the challenges from scientific workflows seems to be converging
> with
> > >>>>>the
> > >>>>> data flow patterns in business workflows.
> > >>>>>
> > >>>>> It will be great if we can all compile a list of potential
> candidates
> > >>>>> and hack them through.
> > >>>>>
> > >>>>> Suresh
> > >>>>> [1] -
> > >>>>>
> > >>>>>
> > http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > >>>>> [2] -
> > >>>>>
> > >>>>>
> > http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> > >>>>>time_for_eScience_workflows
> > >>>>> [3] -
> > >>>>>
> > >>>>>
> > http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> > >>>>>6.pdf
> > >>>>>
> > >>>>>
> > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> > >>>>>
> > >>>>> > Hi Guys,
> > >>>>> >
> > >>>>> > I've been interested in this too - we don't per have a specific
> > >>>>> > OODT workflow language, but we specific workflows using XML, and
> > >>>>> > other configuration (we are also thinking of moving to JSON for
> > >>>>> > this).
> > >>>>> >
> > >>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> > >>>>> > to me.
> > >>>>> >
> > >>>>> > I wonder at the end of the day if we should adopt something more
> > >>>>> > modern like PIG or some other data flow type of language (PIG
> > >>>>> > is really neat).
> > >>>>> >
> > >>>>> > Cheers,
> > >>>>> > Chris
> > >>>>> >
> > >>>>> >
> > >>>>> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>>>> > Chris Mattmann, Ph.D.
> > >>>>> > Chief Architect
> > >>>>> > Instrument Software and Science Data Systems Section (398)
> > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > >>>>> > Office: 168-519, Mailstop: 168-527
> > >>>>> > Email: chris.a.mattmann@nasa.gov
> > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
> > >>>>> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>>>> > Adjunct Associate Professor, Computer Science Department
> > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> > >>>>> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> > -----Original Message-----
> > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> > >>>>> > Reply-To: "architecture@airavata.apache.org"
> > >>>>> > <ar...@airavata.apache.org>
> > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> > >>>>> > To: "architecture@airavata.apache.org" <
> > >>>>> architecture@airavata.apache.org>,
> > >>>>> > dev <de...@airavata.apache.org>
> > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> > >>>>>Airavata.
> > >>>>> >
> > >>>>> >> Hi All,
> > >>>>> >>
> > >>>>> >> As we all know Airavata has its own workflow language call XWF.
> > >>>>>When
> > >>>>> XWF
> > >>>>> >> was introduced, main focus points are interoperability and
> > >>>>> convertibility.
> > >>>>> >> But with years of experience it is convinced that above
> > >>>>>requirements
> > >>>>> are
> > >>>>> >> not really useful when we come to real world use cases. And XWF
> is
> > >>>>> XML
> > >>>>> >> based bulky language where we attache WSDLs and Workflow image
> it
> > >>>>> self.
> > >>>>> >> But
> > >>>>> >> with the recent changes WSDL part is being removed from XWF.
> > >>>>> >>
> > >>>>> >> It is worth to evaluate handy Scientific workflow languages in
> > >>>>> industry
> > >>>>> >> and
> > >>>>> >> find out pros and cons, at the end of this evaluation we need to
> > >>>>> come up
> > >>>>> >> with idea how we should improve Airavata workflow language,
> either
> > >>>>> we can
> > >>>>> >> improve existing XWF language, totally change to a new language
> > >>>>> available
> > >>>>> >> in industry or write a new light weight language. Basic
> > >>>>>requirements
> > >>>>> that
> > >>>>> >> we expect from new improvement are, high usability, flexible,
> > >>>>>light
> > >>>>> weight
> > >>>>> >> and real time monitoring support. As you can see above
> > >>>>>requirements
> > >>>>> are
> > >>>>> >> not
> > >>>>> >> direct comes with workflow languages but we need workflow
> language
> > >>>>> which
> > >>>>> >> help to support above requirements.
> > >>>>> >>
> > >>>>> >> After reading few papers and googling, initially i have come up
> > >>>>>with
> > >>>>> >> following three existing languages,
> > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> > >>>>> >> 2. WS-BPEL
> > >>>>> >> 3. SIDL
> > >>>>> >> <
> http://computation.llnl.gov/casc/components/index.html#page=home
> > >
> > >>>>> >>
> > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> > >>>>> Radical-SAGA
> > >>>>> >> also uses slightly modified version of SIDL. Other than above
> > >>>>>three
> > >>>>> >> languages we can come up with simple workflow language base on
> > >>>>> json(or
> > >>>>> >> yaml) which support all our requirements for some extends.
> > >>>>> >>
> > >>>>> >> It would be grate if I can get more input regarding the $Subject
> > >>>>> form the
> > >>>>> >> airavata community. You all are more than welcome to provide any
> > >>>>> type of
> > >>>>> >> suggestions.
> > >>>>> >>
> > >>>>> >> Thanks,
> > >>>>> >> Shameera.
> > >>>>> >>
> > >>>>> >>
> > >>>>> >>
> > >>>>> >> --
> > >>>>> >> Best Regards,
> > >>>>> >> Shameera Rathnayaka.
> > >>>>> >>
> > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> > >>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> The best way to predict the future is to invent it - Alan Kay
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Best Regards,
> > >>> Shameera Rathnayaka.
> > >>>
> > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > >>> Blog : http://shameerarathnayaka.blogspot.com/
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Supun Kamburugamuva
> > >> Member, Apache Software Foundation; http://www.apache.org
> > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > >> Blog: http://supunk.blogspot.com
> > >>
> > >>
> > >
> > >
> > >--
> > >Supun Kamburugamuva
> > >Member, Apache Software Foundation; http://www.apache.org
> > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > >Blog: http://supunk.blogspot.com
> >
> >
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Supun Kamburugamuva <su...@gmail.com>.
Hi Shameera,
Using python is a radical approach for workflow I think. But I believe this
is a very beautiful and new approach.Here is a possible scenario for
implementation and running using a Python based library.
The Python library facilitates the creation of Applications and submitting
them to Airavata for execution. The underlying functionality is exactly
similar to what Java clients provides.The only difference is that, Python
library should have a more fluent API than Java for easy creation of
workflows. We can generate the Python clients that talk to Airavata server
using Thrift.
Here is an example off the top of my head to a Python script created by
user for a Workflow. This is a very crude example and we need to come up
with a much better API if we are going to go along this path. First we need
to write a Python script that can execute a workflow using Airavata.
import airavata
host = Host("localhost", ....)
app1 = Application(host, ....)
app2 = Application(host, ....)
# we will connect these applications as a workflow using some topology
builder or other constructs
wb = WorkFlowBuilder()
wb.setApp("name1", app1)
# you can do a simple output transformation here etc
# connects the input of app1 to app2 etc
wb.setApp("name2", app2).connectInput("name1")
wb.submit()
Now we can load this Python script from XBaya. When XBaya loads this script
the Python script can output an XML configuration of the topology, XBaya
can render. There are other ways like directly executing the Python script
from command line and connecting XBaya indirectly as well. Now you can run
the workflow from XBaya. Running the Workflow means just executing the
Python script.
XBaya gets the notifications through messaging and update the UI
accordingly.
The users need to write the Python script by hand. XBaya cannot create the
script. Because workflow language is an actual python program the benefits
are immense. For example user can do workflow steering in the workflow
itself by subscribing to messages from Airavata.
Thanks,
Supun..
On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:
> Hi Supun,
>
> I meant to say JS is a well-known client side scripting language i have
> messed scripting part. Even we use Python, ultimately we should convert
> this to java model in server side, somehow we need to serialized python
> representation to the language which java can parse and generate that
> model. In this case we need to parse python script in java isn't it? I am
> not exactly clear how you suggesting to use python here. More details on
> how end system works if we use Python would be great help to clearly
> understand your points.
>
> Thanks,
> Shameera.
>
> On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <ch...@gmail.com>
> wrote:
>
> > Have you guys considered using JCC [1] as a means
> > to expose the workflow API currently in Java as a
> > Python API?
> >
> > We are exploring its use in OODT, and we have already
> > created a Tika [2] JCC-based python API.
> >
> > Cheers,
> > Chris
> >
> > [1] http://lucene.apache.org/pylucene/jcc/
> > [2] http://github.com/chrismattmann/tika-python/
> >
> > ------------------------
> > Chris Mattmann
> > chris.mattmann@gmail.com
> >
> >
> >
> >
> > -----Original Message-----
> > From: Supun Kamburugamuva <su...@gmail.com>
> > Reply-To: <de...@oodt.apache.org>
> > Date: Thursday, October 16, 2014 at 3:43 PM
> > To: dev <de...@airavata.apache.org>
> > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> > <sm...@apache.org>, "architecture@airavata.apache.org"
> > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> > <de...@oodt.apache.org>
> > Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
> >
> > >Once we had an offline discussion about the Airavata Workflow language
> > >(with Milinda, Saliya and Shameera). In that discussion one thing came
> out
> > >was why we need to invent a different language when a simple library
> like
> > >Python will full fill of Airavata requirements.
> > >
> > >There are many benefits in using a Python library as the API for
> > >controlling Airavata workflows.
> > >
> > >1. It is a library, gives the ultimate control over the execution and it
> > >can be simpler than any domain specific language that we can come with
> > >like
> > >XML, JSON etc
> > >2. Most people use python and can learn it easily than any Domain
> specific
> > >language
> > >3. You can easily create a Python library for Airavata because all the
> > >APIs
> > >are thrift based.
> > >4. If you design the constructs correctly you can plug an XBaya.
> > >
> > >Any thoughts?
> > >
> > >Thanks,
> > >Supun..
> > >
> > >
> > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <supun06@gmail.com
> >
> > >wrote:
> > >
> > >> Hi Shameera,
> > >>
> > >> Why you prefer JavaScript over a language like Python?
> > >>
> > >> Thanks,
> > >> Supun..
> > >>
> > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > >> shameerainfo@gmail.com> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> First of all thanks everyone for giving valuable inputs. After doing
> > >>>some
> > >>> background search and talking to different people in the University
> > >>>who has
> > >>> used different workflow languages, I myself convinced that
> introducing
> > >>>an
> > >>> another workflow language is not what actually they need. By changing
> > >>> exiting workflow language to another will not solve problems. What
> they
> > >>> asking is a easy way to construct the workflows. Indirectly what they
> > >>> asking for a sort of API which they can use to generate the workflows
> > >>>and
> > >>> run it. Correct me if i am wrong here.
> > >>>
> > >>> As most of above replies depict, if we can get a simple API, as an
> > >>> example, for a web based application, JavaScript API would be a good
> > >>> solution, and probably JSON would be a good candidate for language,
> > >>>instead
> > >>> of XML.
> > >>>
> > >>> Airavata community already have started to implement web base GUI.
> > >>>Hence
> > >>> introducing a JSON base JavaScript API would be great help. WDYT?
> > >>>
> > >>> Thanks,
> > >>> Shameera.
> > >>>
> > >>>
> > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> > >>> alekny7@gmail.com> wrote:
> > >>>
> > >>>> Hi,
> > >>>>
> > >>>> it is not dataflow instead focused on orchestrating REST services
> but
> > >>>> you may find it useful datapoint - we created worfklow service that
> > >>>>uses
> > >>>> natively JavaScript and JSON to describe what happens during
> workflow
> > >>>> execution:
> > >>>>
> > https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> > >>>>
> > >>>> HTH,
> > >>>>
> > >>>> Alek
> > >>>>
> > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> > >>>>wrote:
> > >>>>
> > >>>>> Hi Chris,
> > >>>>>
> > >>>>> Great to hear OODT community will be interested in adopting a JSON
> > >>>>> based workflow language and potentially a web based composer as
> well.
> > >>>>> Airavata previously had BPEL support initially through a home grown
> > >>>>> implementation [1] by Alek Slominski and later through Apache ODE
> > >>>>>[2]. Also
> > >>>>> a white paper [3] by Alek on this topic is an interesting read.
> > >>>>>
> > >>>>> I am of the same opinion that we should adopt something more modern
> > >>>>>as
> > >>>>> the challenges from scientific workflows seems to be converging
> with
> > >>>>>the
> > >>>>> data flow patterns in business workflows.
> > >>>>>
> > >>>>> It will be great if we can all compile a list of potential
> candidates
> > >>>>> and hack them through.
> > >>>>>
> > >>>>> Suresh
> > >>>>> [1] -
> > >>>>>
> > >>>>>
> > http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > >>>>> [2] -
> > >>>>>
> > >>>>>
> > http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> > >>>>>time_for_eScience_workflows
> > >>>>> [3] -
> > >>>>>
> > >>>>>
> > http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> > >>>>>6.pdf
> > >>>>>
> > >>>>>
> > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> > >>>>>
> > >>>>> > Hi Guys,
> > >>>>> >
> > >>>>> > I've been interested in this too - we don't per have a specific
> > >>>>> > OODT workflow language, but we specific workflows using XML, and
> > >>>>> > other configuration (we are also thinking of moving to JSON for
> > >>>>> > this).
> > >>>>> >
> > >>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> > >>>>> > to me.
> > >>>>> >
> > >>>>> > I wonder at the end of the day if we should adopt something more
> > >>>>> > modern like PIG or some other data flow type of language (PIG
> > >>>>> > is really neat).
> > >>>>> >
> > >>>>> > Cheers,
> > >>>>> > Chris
> > >>>>> >
> > >>>>> >
> > >>>>> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>>>> > Chris Mattmann, Ph.D.
> > >>>>> > Chief Architect
> > >>>>> > Instrument Software and Science Data Systems Section (398)
> > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > >>>>> > Office: 168-519, Mailstop: 168-527
> > >>>>> > Email: chris.a.mattmann@nasa.gov
> > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
> > >>>>> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>>>> > Adjunct Associate Professor, Computer Science Department
> > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> > >>>>> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> > -----Original Message-----
> > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> > >>>>> > Reply-To: "architecture@airavata.apache.org"
> > >>>>> > <ar...@airavata.apache.org>
> > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> > >>>>> > To: "architecture@airavata.apache.org" <
> > >>>>> architecture@airavata.apache.org>,
> > >>>>> > dev <de...@airavata.apache.org>
> > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> > >>>>>Airavata.
> > >>>>> >
> > >>>>> >> Hi All,
> > >>>>> >>
> > >>>>> >> As we all know Airavata has its own workflow language call XWF.
> > >>>>>When
> > >>>>> XWF
> > >>>>> >> was introduced, main focus points are interoperability and
> > >>>>> convertibility.
> > >>>>> >> But with years of experience it is convinced that above
> > >>>>>requirements
> > >>>>> are
> > >>>>> >> not really useful when we come to real world use cases. And XWF
> is
> > >>>>> XML
> > >>>>> >> based bulky language where we attache WSDLs and Workflow image
> it
> > >>>>> self.
> > >>>>> >> But
> > >>>>> >> with the recent changes WSDL part is being removed from XWF.
> > >>>>> >>
> > >>>>> >> It is worth to evaluate handy Scientific workflow languages in
> > >>>>> industry
> > >>>>> >> and
> > >>>>> >> find out pros and cons, at the end of this evaluation we need to
> > >>>>> come up
> > >>>>> >> with idea how we should improve Airavata workflow language,
> either
> > >>>>> we can
> > >>>>> >> improve existing XWF language, totally change to a new language
> > >>>>> available
> > >>>>> >> in industry or write a new light weight language. Basic
> > >>>>>requirements
> > >>>>> that
> > >>>>> >> we expect from new improvement are, high usability, flexible,
> > >>>>>light
> > >>>>> weight
> > >>>>> >> and real time monitoring support. As you can see above
> > >>>>>requirements
> > >>>>> are
> > >>>>> >> not
> > >>>>> >> direct comes with workflow languages but we need workflow
> language
> > >>>>> which
> > >>>>> >> help to support above requirements.
> > >>>>> >>
> > >>>>> >> After reading few papers and googling, initially i have come up
> > >>>>>with
> > >>>>> >> following three existing languages,
> > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> > >>>>> >> 2. WS-BPEL
> > >>>>> >> 3. SIDL
> > >>>>> >> <
> http://computation.llnl.gov/casc/components/index.html#page=home
> > >
> > >>>>> >>
> > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> > >>>>> Radical-SAGA
> > >>>>> >> also uses slightly modified version of SIDL. Other than above
> > >>>>>three
> > >>>>> >> languages we can come up with simple workflow language base on
> > >>>>> json(or
> > >>>>> >> yaml) which support all our requirements for some extends.
> > >>>>> >>
> > >>>>> >> It would be grate if I can get more input regarding the $Subject
> > >>>>> form the
> > >>>>> >> airavata community. You all are more than welcome to provide any
> > >>>>> type of
> > >>>>> >> suggestions.
> > >>>>> >>
> > >>>>> >> Thanks,
> > >>>>> >> Shameera.
> > >>>>> >>
> > >>>>> >>
> > >>>>> >>
> > >>>>> >> --
> > >>>>> >> Best Regards,
> > >>>>> >> Shameera Rathnayaka.
> > >>>>> >>
> > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> > >>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> The best way to predict the future is to invent it - Alan Kay
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Best Regards,
> > >>> Shameera Rathnayaka.
> > >>>
> > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > >>> Blog : http://shameerarathnayaka.blogspot.com/
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Supun Kamburugamuva
> > >> Member, Apache Software Foundation; http://www.apache.org
> > >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > >> Blog: http://supunk.blogspot.com
> > >>
> > >>
> > >
> > >
> > >--
> > >Supun Kamburugamuva
> > >Member, Apache Software Foundation; http://www.apache.org
> > >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > >Blog: http://supunk.blogspot.com
> >
> >
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi Supun,
I meant to say JS is a well-known client side scripting language i have
messed scripting part. Even we use Python, ultimately we should convert
this to java model in server side, somehow we need to serialized python
representation to the language which java can parse and generate that
model. In this case we need to parse python script in java isn't it? I am
not exactly clear how you suggesting to use python here. More details on
how end system works if we use Python would be great help to clearly
understand your points.
Thanks,
Shameera.
On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <ch...@gmail.com>
wrote:
> Have you guys considered using JCC [1] as a means
> to expose the workflow API currently in Java as a
> Python API?
>
> We are exploring its use in OODT, and we have already
> created a Tika [2] JCC-based python API.
>
> Cheers,
> Chris
>
> [1] http://lucene.apache.org/pylucene/jcc/
> [2] http://github.com/chrismattmann/tika-python/
>
> ------------------------
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
> -----Original Message-----
> From: Supun Kamburugamuva <su...@gmail.com>
> Reply-To: <de...@oodt.apache.org>
> Date: Thursday, October 16, 2014 at 3:43 PM
> To: dev <de...@airavata.apache.org>
> Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> <sm...@apache.org>, "architecture@airavata.apache.org"
> <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> <de...@oodt.apache.org>
> Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
>
> >Once we had an offline discussion about the Airavata Workflow language
> >(with Milinda, Saliya and Shameera). In that discussion one thing came out
> >was why we need to invent a different language when a simple library like
> >Python will full fill of Airavata requirements.
> >
> >There are many benefits in using a Python library as the API for
> >controlling Airavata workflows.
> >
> >1. It is a library, gives the ultimate control over the execution and it
> >can be simpler than any domain specific language that we can come with
> >like
> >XML, JSON etc
> >2. Most people use python and can learn it easily than any Domain specific
> >language
> >3. You can easily create a Python library for Airavata because all the
> >APIs
> >are thrift based.
> >4. If you design the constructs correctly you can plug an XBaya.
> >
> >Any thoughts?
> >
> >Thanks,
> >Supun..
> >
> >
> >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> >wrote:
> >
> >> Hi Shameera,
> >>
> >> Why you prefer JavaScript over a language like Python?
> >>
> >> Thanks,
> >> Supun..
> >>
> >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> >> shameerainfo@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> First of all thanks everyone for giving valuable inputs. After doing
> >>>some
> >>> background search and talking to different people in the University
> >>>who has
> >>> used different workflow languages, I myself convinced that introducing
> >>>an
> >>> another workflow language is not what actually they need. By changing
> >>> exiting workflow language to another will not solve problems. What they
> >>> asking is a easy way to construct the workflows. Indirectly what they
> >>> asking for a sort of API which they can use to generate the workflows
> >>>and
> >>> run it. Correct me if i am wrong here.
> >>>
> >>> As most of above replies depict, if we can get a simple API, as an
> >>> example, for a web based application, JavaScript API would be a good
> >>> solution, and probably JSON would be a good candidate for language,
> >>>instead
> >>> of XML.
> >>>
> >>> Airavata community already have started to implement web base GUI.
> >>>Hence
> >>> introducing a JSON base JavaScript API would be great help. WDYT?
> >>>
> >>> Thanks,
> >>> Shameera.
> >>>
> >>>
> >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >>> alekny7@gmail.com> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> it is not dataflow instead focused on orchestrating REST services but
> >>>> you may find it useful datapoint - we created worfklow service that
> >>>>uses
> >>>> natively JavaScript and JSON to describe what happens during workflow
> >>>> execution:
> >>>>
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>>
> >>>> HTH,
> >>>>
> >>>> Alek
> >>>>
> >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> >>>>wrote:
> >>>>
> >>>>> Hi Chris,
> >>>>>
> >>>>> Great to hear OODT community will be interested in adopting a JSON
> >>>>> based workflow language and potentially a web based composer as well.
> >>>>> Airavata previously had BPEL support initially through a home grown
> >>>>> implementation [1] by Alek Slominski and later through Apache ODE
> >>>>>[2]. Also
> >>>>> a white paper [3] by Alek on this topic is an interesting read.
> >>>>>
> >>>>> I am of the same opinion that we should adopt something more modern
> >>>>>as
> >>>>> the challenges from scientific workflows seems to be converging with
> >>>>>the
> >>>>> data flow patterns in business workflows.
> >>>>>
> >>>>> It will be great if we can all compile a list of potential candidates
> >>>>> and hack them through.
> >>>>>
> >>>>> Suresh
> >>>>> [1] -
> >>>>>
> >>>>>
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>>>> [2] -
> >>>>>
> >>>>>
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> >>>>>time_for_eScience_workflows
> >>>>> [3] -
> >>>>>
> >>>>>
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> >>>>>6.pdf
> >>>>>
> >>>>>
> >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>>>>
> >>>>> > Hi Guys,
> >>>>> >
> >>>>> > I've been interested in this too - we don't per have a specific
> >>>>> > OODT workflow language, but we specific workflows using XML, and
> >>>>> > other configuration (we are also thinking of moving to JSON for
> >>>>> > this).
> >>>>> >
> >>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> >>>>> > to me.
> >>>>> >
> >>>>> > I wonder at the end of the day if we should adopt something more
> >>>>> > modern like PIG or some other data flow type of language (PIG
> >>>>> > is really neat).
> >>>>> >
> >>>>> > Cheers,
> >>>>> > Chris
> >>>>> >
> >>>>> >
> >>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>> > Chris Mattmann, Ph.D.
> >>>>> > Chief Architect
> >>>>> > Instrument Software and Science Data Systems Section (398)
> >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>>>> > Office: 168-519, Mailstop: 168-527
> >>>>> > Email: chris.a.mattmann@nasa.gov
> >>>>> > WWW: http://sunset.usc.edu/~mattmann/
> >>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>> > Adjunct Associate Professor, Computer Science Department
> >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> >>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > -----Original Message-----
> >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>>>> > Reply-To: "architecture@airavata.apache.org"
> >>>>> > <ar...@airavata.apache.org>
> >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>>>> > To: "architecture@airavata.apache.org" <
> >>>>> architecture@airavata.apache.org>,
> >>>>> > dev <de...@airavata.apache.org>
> >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> >>>>>Airavata.
> >>>>> >
> >>>>> >> Hi All,
> >>>>> >>
> >>>>> >> As we all know Airavata has its own workflow language call XWF.
> >>>>>When
> >>>>> XWF
> >>>>> >> was introduced, main focus points are interoperability and
> >>>>> convertibility.
> >>>>> >> But with years of experience it is convinced that above
> >>>>>requirements
> >>>>> are
> >>>>> >> not really useful when we come to real world use cases. And XWF is
> >>>>> XML
> >>>>> >> based bulky language where we attache WSDLs and Workflow image it
> >>>>> self.
> >>>>> >> But
> >>>>> >> with the recent changes WSDL part is being removed from XWF.
> >>>>> >>
> >>>>> >> It is worth to evaluate handy Scientific workflow languages in
> >>>>> industry
> >>>>> >> and
> >>>>> >> find out pros and cons, at the end of this evaluation we need to
> >>>>> come up
> >>>>> >> with idea how we should improve Airavata workflow language, either
> >>>>> we can
> >>>>> >> improve existing XWF language, totally change to a new language
> >>>>> available
> >>>>> >> in industry or write a new light weight language. Basic
> >>>>>requirements
> >>>>> that
> >>>>> >> we expect from new improvement are, high usability, flexible,
> >>>>>light
> >>>>> weight
> >>>>> >> and real time monitoring support. As you can see above
> >>>>>requirements
> >>>>> are
> >>>>> >> not
> >>>>> >> direct comes with workflow languages but we need workflow language
> >>>>> which
> >>>>> >> help to support above requirements.
> >>>>> >>
> >>>>> >> After reading few papers and googling, initially i have come up
> >>>>>with
> >>>>> >> following three existing languages,
> >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>>>> >> 2. WS-BPEL
> >>>>> >> 3. SIDL
> >>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home
> >
> >>>>> >>
> >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> >>>>> Radical-SAGA
> >>>>> >> also uses slightly modified version of SIDL. Other than above
> >>>>>three
> >>>>> >> languages we can come up with simple workflow language base on
> >>>>> json(or
> >>>>> >> yaml) which support all our requirements for some extends.
> >>>>> >>
> >>>>> >> It would be grate if I can get more input regarding the $Subject
> >>>>> form the
> >>>>> >> airavata community. You all are more than welcome to provide any
> >>>>> type of
> >>>>> >> suggestions.
> >>>>> >>
> >>>>> >> Thanks,
> >>>>> >> Shameera.
> >>>>> >>
> >>>>> >>
> >>>>> >>
> >>>>> >> --
> >>>>> >> Best Regards,
> >>>>> >> Shameera Rathnayaka.
> >>>>> >>
> >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> The best way to predict the future is to invent it - Alan Kay
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards,
> >>> Shameera Rathnayaka.
> >>>
> >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> Blog : http://shameerarathnayaka.blogspot.com/
> >>>
> >>
> >>
> >>
> >> --
> >> Supun Kamburugamuva
> >> Member, Apache Software Foundation; http://www.apache.org
> >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> >> Blog: http://supunk.blogspot.com
> >>
> >>
> >
> >
> >--
> >Supun Kamburugamuva
> >Member, Apache Software Foundation; http://www.apache.org
> >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> >Blog: http://supunk.blogspot.com
>
>
>
--
Best Regards,
Shameera Rathnayaka.
email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi Supun,
I meant to say JS is a well-known client side scripting language i have
messed scripting part. Even we use Python, ultimately we should convert
this to java model in server side, somehow we need to serialized python
representation to the language which java can parse and generate that
model. In this case we need to parse python script in java isn't it? I am
not exactly clear how you suggesting to use python here. More details on
how end system works if we use Python would be great help to clearly
understand your points.
Thanks,
Shameera.
On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <ch...@gmail.com>
wrote:
> Have you guys considered using JCC [1] as a means
> to expose the workflow API currently in Java as a
> Python API?
>
> We are exploring its use in OODT, and we have already
> created a Tika [2] JCC-based python API.
>
> Cheers,
> Chris
>
> [1] http://lucene.apache.org/pylucene/jcc/
> [2] http://github.com/chrismattmann/tika-python/
>
> ------------------------
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
> -----Original Message-----
> From: Supun Kamburugamuva <su...@gmail.com>
> Reply-To: <de...@oodt.apache.org>
> Date: Thursday, October 16, 2014 at 3:43 PM
> To: dev <de...@airavata.apache.org>
> Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> <sm...@apache.org>, "architecture@airavata.apache.org"
> <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> <de...@oodt.apache.org>
> Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
>
> >Once we had an offline discussion about the Airavata Workflow language
> >(with Milinda, Saliya and Shameera). In that discussion one thing came out
> >was why we need to invent a different language when a simple library like
> >Python will full fill of Airavata requirements.
> >
> >There are many benefits in using a Python library as the API for
> >controlling Airavata workflows.
> >
> >1. It is a library, gives the ultimate control over the execution and it
> >can be simpler than any domain specific language that we can come with
> >like
> >XML, JSON etc
> >2. Most people use python and can learn it easily than any Domain specific
> >language
> >3. You can easily create a Python library for Airavata because all the
> >APIs
> >are thrift based.
> >4. If you design the constructs correctly you can plug an XBaya.
> >
> >Any thoughts?
> >
> >Thanks,
> >Supun..
> >
> >
> >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> >wrote:
> >
> >> Hi Shameera,
> >>
> >> Why you prefer JavaScript over a language like Python?
> >>
> >> Thanks,
> >> Supun..
> >>
> >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> >> shameerainfo@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> First of all thanks everyone for giving valuable inputs. After doing
> >>>some
> >>> background search and talking to different people in the University
> >>>who has
> >>> used different workflow languages, I myself convinced that introducing
> >>>an
> >>> another workflow language is not what actually they need. By changing
> >>> exiting workflow language to another will not solve problems. What they
> >>> asking is a easy way to construct the workflows. Indirectly what they
> >>> asking for a sort of API which they can use to generate the workflows
> >>>and
> >>> run it. Correct me if i am wrong here.
> >>>
> >>> As most of above replies depict, if we can get a simple API, as an
> >>> example, for a web based application, JavaScript API would be a good
> >>> solution, and probably JSON would be a good candidate for language,
> >>>instead
> >>> of XML.
> >>>
> >>> Airavata community already have started to implement web base GUI.
> >>>Hence
> >>> introducing a JSON base JavaScript API would be great help. WDYT?
> >>>
> >>> Thanks,
> >>> Shameera.
> >>>
> >>>
> >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >>> alekny7@gmail.com> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> it is not dataflow instead focused on orchestrating REST services but
> >>>> you may find it useful datapoint - we created worfklow service that
> >>>>uses
> >>>> natively JavaScript and JSON to describe what happens during workflow
> >>>> execution:
> >>>>
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>>
> >>>> HTH,
> >>>>
> >>>> Alek
> >>>>
> >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> >>>>wrote:
> >>>>
> >>>>> Hi Chris,
> >>>>>
> >>>>> Great to hear OODT community will be interested in adopting a JSON
> >>>>> based workflow language and potentially a web based composer as well.
> >>>>> Airavata previously had BPEL support initially through a home grown
> >>>>> implementation [1] by Alek Slominski and later through Apache ODE
> >>>>>[2]. Also
> >>>>> a white paper [3] by Alek on this topic is an interesting read.
> >>>>>
> >>>>> I am of the same opinion that we should adopt something more modern
> >>>>>as
> >>>>> the challenges from scientific workflows seems to be converging with
> >>>>>the
> >>>>> data flow patterns in business workflows.
> >>>>>
> >>>>> It will be great if we can all compile a list of potential candidates
> >>>>> and hack them through.
> >>>>>
> >>>>> Suresh
> >>>>> [1] -
> >>>>>
> >>>>>
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>>>> [2] -
> >>>>>
> >>>>>
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> >>>>>time_for_eScience_workflows
> >>>>> [3] -
> >>>>>
> >>>>>
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> >>>>>6.pdf
> >>>>>
> >>>>>
> >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>>>>
> >>>>> > Hi Guys,
> >>>>> >
> >>>>> > I've been interested in this too - we don't per have a specific
> >>>>> > OODT workflow language, but we specific workflows using XML, and
> >>>>> > other configuration (we are also thinking of moving to JSON for
> >>>>> > this).
> >>>>> >
> >>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> >>>>> > to me.
> >>>>> >
> >>>>> > I wonder at the end of the day if we should adopt something more
> >>>>> > modern like PIG or some other data flow type of language (PIG
> >>>>> > is really neat).
> >>>>> >
> >>>>> > Cheers,
> >>>>> > Chris
> >>>>> >
> >>>>> >
> >>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>> > Chris Mattmann, Ph.D.
> >>>>> > Chief Architect
> >>>>> > Instrument Software and Science Data Systems Section (398)
> >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>>>> > Office: 168-519, Mailstop: 168-527
> >>>>> > Email: chris.a.mattmann@nasa.gov
> >>>>> > WWW: http://sunset.usc.edu/~mattmann/
> >>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>> > Adjunct Associate Professor, Computer Science Department
> >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> >>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > -----Original Message-----
> >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>>>> > Reply-To: "architecture@airavata.apache.org"
> >>>>> > <ar...@airavata.apache.org>
> >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>>>> > To: "architecture@airavata.apache.org" <
> >>>>> architecture@airavata.apache.org>,
> >>>>> > dev <de...@airavata.apache.org>
> >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> >>>>>Airavata.
> >>>>> >
> >>>>> >> Hi All,
> >>>>> >>
> >>>>> >> As we all know Airavata has its own workflow language call XWF.
> >>>>>When
> >>>>> XWF
> >>>>> >> was introduced, main focus points are interoperability and
> >>>>> convertibility.
> >>>>> >> But with years of experience it is convinced that above
> >>>>>requirements
> >>>>> are
> >>>>> >> not really useful when we come to real world use cases. And XWF is
> >>>>> XML
> >>>>> >> based bulky language where we attache WSDLs and Workflow image it
> >>>>> self.
> >>>>> >> But
> >>>>> >> with the recent changes WSDL part is being removed from XWF.
> >>>>> >>
> >>>>> >> It is worth to evaluate handy Scientific workflow languages in
> >>>>> industry
> >>>>> >> and
> >>>>> >> find out pros and cons, at the end of this evaluation we need to
> >>>>> come up
> >>>>> >> with idea how we should improve Airavata workflow language, either
> >>>>> we can
> >>>>> >> improve existing XWF language, totally change to a new language
> >>>>> available
> >>>>> >> in industry or write a new light weight language. Basic
> >>>>>requirements
> >>>>> that
> >>>>> >> we expect from new improvement are, high usability, flexible,
> >>>>>light
> >>>>> weight
> >>>>> >> and real time monitoring support. As you can see above
> >>>>>requirements
> >>>>> are
> >>>>> >> not
> >>>>> >> direct comes with workflow languages but we need workflow language
> >>>>> which
> >>>>> >> help to support above requirements.
> >>>>> >>
> >>>>> >> After reading few papers and googling, initially i have come up
> >>>>>with
> >>>>> >> following three existing languages,
> >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>>>> >> 2. WS-BPEL
> >>>>> >> 3. SIDL
> >>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home
> >
> >>>>> >>
> >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> >>>>> Radical-SAGA
> >>>>> >> also uses slightly modified version of SIDL. Other than above
> >>>>>three
> >>>>> >> languages we can come up with simple workflow language base on
> >>>>> json(or
> >>>>> >> yaml) which support all our requirements for some extends.
> >>>>> >>
> >>>>> >> It would be grate if I can get more input regarding the $Subject
> >>>>> form the
> >>>>> >> airavata community. You all are more than welcome to provide any
> >>>>> type of
> >>>>> >> suggestions.
> >>>>> >>
> >>>>> >> Thanks,
> >>>>> >> Shameera.
> >>>>> >>
> >>>>> >>
> >>>>> >>
> >>>>> >> --
> >>>>> >> Best Regards,
> >>>>> >> Shameera Rathnayaka.
> >>>>> >>
> >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> The best way to predict the future is to invent it - Alan Kay
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards,
> >>> Shameera Rathnayaka.
> >>>
> >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> Blog : http://shameerarathnayaka.blogspot.com/
> >>>
> >>
> >>
> >>
> >> --
> >> Supun Kamburugamuva
> >> Member, Apache Software Foundation; http://www.apache.org
> >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> >> Blog: http://supunk.blogspot.com
> >>
> >>
> >
> >
> >--
> >Supun Kamburugamuva
> >Member, Apache Software Foundation; http://www.apache.org
> >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> >Blog: http://supunk.blogspot.com
>
>
>
--
Best Regards,
Shameera Rathnayaka.
email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi Supun,
I meant to say JS is a well-known client side scripting language i have
messed scripting part. Even we use Python, ultimately we should convert
this to java model in server side, somehow we need to serialized python
representation to the language which java can parse and generate that
model. In this case we need to parse python script in java isn't it? I am
not exactly clear how you suggesting to use python here. More details on
how end system works if we use Python would be great help to clearly
understand your points.
Thanks,
Shameera.
On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <ch...@gmail.com>
wrote:
> Have you guys considered using JCC [1] as a means
> to expose the workflow API currently in Java as a
> Python API?
>
> We are exploring its use in OODT, and we have already
> created a Tika [2] JCC-based python API.
>
> Cheers,
> Chris
>
> [1] http://lucene.apache.org/pylucene/jcc/
> [2] http://github.com/chrismattmann/tika-python/
>
> ------------------------
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
> -----Original Message-----
> From: Supun Kamburugamuva <su...@gmail.com>
> Reply-To: <de...@oodt.apache.org>
> Date: Thursday, October 16, 2014 at 3:43 PM
> To: dev <de...@airavata.apache.org>
> Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> <sm...@apache.org>, "architecture@airavata.apache.org"
> <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> <de...@oodt.apache.org>
> Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
>
> >Once we had an offline discussion about the Airavata Workflow language
> >(with Milinda, Saliya and Shameera). In that discussion one thing came out
> >was why we need to invent a different language when a simple library like
> >Python will full fill of Airavata requirements.
> >
> >There are many benefits in using a Python library as the API for
> >controlling Airavata workflows.
> >
> >1. It is a library, gives the ultimate control over the execution and it
> >can be simpler than any domain specific language that we can come with
> >like
> >XML, JSON etc
> >2. Most people use python and can learn it easily than any Domain specific
> >language
> >3. You can easily create a Python library for Airavata because all the
> >APIs
> >are thrift based.
> >4. If you design the constructs correctly you can plug an XBaya.
> >
> >Any thoughts?
> >
> >Thanks,
> >Supun..
> >
> >
> >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> >wrote:
> >
> >> Hi Shameera,
> >>
> >> Why you prefer JavaScript over a language like Python?
> >>
> >> Thanks,
> >> Supun..
> >>
> >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> >> shameerainfo@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> First of all thanks everyone for giving valuable inputs. After doing
> >>>some
> >>> background search and talking to different people in the University
> >>>who has
> >>> used different workflow languages, I myself convinced that introducing
> >>>an
> >>> another workflow language is not what actually they need. By changing
> >>> exiting workflow language to another will not solve problems. What they
> >>> asking is a easy way to construct the workflows. Indirectly what they
> >>> asking for a sort of API which they can use to generate the workflows
> >>>and
> >>> run it. Correct me if i am wrong here.
> >>>
> >>> As most of above replies depict, if we can get a simple API, as an
> >>> example, for a web based application, JavaScript API would be a good
> >>> solution, and probably JSON would be a good candidate for language,
> >>>instead
> >>> of XML.
> >>>
> >>> Airavata community already have started to implement web base GUI.
> >>>Hence
> >>> introducing a JSON base JavaScript API would be great help. WDYT?
> >>>
> >>> Thanks,
> >>> Shameera.
> >>>
> >>>
> >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >>> alekny7@gmail.com> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> it is not dataflow instead focused on orchestrating REST services but
> >>>> you may find it useful datapoint - we created worfklow service that
> >>>>uses
> >>>> natively JavaScript and JSON to describe what happens during workflow
> >>>> execution:
> >>>>
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>>
> >>>> HTH,
> >>>>
> >>>> Alek
> >>>>
> >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> >>>>wrote:
> >>>>
> >>>>> Hi Chris,
> >>>>>
> >>>>> Great to hear OODT community will be interested in adopting a JSON
> >>>>> based workflow language and potentially a web based composer as well.
> >>>>> Airavata previously had BPEL support initially through a home grown
> >>>>> implementation [1] by Alek Slominski and later through Apache ODE
> >>>>>[2]. Also
> >>>>> a white paper [3] by Alek on this topic is an interesting read.
> >>>>>
> >>>>> I am of the same opinion that we should adopt something more modern
> >>>>>as
> >>>>> the challenges from scientific workflows seems to be converging with
> >>>>>the
> >>>>> data flow patterns in business workflows.
> >>>>>
> >>>>> It will be great if we can all compile a list of potential candidates
> >>>>> and hack them through.
> >>>>>
> >>>>> Suresh
> >>>>> [1] -
> >>>>>
> >>>>>
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>>>> [2] -
> >>>>>
> >>>>>
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> >>>>>time_for_eScience_workflows
> >>>>> [3] -
> >>>>>
> >>>>>
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> >>>>>6.pdf
> >>>>>
> >>>>>
> >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>>>>
> >>>>> > Hi Guys,
> >>>>> >
> >>>>> > I've been interested in this too - we don't per have a specific
> >>>>> > OODT workflow language, but we specific workflows using XML, and
> >>>>> > other configuration (we are also thinking of moving to JSON for
> >>>>> > this).
> >>>>> >
> >>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> >>>>> > to me.
> >>>>> >
> >>>>> > I wonder at the end of the day if we should adopt something more
> >>>>> > modern like PIG or some other data flow type of language (PIG
> >>>>> > is really neat).
> >>>>> >
> >>>>> > Cheers,
> >>>>> > Chris
> >>>>> >
> >>>>> >
> >>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>> > Chris Mattmann, Ph.D.
> >>>>> > Chief Architect
> >>>>> > Instrument Software and Science Data Systems Section (398)
> >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>>>> > Office: 168-519, Mailstop: 168-527
> >>>>> > Email: chris.a.mattmann@nasa.gov
> >>>>> > WWW: http://sunset.usc.edu/~mattmann/
> >>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>> > Adjunct Associate Professor, Computer Science Department
> >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> >>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > -----Original Message-----
> >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>>>> > Reply-To: "architecture@airavata.apache.org"
> >>>>> > <ar...@airavata.apache.org>
> >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>>>> > To: "architecture@airavata.apache.org" <
> >>>>> architecture@airavata.apache.org>,
> >>>>> > dev <de...@airavata.apache.org>
> >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> >>>>>Airavata.
> >>>>> >
> >>>>> >> Hi All,
> >>>>> >>
> >>>>> >> As we all know Airavata has its own workflow language call XWF.
> >>>>>When
> >>>>> XWF
> >>>>> >> was introduced, main focus points are interoperability and
> >>>>> convertibility.
> >>>>> >> But with years of experience it is convinced that above
> >>>>>requirements
> >>>>> are
> >>>>> >> not really useful when we come to real world use cases. And XWF is
> >>>>> XML
> >>>>> >> based bulky language where we attache WSDLs and Workflow image it
> >>>>> self.
> >>>>> >> But
> >>>>> >> with the recent changes WSDL part is being removed from XWF.
> >>>>> >>
> >>>>> >> It is worth to evaluate handy Scientific workflow languages in
> >>>>> industry
> >>>>> >> and
> >>>>> >> find out pros and cons, at the end of this evaluation we need to
> >>>>> come up
> >>>>> >> with idea how we should improve Airavata workflow language, either
> >>>>> we can
> >>>>> >> improve existing XWF language, totally change to a new language
> >>>>> available
> >>>>> >> in industry or write a new light weight language. Basic
> >>>>>requirements
> >>>>> that
> >>>>> >> we expect from new improvement are, high usability, flexible,
> >>>>>light
> >>>>> weight
> >>>>> >> and real time monitoring support. As you can see above
> >>>>>requirements
> >>>>> are
> >>>>> >> not
> >>>>> >> direct comes with workflow languages but we need workflow language
> >>>>> which
> >>>>> >> help to support above requirements.
> >>>>> >>
> >>>>> >> After reading few papers and googling, initially i have come up
> >>>>>with
> >>>>> >> following three existing languages,
> >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>>>> >> 2. WS-BPEL
> >>>>> >> 3. SIDL
> >>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home
> >
> >>>>> >>
> >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> >>>>> Radical-SAGA
> >>>>> >> also uses slightly modified version of SIDL. Other than above
> >>>>>three
> >>>>> >> languages we can come up with simple workflow language base on
> >>>>> json(or
> >>>>> >> yaml) which support all our requirements for some extends.
> >>>>> >>
> >>>>> >> It would be grate if I can get more input regarding the $Subject
> >>>>> form the
> >>>>> >> airavata community. You all are more than welcome to provide any
> >>>>> type of
> >>>>> >> suggestions.
> >>>>> >>
> >>>>> >> Thanks,
> >>>>> >> Shameera.
> >>>>> >>
> >>>>> >>
> >>>>> >>
> >>>>> >> --
> >>>>> >> Best Regards,
> >>>>> >> Shameera Rathnayaka.
> >>>>> >>
> >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> The best way to predict the future is to invent it - Alan Kay
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards,
> >>> Shameera Rathnayaka.
> >>>
> >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> Blog : http://shameerarathnayaka.blogspot.com/
> >>>
> >>
> >>
> >>
> >> --
> >> Supun Kamburugamuva
> >> Member, Apache Software Foundation; http://www.apache.org
> >> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> >> Blog: http://supunk.blogspot.com
> >>
> >>
> >
> >
> >--
> >Supun Kamburugamuva
> >Member, Apache Software Foundation; http://www.apache.org
> >E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> >Blog: http://supunk.blogspot.com
>
>
>
--
Best Regards,
Shameera Rathnayaka.
email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Chris Mattmann <ch...@gmail.com>.
Have you guys considered using JCC [1] as a means
to expose the workflow API currently in Java as a
Python API?
We are exploring its use in OODT, and we have already
created a Tika [2] JCC-based python API.
Cheers,
Chris
[1] http://lucene.apache.org/pylucene/jcc/
[2] http://github.com/chrismattmann/tika-python/
------------------------
Chris Mattmann
chris.mattmann@gmail.com
-----Original Message-----
From: Supun Kamburugamuva <su...@gmail.com>
Reply-To: <de...@oodt.apache.org>
Date: Thursday, October 16, 2014 at 3:43 PM
To: dev <de...@airavata.apache.org>
Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
<sm...@apache.org>, "architecture@airavata.apache.org"
<ar...@airavata.apache.org>, "dev@oodt.apache.org"
<de...@oodt.apache.org>
Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
>Once we had an offline discussion about the Airavata Workflow language
>(with Milinda, Saliya and Shameera). In that discussion one thing came out
>was why we need to invent a different language when a simple library like
>Python will full fill of Airavata requirements.
>
>There are many benefits in using a Python library as the API for
>controlling Airavata workflows.
>
>1. It is a library, gives the ultimate control over the execution and it
>can be simpler than any domain specific language that we can come with
>like
>XML, JSON etc
>2. Most people use python and can learn it easily than any Domain specific
>language
>3. You can easily create a Python library for Airavata because all the
>APIs
>are thrift based.
>4. If you design the constructs correctly you can plug an XBaya.
>
>Any thoughts?
>
>Thanks,
>Supun..
>
>
>On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
>wrote:
>
>> Hi Shameera,
>>
>> Why you prefer JavaScript over a language like Python?
>>
>> Thanks,
>> Supun..
>>
>> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> First of all thanks everyone for giving valuable inputs. After doing
>>>some
>>> background search and talking to different people in the University
>>>who has
>>> used different workflow languages, I myself convinced that introducing
>>>an
>>> another workflow language is not what actually they need. By changing
>>> exiting workflow language to another will not solve problems. What they
>>> asking is a easy way to construct the workflows. Indirectly what they
>>> asking for a sort of API which they can use to generate the workflows
>>>and
>>> run it. Correct me if i am wrong here.
>>>
>>> As most of above replies depict, if we can get a simple API, as an
>>> example, for a web based application, JavaScript API would be a good
>>> solution, and probably JSON would be a good candidate for language,
>>>instead
>>> of XML.
>>>
>>> Airavata community already have started to implement web base GUI.
>>>Hence
>>> introducing a JSON base JavaScript API would be great help. WDYT?
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> alekny7@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> it is not dataflow instead focused on orchestrating REST services but
>>>> you may find it useful datapoint - we created worfklow service that
>>>>uses
>>>> natively JavaScript and JSON to describe what happens during workflow
>>>> execution:
>>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>>
>>>> HTH,
>>>>
>>>> Alek
>>>>
>>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
>>>>wrote:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>> Great to hear OODT community will be interested in adopting a JSON
>>>>> based workflow language and potentially a web based composer as well.
>>>>> Airavata previously had BPEL support initially through a home grown
>>>>> implementation [1] by Alek Slominski and later through Apache ODE
>>>>>[2]. Also
>>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>>
>>>>> I am of the same opinion that we should adopt something more modern
>>>>>as
>>>>> the challenges from scientific workflows seems to be converging with
>>>>>the
>>>>> data flow patterns in business workflows.
>>>>>
>>>>> It will be great if we can all compile a list of potential candidates
>>>>> and hack them through.
>>>>>
>>>>> Suresh
>>>>> [1] -
>>>>>
>>>>>http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>>> [2] -
>>>>>
>>>>>http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>>>>>time_for_eScience_workflows
>>>>> [3] -
>>>>>
>>>>>http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>>>>>6.pdf
>>>>>
>>>>>
>>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>>
>>>>> > Hi Guys,
>>>>> >
>>>>> > I've been interested in this too - we don't per have a specific
>>>>> > OODT workflow language, but we specific workflows using XML, and
>>>>> > other configuration (we are also thinking of moving to JSON for
>>>>> > this).
>>>>> >
>>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>>> > to me.
>>>>> >
>>>>> > I wonder at the end of the day if we should adopt something more
>>>>> > modern like PIG or some other data flow type of language (PIG
>>>>> > is really neat).
>>>>> >
>>>>> > Cheers,
>>>>> > Chris
>>>>> >
>>>>> >
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Chris Mattmann, Ph.D.
>>>>> > Chief Architect
>>>>> > Instrument Software and Science Data Systems Section (398)
>>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>> > Office: 168-519, Mailstop: 168-527
>>>>> > Email: chris.a.mattmann@nasa.gov
>>>>> > WWW: http://sunset.usc.edu/~mattmann/
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Adjunct Associate Professor, Computer Science Department
>>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > -----Original Message-----
>>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>>> > Reply-To: "architecture@airavata.apache.org"
>>>>> > <ar...@airavata.apache.org>
>>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>>> > To: "architecture@airavata.apache.org" <
>>>>> architecture@airavata.apache.org>,
>>>>> > dev <de...@airavata.apache.org>
>>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
>>>>>Airavata.
>>>>> >
>>>>> >> Hi All,
>>>>> >>
>>>>> >> As we all know Airavata has its own workflow language call XWF.
>>>>>When
>>>>> XWF
>>>>> >> was introduced, main focus points are interoperability and
>>>>> convertibility.
>>>>> >> But with years of experience it is convinced that above
>>>>>requirements
>>>>> are
>>>>> >> not really useful when we come to real world use cases. And XWF is
>>>>> XML
>>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>>> self.
>>>>> >> But
>>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>>> >>
>>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>>> industry
>>>>> >> and
>>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>>> come up
>>>>> >> with idea how we should improve Airavata workflow language, either
>>>>> we can
>>>>> >> improve existing XWF language, totally change to a new language
>>>>> available
>>>>> >> in industry or write a new light weight language. Basic
>>>>>requirements
>>>>> that
>>>>> >> we expect from new improvement are, high usability, flexible,
>>>>>light
>>>>> weight
>>>>> >> and real time monitoring support. As you can see above
>>>>>requirements
>>>>> are
>>>>> >> not
>>>>> >> direct comes with workflow languages but we need workflow language
>>>>> which
>>>>> >> help to support above requirements.
>>>>> >>
>>>>> >> After reading few papers and googling, initially i have come up
>>>>>with
>>>>> >> following three existing languages,
>>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>>> >> 2. WS-BPEL
>>>>> >> 3. SIDL
>>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>>> >>
>>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>>> Radical-SAGA
>>>>> >> also uses slightly modified version of SIDL. Other than above
>>>>>three
>>>>> >> languages we can come up with simple workflow language base on
>>>>> json(or
>>>>> >> yaml) which support all our requirements for some extends.
>>>>> >>
>>>>> >> It would be grate if I can get more input regarding the $Subject
>>>>> form the
>>>>> >> airavata community. You all are more than welcome to provide any
>>>>> type of
>>>>> >> suggestions.
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Shameera.
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Best Regards,
>>>>> >> Shameera Rathnayaka.
>>>>> >>
>>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> The best way to predict the future is to invent it - Alan Kay
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>>
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>
>
>--
>Supun Kamburugamuva
>Member, Apache Software Foundation; http://www.apache.org
>E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>Blog: http://supunk.blogspot.com
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Chris Mattmann <ch...@gmail.com>.
Have you guys considered using JCC [1] as a means
to expose the workflow API currently in Java as a
Python API?
We are exploring its use in OODT, and we have already
created a Tika [2] JCC-based python API.
Cheers,
Chris
[1] http://lucene.apache.org/pylucene/jcc/
[2] http://github.com/chrismattmann/tika-python/
------------------------
Chris Mattmann
chris.mattmann@gmail.com
-----Original Message-----
From: Supun Kamburugamuva <su...@gmail.com>
Reply-To: <de...@oodt.apache.org>
Date: Thursday, October 16, 2014 at 3:43 PM
To: dev <de...@airavata.apache.org>
Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
<sm...@apache.org>, "architecture@airavata.apache.org"
<ar...@airavata.apache.org>, "dev@oodt.apache.org"
<de...@oodt.apache.org>
Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
>Once we had an offline discussion about the Airavata Workflow language
>(with Milinda, Saliya and Shameera). In that discussion one thing came out
>was why we need to invent a different language when a simple library like
>Python will full fill of Airavata requirements.
>
>There are many benefits in using a Python library as the API for
>controlling Airavata workflows.
>
>1. It is a library, gives the ultimate control over the execution and it
>can be simpler than any domain specific language that we can come with
>like
>XML, JSON etc
>2. Most people use python and can learn it easily than any Domain specific
>language
>3. You can easily create a Python library for Airavata because all the
>APIs
>are thrift based.
>4. If you design the constructs correctly you can plug an XBaya.
>
>Any thoughts?
>
>Thanks,
>Supun..
>
>
>On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
>wrote:
>
>> Hi Shameera,
>>
>> Why you prefer JavaScript over a language like Python?
>>
>> Thanks,
>> Supun..
>>
>> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> First of all thanks everyone for giving valuable inputs. After doing
>>>some
>>> background search and talking to different people in the University
>>>who has
>>> used different workflow languages, I myself convinced that introducing
>>>an
>>> another workflow language is not what actually they need. By changing
>>> exiting workflow language to another will not solve problems. What they
>>> asking is a easy way to construct the workflows. Indirectly what they
>>> asking for a sort of API which they can use to generate the workflows
>>>and
>>> run it. Correct me if i am wrong here.
>>>
>>> As most of above replies depict, if we can get a simple API, as an
>>> example, for a web based application, JavaScript API would be a good
>>> solution, and probably JSON would be a good candidate for language,
>>>instead
>>> of XML.
>>>
>>> Airavata community already have started to implement web base GUI.
>>>Hence
>>> introducing a JSON base JavaScript API would be great help. WDYT?
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> alekny7@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> it is not dataflow instead focused on orchestrating REST services but
>>>> you may find it useful datapoint - we created worfklow service that
>>>>uses
>>>> natively JavaScript and JSON to describe what happens during workflow
>>>> execution:
>>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>>
>>>> HTH,
>>>>
>>>> Alek
>>>>
>>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
>>>>wrote:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>> Great to hear OODT community will be interested in adopting a JSON
>>>>> based workflow language and potentially a web based composer as well.
>>>>> Airavata previously had BPEL support initially through a home grown
>>>>> implementation [1] by Alek Slominski and later through Apache ODE
>>>>>[2]. Also
>>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>>
>>>>> I am of the same opinion that we should adopt something more modern
>>>>>as
>>>>> the challenges from scientific workflows seems to be converging with
>>>>>the
>>>>> data flow patterns in business workflows.
>>>>>
>>>>> It will be great if we can all compile a list of potential candidates
>>>>> and hack them through.
>>>>>
>>>>> Suresh
>>>>> [1] -
>>>>>
>>>>>http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>>> [2] -
>>>>>
>>>>>http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>>>>>time_for_eScience_workflows
>>>>> [3] -
>>>>>
>>>>>http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>>>>>6.pdf
>>>>>
>>>>>
>>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>>
>>>>> > Hi Guys,
>>>>> >
>>>>> > I've been interested in this too - we don't per have a specific
>>>>> > OODT workflow language, but we specific workflows using XML, and
>>>>> > other configuration (we are also thinking of moving to JSON for
>>>>> > this).
>>>>> >
>>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>>> > to me.
>>>>> >
>>>>> > I wonder at the end of the day if we should adopt something more
>>>>> > modern like PIG or some other data flow type of language (PIG
>>>>> > is really neat).
>>>>> >
>>>>> > Cheers,
>>>>> > Chris
>>>>> >
>>>>> >
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Chris Mattmann, Ph.D.
>>>>> > Chief Architect
>>>>> > Instrument Software and Science Data Systems Section (398)
>>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>> > Office: 168-519, Mailstop: 168-527
>>>>> > Email: chris.a.mattmann@nasa.gov
>>>>> > WWW: http://sunset.usc.edu/~mattmann/
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Adjunct Associate Professor, Computer Science Department
>>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > -----Original Message-----
>>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>>> > Reply-To: "architecture@airavata.apache.org"
>>>>> > <ar...@airavata.apache.org>
>>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>>> > To: "architecture@airavata.apache.org" <
>>>>> architecture@airavata.apache.org>,
>>>>> > dev <de...@airavata.apache.org>
>>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
>>>>>Airavata.
>>>>> >
>>>>> >> Hi All,
>>>>> >>
>>>>> >> As we all know Airavata has its own workflow language call XWF.
>>>>>When
>>>>> XWF
>>>>> >> was introduced, main focus points are interoperability and
>>>>> convertibility.
>>>>> >> But with years of experience it is convinced that above
>>>>>requirements
>>>>> are
>>>>> >> not really useful when we come to real world use cases. And XWF is
>>>>> XML
>>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>>> self.
>>>>> >> But
>>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>>> >>
>>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>>> industry
>>>>> >> and
>>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>>> come up
>>>>> >> with idea how we should improve Airavata workflow language, either
>>>>> we can
>>>>> >> improve existing XWF language, totally change to a new language
>>>>> available
>>>>> >> in industry or write a new light weight language. Basic
>>>>>requirements
>>>>> that
>>>>> >> we expect from new improvement are, high usability, flexible,
>>>>>light
>>>>> weight
>>>>> >> and real time monitoring support. As you can see above
>>>>>requirements
>>>>> are
>>>>> >> not
>>>>> >> direct comes with workflow languages but we need workflow language
>>>>> which
>>>>> >> help to support above requirements.
>>>>> >>
>>>>> >> After reading few papers and googling, initially i have come up
>>>>>with
>>>>> >> following three existing languages,
>>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>>> >> 2. WS-BPEL
>>>>> >> 3. SIDL
>>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>>> >>
>>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>>> Radical-SAGA
>>>>> >> also uses slightly modified version of SIDL. Other than above
>>>>>three
>>>>> >> languages we can come up with simple workflow language base on
>>>>> json(or
>>>>> >> yaml) which support all our requirements for some extends.
>>>>> >>
>>>>> >> It would be grate if I can get more input regarding the $Subject
>>>>> form the
>>>>> >> airavata community. You all are more than welcome to provide any
>>>>> type of
>>>>> >> suggestions.
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Shameera.
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Best Regards,
>>>>> >> Shameera Rathnayaka.
>>>>> >>
>>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> The best way to predict the future is to invent it - Alan Kay
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>>
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>
>
>--
>Supun Kamburugamuva
>Member, Apache Software Foundation; http://www.apache.org
>E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>Blog: http://supunk.blogspot.com
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Chris Mattmann <ch...@gmail.com>.
Have you guys considered using JCC [1] as a means
to expose the workflow API currently in Java as a
Python API?
We are exploring its use in OODT, and we have already
created a Tika [2] JCC-based python API.
Cheers,
Chris
[1] http://lucene.apache.org/pylucene/jcc/
[2] http://github.com/chrismattmann/tika-python/
------------------------
Chris Mattmann
chris.mattmann@gmail.com
-----Original Message-----
From: Supun Kamburugamuva <su...@gmail.com>
Reply-To: <de...@oodt.apache.org>
Date: Thursday, October 16, 2014 at 3:43 PM
To: dev <de...@airavata.apache.org>
Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
<sm...@apache.org>, "architecture@airavata.apache.org"
<ar...@airavata.apache.org>, "dev@oodt.apache.org"
<de...@oodt.apache.org>
Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
>Once we had an offline discussion about the Airavata Workflow language
>(with Milinda, Saliya and Shameera). In that discussion one thing came out
>was why we need to invent a different language when a simple library like
>Python will full fill of Airavata requirements.
>
>There are many benefits in using a Python library as the API for
>controlling Airavata workflows.
>
>1. It is a library, gives the ultimate control over the execution and it
>can be simpler than any domain specific language that we can come with
>like
>XML, JSON etc
>2. Most people use python and can learn it easily than any Domain specific
>language
>3. You can easily create a Python library for Airavata because all the
>APIs
>are thrift based.
>4. If you design the constructs correctly you can plug an XBaya.
>
>Any thoughts?
>
>Thanks,
>Supun..
>
>
>On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
>wrote:
>
>> Hi Shameera,
>>
>> Why you prefer JavaScript over a language like Python?
>>
>> Thanks,
>> Supun..
>>
>> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> First of all thanks everyone for giving valuable inputs. After doing
>>>some
>>> background search and talking to different people in the University
>>>who has
>>> used different workflow languages, I myself convinced that introducing
>>>an
>>> another workflow language is not what actually they need. By changing
>>> exiting workflow language to another will not solve problems. What they
>>> asking is a easy way to construct the workflows. Indirectly what they
>>> asking for a sort of API which they can use to generate the workflows
>>>and
>>> run it. Correct me if i am wrong here.
>>>
>>> As most of above replies depict, if we can get a simple API, as an
>>> example, for a web based application, JavaScript API would be a good
>>> solution, and probably JSON would be a good candidate for language,
>>>instead
>>> of XML.
>>>
>>> Airavata community already have started to implement web base GUI.
>>>Hence
>>> introducing a JSON base JavaScript API would be great help. WDYT?
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> alekny7@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> it is not dataflow instead focused on orchestrating REST services but
>>>> you may find it useful datapoint - we created worfklow service that
>>>>uses
>>>> natively JavaScript and JSON to describe what happens during workflow
>>>> execution:
>>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>>
>>>> HTH,
>>>>
>>>> Alek
>>>>
>>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
>>>>wrote:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>> Great to hear OODT community will be interested in adopting a JSON
>>>>> based workflow language and potentially a web based composer as well.
>>>>> Airavata previously had BPEL support initially through a home grown
>>>>> implementation [1] by Alek Slominski and later through Apache ODE
>>>>>[2]. Also
>>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>>
>>>>> I am of the same opinion that we should adopt something more modern
>>>>>as
>>>>> the challenges from scientific workflows seems to be converging with
>>>>>the
>>>>> data flow patterns in business workflows.
>>>>>
>>>>> It will be great if we can all compile a list of potential candidates
>>>>> and hack them through.
>>>>>
>>>>> Suresh
>>>>> [1] -
>>>>>
>>>>>http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>>> [2] -
>>>>>
>>>>>http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>>>>>time_for_eScience_workflows
>>>>> [3] -
>>>>>
>>>>>http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>>>>>6.pdf
>>>>>
>>>>>
>>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>>
>>>>> > Hi Guys,
>>>>> >
>>>>> > I've been interested in this too - we don't per have a specific
>>>>> > OODT workflow language, but we specific workflows using XML, and
>>>>> > other configuration (we are also thinking of moving to JSON for
>>>>> > this).
>>>>> >
>>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>>> > to me.
>>>>> >
>>>>> > I wonder at the end of the day if we should adopt something more
>>>>> > modern like PIG or some other data flow type of language (PIG
>>>>> > is really neat).
>>>>> >
>>>>> > Cheers,
>>>>> > Chris
>>>>> >
>>>>> >
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Chris Mattmann, Ph.D.
>>>>> > Chief Architect
>>>>> > Instrument Software and Science Data Systems Section (398)
>>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>> > Office: 168-519, Mailstop: 168-527
>>>>> > Email: chris.a.mattmann@nasa.gov
>>>>> > WWW: http://sunset.usc.edu/~mattmann/
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Adjunct Associate Professor, Computer Science Department
>>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > -----Original Message-----
>>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>>> > Reply-To: "architecture@airavata.apache.org"
>>>>> > <ar...@airavata.apache.org>
>>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>>> > To: "architecture@airavata.apache.org" <
>>>>> architecture@airavata.apache.org>,
>>>>> > dev <de...@airavata.apache.org>
>>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
>>>>>Airavata.
>>>>> >
>>>>> >> Hi All,
>>>>> >>
>>>>> >> As we all know Airavata has its own workflow language call XWF.
>>>>>When
>>>>> XWF
>>>>> >> was introduced, main focus points are interoperability and
>>>>> convertibility.
>>>>> >> But with years of experience it is convinced that above
>>>>>requirements
>>>>> are
>>>>> >> not really useful when we come to real world use cases. And XWF is
>>>>> XML
>>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>>> self.
>>>>> >> But
>>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>>> >>
>>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>>> industry
>>>>> >> and
>>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>>> come up
>>>>> >> with idea how we should improve Airavata workflow language, either
>>>>> we can
>>>>> >> improve existing XWF language, totally change to a new language
>>>>> available
>>>>> >> in industry or write a new light weight language. Basic
>>>>>requirements
>>>>> that
>>>>> >> we expect from new improvement are, high usability, flexible,
>>>>>light
>>>>> weight
>>>>> >> and real time monitoring support. As you can see above
>>>>>requirements
>>>>> are
>>>>> >> not
>>>>> >> direct comes with workflow languages but we need workflow language
>>>>> which
>>>>> >> help to support above requirements.
>>>>> >>
>>>>> >> After reading few papers and googling, initially i have come up
>>>>>with
>>>>> >> following three existing languages,
>>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>>> >> 2. WS-BPEL
>>>>> >> 3. SIDL
>>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>>> >>
>>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>>> Radical-SAGA
>>>>> >> also uses slightly modified version of SIDL. Other than above
>>>>>three
>>>>> >> languages we can come up with simple workflow language base on
>>>>> json(or
>>>>> >> yaml) which support all our requirements for some extends.
>>>>> >>
>>>>> >> It would be grate if I can get more input regarding the $Subject
>>>>> form the
>>>>> >> airavata community. You all are more than welcome to provide any
>>>>> type of
>>>>> >> suggestions.
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Shameera.
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Best Regards,
>>>>> >> Shameera Rathnayaka.
>>>>> >>
>>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> The best way to predict the future is to invent it - Alan Kay
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>>
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>
>
>--
>Supun Kamburugamuva
>Member, Apache Software Foundation; http://www.apache.org
>E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>Blog: http://supunk.blogspot.com
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Supun Kamburugamuva <su...@gmail.com>.
Once we had an offline discussion about the Airavata Workflow language
(with Milinda, Saliya and Shameera). In that discussion one thing came out
was why we need to invent a different language when a simple library like
Python will full fill of Airavata requirements.
There are many benefits in using a Python library as the API for
controlling Airavata workflows.
1. It is a library, gives the ultimate control over the execution and it
can be simpler than any domain specific language that we can come with like
XML, JSON etc
2. Most people use python and can learn it easily than any Domain specific
language
3. You can easily create a Python library for Airavata because all the APIs
are thrift based.
4. If you design the constructs correctly you can plug an XBaya.
Any thoughts?
Thanks,
Supun..
On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:
> Hi Shameera,
>
> Why you prefer JavaScript over a language like Python?
>
> Thanks,
> Supun..
>
> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi,
>>
>> First of all thanks everyone for giving valuable inputs. After doing some
>> background search and talking to different people in the University who has
>> used different workflow languages, I myself convinced that introducing an
>> another workflow language is not what actually they need. By changing
>> exiting workflow language to another will not solve problems. What they
>> asking is a easy way to construct the workflows. Indirectly what they
>> asking for a sort of API which they can use to generate the workflows and
>> run it. Correct me if i am wrong here.
>>
>> As most of above replies depict, if we can get a simple API, as an
>> example, for a web based application, JavaScript API would be a good
>> solution, and probably JSON would be a good candidate for language, instead
>> of XML.
>>
>> Airavata community already have started to implement web base GUI. Hence
>> introducing a JSON base JavaScript API would be great help. WDYT?
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> alekny7@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> it is not dataflow instead focused on orchestrating REST services but
>>> you may find it useful datapoint - we created worfklow service that uses
>>> natively JavaScript and JSON to describe what happens during workflow
>>> execution:
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>
>>> HTH,
>>>
>>> Alek
>>>
>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>>>
>>>> Hi Chris,
>>>>
>>>> Great to hear OODT community will be interested in adopting a JSON
>>>> based workflow language and potentially a web based composer as well.
>>>> Airavata previously had BPEL support initially through a home grown
>>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>
>>>> I am of the same opinion that we should adopt something more modern as
>>>> the challenges from scientific workflows seems to be converging with the
>>>> data flow patterns in business workflows.
>>>>
>>>> It will be great if we can all compile a list of potential candidates
>>>> and hack them through.
>>>>
>>>> Suresh
>>>> [1] -
>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>> [2] -
>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>>> [3] -
>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>>
>>>>
>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>
>>>> > Hi Guys,
>>>> >
>>>> > I've been interested in this too - we don't per have a specific
>>>> > OODT workflow language, but we specific workflows using XML, and
>>>> > other configuration (we are also thinking of moving to JSON for
>>>> > this).
>>>> >
>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>> > to me.
>>>> >
>>>> > I wonder at the end of the day if we should adopt something more
>>>> > modern like PIG or some other data flow type of language (PIG
>>>> > is really neat).
>>>> >
>>>> > Cheers,
>>>> > Chris
>>>> >
>>>> >
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Chris Mattmann, Ph.D.
>>>> > Chief Architect
>>>> > Instrument Software and Science Data Systems Section (398)
>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> > Office: 168-519, Mailstop: 168-527
>>>> > Email: chris.a.mattmann@nasa.gov
>>>> > WWW: http://sunset.usc.edu/~mattmann/
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Adjunct Associate Professor, Computer Science Department
>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > -----Original Message-----
>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>> > Reply-To: "architecture@airavata.apache.org"
>>>> > <ar...@airavata.apache.org>
>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>> > To: "architecture@airavata.apache.org" <
>>>> architecture@airavata.apache.org>,
>>>> > dev <de...@airavata.apache.org>
>>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>>> >
>>>> >> Hi All,
>>>> >>
>>>> >> As we all know Airavata has its own workflow language call XWF. When
>>>> XWF
>>>> >> was introduced, main focus points are interoperability and
>>>> convertibility.
>>>> >> But with years of experience it is convinced that above requirements
>>>> are
>>>> >> not really useful when we come to real world use cases. And XWF is
>>>> XML
>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>> self.
>>>> >> But
>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>> >>
>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>> industry
>>>> >> and
>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>> come up
>>>> >> with idea how we should improve Airavata workflow language, either
>>>> we can
>>>> >> improve existing XWF language, totally change to a new language
>>>> available
>>>> >> in industry or write a new light weight language. Basic requirements
>>>> that
>>>> >> we expect from new improvement are, high usability, flexible, light
>>>> weight
>>>> >> and real time monitoring support. As you can see above requirements
>>>> are
>>>> >> not
>>>> >> direct comes with workflow languages but we need workflow language
>>>> which
>>>> >> help to support above requirements.
>>>> >>
>>>> >> After reading few papers and googling, initially i have come up with
>>>> >> following three existing languages,
>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>> >> 2. WS-BPEL
>>>> >> 3. SIDL
>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>> >>
>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>> Radical-SAGA
>>>> >> also uses slightly modified version of SIDL. Other than above three
>>>> >> languages we can come up with simple workflow language base on
>>>> json(or
>>>> >> yaml) which support all our requirements for some extends.
>>>> >>
>>>> >> It would be grate if I can get more input regarding the $Subject
>>>> form the
>>>> >> airavata community. You all are more than welcome to provide any
>>>> type of
>>>> >> suggestions.
>>>> >>
>>>> >> Thanks,
>>>> >> Shameera.
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Best Regards,
>>>> >> Shameera Rathnayaka.
>>>> >>
>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>
>>>>
>>>
>>>
>>> --
>>> The best way to predict the future is to invent it - Alan Kay
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by K Yoshimoto <ke...@sdsc.edu>.
I prefer python myself, but if the API is clean, it should be
usable from a large number of languages on the client side.
On Thu, Oct 16, 2014 at 06:46:36PM -0400, Shameera Rathnayaka wrote:
> Hi Supun,
>
> Considering we are going to provide web base GUI support and JavaScript is
> a well-known client side language, I would select JavaScript over Python. I
> am not much familiar with Python, so would like to know the advantages we
> get by selecting python here.
>
>
> Thanks,
> Shameera.
>
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
> > Hi Shameera,
> >
> > Why you prefer JavaScript over a language like Python?
> >
> > Thanks,
> > Supun..
> >
> > On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > shameerainfo@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> First of all thanks everyone for giving valuable inputs. After doing some
> >> background search and talking to different people in the University who has
> >> used different workflow languages, I myself convinced that introducing an
> >> another workflow language is not what actually they need. By changing
> >> exiting workflow language to another will not solve problems. What they
> >> asking is a easy way to construct the workflows. Indirectly what they
> >> asking for a sort of API which they can use to generate the workflows and
> >> run it. Correct me if i am wrong here.
> >>
> >> As most of above replies depict, if we can get a simple API, as an
> >> example, for a web based application, JavaScript API would be a good
> >> solution, and probably JSON would be a good candidate for language, instead
> >> of XML.
> >>
> >> Airavata community already have started to implement web base GUI. Hence
> >> introducing a JSON base JavaScript API would be great help. WDYT?
> >>
> >> Thanks,
> >> Shameera.
> >>
> >>
> >> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >> alekny7@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> it is not dataflow instead focused on orchestrating REST services but
> >>> you may find it useful datapoint - we created worfklow service that uses
> >>> natively JavaScript and JSON to describe what happens during workflow
> >>> execution:
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>
> >>> HTH,
> >>>
> >>> Alek
> >>>
> >>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
> >>>
> >>>> Hi Chris,
> >>>>
> >>>> Great to hear OODT community will be interested in adopting a JSON
> >>>> based workflow language and potentially a web based composer as well.
> >>>> Airavata previously had BPEL support initially through a home grown
> >>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
> >>>> a white paper [3] by Alek on this topic is an interesting read.
> >>>>
> >>>> I am of the same opinion that we should adopt something more modern as
> >>>> the challenges from scientific workflows seems to be converging with the
> >>>> data flow patterns in business workflows.
> >>>>
> >>>> It will be great if we can all compile a list of potential candidates
> >>>> and hack them through.
> >>>>
> >>>> Suresh
> >>>> [1] -
> >>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>>> [2] -
> >>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
> >>>> [3] -
> >>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
> >>>>
> >>>>
> >>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>>>
> >>>> > Hi Guys,
> >>>> >
> >>>> > I've been interested in this too - we don't per have a specific
> >>>> > OODT workflow language, but we specific workflows using XML, and
> >>>> > other configuration (we are also thinking of moving to JSON for
> >>>> > this).
> >>>> >
> >>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> >>>> > to me.
> >>>> >
> >>>> > I wonder at the end of the day if we should adopt something more
> >>>> > modern like PIG or some other data flow type of language (PIG
> >>>> > is really neat).
> >>>> >
> >>>> > Cheers,
> >>>> > Chris
> >>>> >
> >>>> >
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Chris Mattmann, Ph.D.
> >>>> > Chief Architect
> >>>> > Instrument Software and Science Data Systems Section (398)
> >>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>>> > Office: 168-519, Mailstop: 168-527
> >>>> > Email: chris.a.mattmann@nasa.gov
> >>>> > WWW: http://sunset.usc.edu/~mattmann/
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Adjunct Associate Professor, Computer Science Department
> >>>> > University of Southern California, Los Angeles, CA 90089 USA
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > -----Original Message-----
> >>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>>> > Reply-To: "architecture@airavata.apache.org"
> >>>> > <ar...@airavata.apache.org>
> >>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>>> > To: "architecture@airavata.apache.org" <
> >>>> architecture@airavata.apache.org>,
> >>>> > dev <de...@airavata.apache.org>
> >>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> >>>> >
> >>>> >> Hi All,
> >>>> >>
> >>>> >> As we all know Airavata has its own workflow language call XWF. When
> >>>> XWF
> >>>> >> was introduced, main focus points are interoperability and
> >>>> convertibility.
> >>>> >> But with years of experience it is convinced that above requirements
> >>>> are
> >>>> >> not really useful when we come to real world use cases. And XWF is
> >>>> XML
> >>>> >> based bulky language where we attache WSDLs and Workflow image it
> >>>> self.
> >>>> >> But
> >>>> >> with the recent changes WSDL part is being removed from XWF.
> >>>> >>
> >>>> >> It is worth to evaluate handy Scientific workflow languages in
> >>>> industry
> >>>> >> and
> >>>> >> find out pros and cons, at the end of this evaluation we need to
> >>>> come up
> >>>> >> with idea how we should improve Airavata workflow language, either
> >>>> we can
> >>>> >> improve existing XWF language, totally change to a new language
> >>>> available
> >>>> >> in industry or write a new light weight language. Basic requirements
> >>>> that
> >>>> >> we expect from new improvement are, high usability, flexible, light
> >>>> weight
> >>>> >> and real time monitoring support. As you can see above requirements
> >>>> are
> >>>> >> not
> >>>> >> direct comes with workflow languages but we need workflow language
> >>>> which
> >>>> >> help to support above requirements.
> >>>> >>
> >>>> >> After reading few papers and googling, initially i have come up with
> >>>> >> following three existing languages,
> >>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>>> >> 2. WS-BPEL
> >>>> >> 3. SIDL
> >>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
> >>>> >>
> >>>> >> In my opinion SIDL is more familiar with scientific domain,
> >>>> Radical-SAGA
> >>>> >> also uses slightly modified version of SIDL. Other than above three
> >>>> >> languages we can come up with simple workflow language base on
> >>>> json(or
> >>>> >> yaml) which support all our requirements for some extends.
> >>>> >>
> >>>> >> It would be grate if I can get more input regarding the $Subject
> >>>> form the
> >>>> >> airavata community. You all are more than welcome to provide any
> >>>> type of
> >>>> >> suggestions.
> >>>> >>
> >>>> >> Thanks,
> >>>> >> Shameera.
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> --
> >>>> >> Best Regards,
> >>>> >> Shameera Rathnayaka.
> >>>> >>
> >>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> The best way to predict the future is to invent it - Alan Kay
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Shameera Rathnayaka.
> >>
> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Nipurn Doshi <ni...@umail.iu.edu>.
Hi,
How about I suggest a third option where I have the current code setup
which has access to thrift classes, airavata interface classes and a
complete framework ( Laravel ) that supports rest apis to generate json
data that we are looking at?
I am suggesting this only because I have done this in previous projects I
have worked on using Laravel.
Although, it's in PHP.
On 16-Oct-2014 6:47 pm, "Shameera Rathnayaka" <sh...@gmail.com>
wrote:
> Hi Supun,
>
> Considering we are going to provide web base GUI support and JavaScript
> is a well-known client side language, I would select JavaScript over
> Python. I am not much familiar with Python, so would like to know the
> advantages we get by selecting python here.
>
>
> Thanks,
> Shameera.
>
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
>> Hi Shameera,
>>
>> Why you prefer JavaScript over a language like Python?
>>
>> Thanks,
>> Supun..
>>
>> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> First of all thanks everyone for giving valuable inputs. After doing
>>> some background search and talking to different people in the University
>>> who has used different workflow languages, I myself convinced that
>>> introducing an another workflow language is not what actually they need. By
>>> changing exiting workflow language to another will not solve problems. What
>>> they asking is a easy way to construct the workflows. Indirectly what they
>>> asking for a sort of API which they can use to generate the workflows and
>>> run it. Correct me if i am wrong here.
>>>
>>> As most of above replies depict, if we can get a simple API, as an
>>> example, for a web based application, JavaScript API would be a good
>>> solution, and probably JSON would be a good candidate for language, instead
>>> of XML.
>>>
>>> Airavata community already have started to implement web base GUI. Hence
>>> introducing a JSON base JavaScript API would be great help. WDYT?
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> alekny7@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> it is not dataflow instead focused on orchestrating REST services but
>>>> you may find it useful datapoint - we created worfklow service that uses
>>>> natively JavaScript and JSON to describe what happens during workflow
>>>> execution:
>>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>>
>>>> HTH,
>>>>
>>>> Alek
>>>>
>>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>> Great to hear OODT community will be interested in adopting a JSON
>>>>> based workflow language and potentially a web based composer as well.
>>>>> Airavata previously had BPEL support initially through a home grown
>>>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
>>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>>
>>>>> I am of the same opinion that we should adopt something more modern as
>>>>> the challenges from scientific workflows seems to be converging with the
>>>>> data flow patterns in business workflows.
>>>>>
>>>>> It will be great if we can all compile a list of potential candidates
>>>>> and hack them through.
>>>>>
>>>>> Suresh
>>>>> [1] -
>>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>>> [2] -
>>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>>>> [3] -
>>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>>>
>>>>>
>>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>>
>>>>> > Hi Guys,
>>>>> >
>>>>> > I've been interested in this too - we don't per have a specific
>>>>> > OODT workflow language, but we specific workflows using XML, and
>>>>> > other configuration (we are also thinking of moving to JSON for
>>>>> > this).
>>>>> >
>>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>>> > to me.
>>>>> >
>>>>> > I wonder at the end of the day if we should adopt something more
>>>>> > modern like PIG or some other data flow type of language (PIG
>>>>> > is really neat).
>>>>> >
>>>>> > Cheers,
>>>>> > Chris
>>>>> >
>>>>> >
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Chris Mattmann, Ph.D.
>>>>> > Chief Architect
>>>>> > Instrument Software and Science Data Systems Section (398)
>>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>> > Office: 168-519, Mailstop: 168-527
>>>>> > Email: chris.a.mattmann@nasa.gov
>>>>> > WWW: http://sunset.usc.edu/~mattmann/
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Adjunct Associate Professor, Computer Science Department
>>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > -----Original Message-----
>>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>>> > Reply-To: "architecture@airavata.apache.org"
>>>>> > <ar...@airavata.apache.org>
>>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>>> > To: "architecture@airavata.apache.org" <
>>>>> architecture@airavata.apache.org>,
>>>>> > dev <de...@airavata.apache.org>
>>>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>>>> >
>>>>> >> Hi All,
>>>>> >>
>>>>> >> As we all know Airavata has its own workflow language call XWF.
>>>>> When XWF
>>>>> >> was introduced, main focus points are interoperability and
>>>>> convertibility.
>>>>> >> But with years of experience it is convinced that above
>>>>> requirements are
>>>>> >> not really useful when we come to real world use cases. And XWF is
>>>>> XML
>>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>>> self.
>>>>> >> But
>>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>>> >>
>>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>>> industry
>>>>> >> and
>>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>>> come up
>>>>> >> with idea how we should improve Airavata workflow language, either
>>>>> we can
>>>>> >> improve existing XWF language, totally change to a new language
>>>>> available
>>>>> >> in industry or write a new light weight language. Basic
>>>>> requirements that
>>>>> >> we expect from new improvement are, high usability, flexible, light
>>>>> weight
>>>>> >> and real time monitoring support. As you can see above requirements
>>>>> are
>>>>> >> not
>>>>> >> direct comes with workflow languages but we need workflow language
>>>>> which
>>>>> >> help to support above requirements.
>>>>> >>
>>>>> >> After reading few papers and googling, initially i have come up with
>>>>> >> following three existing languages,
>>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>>> >> 2. WS-BPEL
>>>>> >> 3. SIDL
>>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>>> >>
>>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>>> Radical-SAGA
>>>>> >> also uses slightly modified version of SIDL. Other than above three
>>>>> >> languages we can come up with simple workflow language base on
>>>>> json(or
>>>>> >> yaml) which support all our requirements for some extends.
>>>>> >>
>>>>> >> It would be grate if I can get more input regarding the $Subject
>>>>> form the
>>>>> >> airavata community. You all are more than welcome to provide any
>>>>> type of
>>>>> >> suggestions.
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Shameera.
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Best Regards,
>>>>> >> Shameera Rathnayaka.
>>>>> >>
>>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> The best way to predict the future is to invent it - Alan Kay
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>>
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by K Yoshimoto <ke...@sdsc.edu>.
I prefer python myself, but if the API is clean, it should be
usable from a large number of languages on the client side.
On Thu, Oct 16, 2014 at 06:46:36PM -0400, Shameera Rathnayaka wrote:
> Hi Supun,
>
> Considering we are going to provide web base GUI support and JavaScript is
> a well-known client side language, I would select JavaScript over Python. I
> am not much familiar with Python, so would like to know the advantages we
> get by selecting python here.
>
>
> Thanks,
> Shameera.
>
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
> > Hi Shameera,
> >
> > Why you prefer JavaScript over a language like Python?
> >
> > Thanks,
> > Supun..
> >
> > On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > shameerainfo@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> First of all thanks everyone for giving valuable inputs. After doing some
> >> background search and talking to different people in the University who has
> >> used different workflow languages, I myself convinced that introducing an
> >> another workflow language is not what actually they need. By changing
> >> exiting workflow language to another will not solve problems. What they
> >> asking is a easy way to construct the workflows. Indirectly what they
> >> asking for a sort of API which they can use to generate the workflows and
> >> run it. Correct me if i am wrong here.
> >>
> >> As most of above replies depict, if we can get a simple API, as an
> >> example, for a web based application, JavaScript API would be a good
> >> solution, and probably JSON would be a good candidate for language, instead
> >> of XML.
> >>
> >> Airavata community already have started to implement web base GUI. Hence
> >> introducing a JSON base JavaScript API would be great help. WDYT?
> >>
> >> Thanks,
> >> Shameera.
> >>
> >>
> >> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >> alekny7@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> it is not dataflow instead focused on orchestrating REST services but
> >>> you may find it useful datapoint - we created worfklow service that uses
> >>> natively JavaScript and JSON to describe what happens during workflow
> >>> execution:
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>
> >>> HTH,
> >>>
> >>> Alek
> >>>
> >>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
> >>>
> >>>> Hi Chris,
> >>>>
> >>>> Great to hear OODT community will be interested in adopting a JSON
> >>>> based workflow language and potentially a web based composer as well.
> >>>> Airavata previously had BPEL support initially through a home grown
> >>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
> >>>> a white paper [3] by Alek on this topic is an interesting read.
> >>>>
> >>>> I am of the same opinion that we should adopt something more modern as
> >>>> the challenges from scientific workflows seems to be converging with the
> >>>> data flow patterns in business workflows.
> >>>>
> >>>> It will be great if we can all compile a list of potential candidates
> >>>> and hack them through.
> >>>>
> >>>> Suresh
> >>>> [1] -
> >>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>>> [2] -
> >>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
> >>>> [3] -
> >>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
> >>>>
> >>>>
> >>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>>>
> >>>> > Hi Guys,
> >>>> >
> >>>> > I've been interested in this too - we don't per have a specific
> >>>> > OODT workflow language, but we specific workflows using XML, and
> >>>> > other configuration (we are also thinking of moving to JSON for
> >>>> > this).
> >>>> >
> >>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> >>>> > to me.
> >>>> >
> >>>> > I wonder at the end of the day if we should adopt something more
> >>>> > modern like PIG or some other data flow type of language (PIG
> >>>> > is really neat).
> >>>> >
> >>>> > Cheers,
> >>>> > Chris
> >>>> >
> >>>> >
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Chris Mattmann, Ph.D.
> >>>> > Chief Architect
> >>>> > Instrument Software and Science Data Systems Section (398)
> >>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>>> > Office: 168-519, Mailstop: 168-527
> >>>> > Email: chris.a.mattmann@nasa.gov
> >>>> > WWW: http://sunset.usc.edu/~mattmann/
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Adjunct Associate Professor, Computer Science Department
> >>>> > University of Southern California, Los Angeles, CA 90089 USA
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > -----Original Message-----
> >>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>>> > Reply-To: "architecture@airavata.apache.org"
> >>>> > <ar...@airavata.apache.org>
> >>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>>> > To: "architecture@airavata.apache.org" <
> >>>> architecture@airavata.apache.org>,
> >>>> > dev <de...@airavata.apache.org>
> >>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> >>>> >
> >>>> >> Hi All,
> >>>> >>
> >>>> >> As we all know Airavata has its own workflow language call XWF. When
> >>>> XWF
> >>>> >> was introduced, main focus points are interoperability and
> >>>> convertibility.
> >>>> >> But with years of experience it is convinced that above requirements
> >>>> are
> >>>> >> not really useful when we come to real world use cases. And XWF is
> >>>> XML
> >>>> >> based bulky language where we attache WSDLs and Workflow image it
> >>>> self.
> >>>> >> But
> >>>> >> with the recent changes WSDL part is being removed from XWF.
> >>>> >>
> >>>> >> It is worth to evaluate handy Scientific workflow languages in
> >>>> industry
> >>>> >> and
> >>>> >> find out pros and cons, at the end of this evaluation we need to
> >>>> come up
> >>>> >> with idea how we should improve Airavata workflow language, either
> >>>> we can
> >>>> >> improve existing XWF language, totally change to a new language
> >>>> available
> >>>> >> in industry or write a new light weight language. Basic requirements
> >>>> that
> >>>> >> we expect from new improvement are, high usability, flexible, light
> >>>> weight
> >>>> >> and real time monitoring support. As you can see above requirements
> >>>> are
> >>>> >> not
> >>>> >> direct comes with workflow languages but we need workflow language
> >>>> which
> >>>> >> help to support above requirements.
> >>>> >>
> >>>> >> After reading few papers and googling, initially i have come up with
> >>>> >> following three existing languages,
> >>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>>> >> 2. WS-BPEL
> >>>> >> 3. SIDL
> >>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
> >>>> >>
> >>>> >> In my opinion SIDL is more familiar with scientific domain,
> >>>> Radical-SAGA
> >>>> >> also uses slightly modified version of SIDL. Other than above three
> >>>> >> languages we can come up with simple workflow language base on
> >>>> json(or
> >>>> >> yaml) which support all our requirements for some extends.
> >>>> >>
> >>>> >> It would be grate if I can get more input regarding the $Subject
> >>>> form the
> >>>> >> airavata community. You all are more than welcome to provide any
> >>>> type of
> >>>> >> suggestions.
> >>>> >>
> >>>> >> Thanks,
> >>>> >> Shameera.
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> --
> >>>> >> Best Regards,
> >>>> >> Shameera Rathnayaka.
> >>>> >>
> >>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> The best way to predict the future is to invent it - Alan Kay
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Shameera Rathnayaka.
> >>
> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Supun Kamburugamuva <su...@gmail.com>.
Hi Shameera,
To create a web based UI I don't see an immediate advantage of using
JavaScript over a Python API. Because Airavata's API is Thrift (not any
language based API) you can generate code for any language (including
JavaScript). Anyway most probably you will not use direct Airavata API
calls for creating a web UI. I'm also not sure what you exactly mean by
JavaScript language as well. If you explain in more detail that would be
really helpful.
If you just give a Python API to create workflows, you really don't even
need XBaya to create/run workflows. You can create them as scripts and run
them from command line. You can debug your workflows written in Python
using normal python debuggers. You can generate the XBaya workflow by using
the structure you create in Python as well. Then you can use XBaya to
monitor the workflow. These things are hard to do with an JavaScript API.
As a language also I would pick Python over JavaScript.
Thanks,
Supun..
On Thu, Oct 16, 2014 at 6:46 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:
> Hi Supun,
>
> Considering we are going to provide web base GUI support and JavaScript is
> a well-known client side language, I would select JavaScript over Python. I
> am not much familiar with Python, so would like to know the advantages we
> get by selecting python here.
>
>
> Thanks,
> Shameera.
>
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
> > Hi Shameera,
> >
> > Why you prefer JavaScript over a language like Python?
> >
> > Thanks,
> > Supun..
> >
> > On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > shameerainfo@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> First of all thanks everyone for giving valuable inputs. After doing
> some
> >> background search and talking to different people in the University who
> has
> >> used different workflow languages, I myself convinced that introducing
> an
> >> another workflow language is not what actually they need. By changing
> >> exiting workflow language to another will not solve problems. What they
> >> asking is a easy way to construct the workflows. Indirectly what they
> >> asking for a sort of API which they can use to generate the workflows
> and
> >> run it. Correct me if i am wrong here.
> >>
> >> As most of above replies depict, if we can get a simple API, as an
> >> example, for a web based application, JavaScript API would be a good
> >> solution, and probably JSON would be a good candidate for language,
> instead
> >> of XML.
> >>
> >> Airavata community already have started to implement web base GUI. Hence
> >> introducing a JSON base JavaScript API would be great help. WDYT?
> >>
> >> Thanks,
> >> Shameera.
> >>
> >>
> >> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >> alekny7@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> it is not dataflow instead focused on orchestrating REST services but
> >>> you may find it useful datapoint - we created worfklow service that
> uses
> >>> natively JavaScript and JSON to describe what happens during workflow
> >>> execution:
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>
> >>> HTH,
> >>>
> >>> Alek
> >>>
> >>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> wrote:
> >>>
> >>>> Hi Chris,
> >>>>
> >>>> Great to hear OODT community will be interested in adopting a JSON
> >>>> based workflow language and potentially a web based composer as well.
> >>>> Airavata previously had BPEL support initially through a home grown
> >>>> implementation [1] by Alek Slominski and later through Apache ODE
> [2]. Also
> >>>> a white paper [3] by Alek on this topic is an interesting read.
> >>>>
> >>>> I am of the same opinion that we should adopt something more modern as
> >>>> the challenges from scientific workflows seems to be converging with
> the
> >>>> data flow patterns in business workflows.
> >>>>
> >>>> It will be great if we can all compile a list of potential candidates
> >>>> and hack them through.
> >>>>
> >>>> Suresh
> >>>> [1] -
> >>>>
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>>> [2] -
> >>>>
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
> >>>> [3] -
> >>>>
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
> >>>>
> >>>>
> >>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>>>
> >>>> > Hi Guys,
> >>>> >
> >>>> > I've been interested in this too - we don't per have a specific
> >>>> > OODT workflow language, but we specific workflows using XML, and
> >>>> > other configuration (we are also thinking of moving to JSON for
> >>>> > this).
> >>>> >
> >>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> >>>> > to me.
> >>>> >
> >>>> > I wonder at the end of the day if we should adopt something more
> >>>> > modern like PIG or some other data flow type of language (PIG
> >>>> > is really neat).
> >>>> >
> >>>> > Cheers,
> >>>> > Chris
> >>>> >
> >>>> >
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Chris Mattmann, Ph.D.
> >>>> > Chief Architect
> >>>> > Instrument Software and Science Data Systems Section (398)
> >>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>>> > Office: 168-519, Mailstop: 168-527
> >>>> > Email: chris.a.mattmann@nasa.gov
> >>>> > WWW: http://sunset.usc.edu/~mattmann/
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Adjunct Associate Professor, Computer Science Department
> >>>> > University of Southern California, Los Angeles, CA 90089 USA
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > -----Original Message-----
> >>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>>> > Reply-To: "architecture@airavata.apache.org"
> >>>> > <ar...@airavata.apache.org>
> >>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>>> > To: "architecture@airavata.apache.org" <
> >>>> architecture@airavata.apache.org>,
> >>>> > dev <de...@airavata.apache.org>
> >>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> Airavata.
> >>>> >
> >>>> >> Hi All,
> >>>> >>
> >>>> >> As we all know Airavata has its own workflow language call XWF.
> When
> >>>> XWF
> >>>> >> was introduced, main focus points are interoperability and
> >>>> convertibility.
> >>>> >> But with years of experience it is convinced that above
> requirements
> >>>> are
> >>>> >> not really useful when we come to real world use cases. And XWF is
> >>>> XML
> >>>> >> based bulky language where we attache WSDLs and Workflow image it
> >>>> self.
> >>>> >> But
> >>>> >> with the recent changes WSDL part is being removed from XWF.
> >>>> >>
> >>>> >> It is worth to evaluate handy Scientific workflow languages in
> >>>> industry
> >>>> >> and
> >>>> >> find out pros and cons, at the end of this evaluation we need to
> >>>> come up
> >>>> >> with idea how we should improve Airavata workflow language, either
> >>>> we can
> >>>> >> improve existing XWF language, totally change to a new language
> >>>> available
> >>>> >> in industry or write a new light weight language. Basic
> requirements
> >>>> that
> >>>> >> we expect from new improvement are, high usability, flexible, light
> >>>> weight
> >>>> >> and real time monitoring support. As you can see above requirements
> >>>> are
> >>>> >> not
> >>>> >> direct comes with workflow languages but we need workflow language
> >>>> which
> >>>> >> help to support above requirements.
> >>>> >>
> >>>> >> After reading few papers and googling, initially i have come up
> with
> >>>> >> following three existing languages,
> >>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>>> >> 2. WS-BPEL
> >>>> >> 3. SIDL
> >>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
> >>>> >>
> >>>> >> In my opinion SIDL is more familiar with scientific domain,
> >>>> Radical-SAGA
> >>>> >> also uses slightly modified version of SIDL. Other than above three
> >>>> >> languages we can come up with simple workflow language base on
> >>>> json(or
> >>>> >> yaml) which support all our requirements for some extends.
> >>>> >>
> >>>> >> It would be grate if I can get more input regarding the $Subject
> >>>> form the
> >>>> >> airavata community. You all are more than welcome to provide any
> >>>> type of
> >>>> >> suggestions.
> >>>> >>
> >>>> >> Thanks,
> >>>> >> Shameera.
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> --
> >>>> >> Best Regards,
> >>>> >> Shameera Rathnayaka.
> >>>> >>
> >>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> The best way to predict the future is to invent it - Alan Kay
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Shameera Rathnayaka.
> >>
> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Supun Kamburugamuva <su...@gmail.com>.
Hi Shameera,
To create a web based UI I don't see an immediate advantage of using
JavaScript over a Python API. Because Airavata's API is Thrift (not any
language based API) you can generate code for any language (including
JavaScript). Anyway most probably you will not use direct Airavata API
calls for creating a web UI. I'm also not sure what you exactly mean by
JavaScript language as well. If you explain in more detail that would be
really helpful.
If you just give a Python API to create workflows, you really don't even
need XBaya to create/run workflows. You can create them as scripts and run
them from command line. You can debug your workflows written in Python
using normal python debuggers. You can generate the XBaya workflow by using
the structure you create in Python as well. Then you can use XBaya to
monitor the workflow. These things are hard to do with an JavaScript API.
As a language also I would pick Python over JavaScript.
Thanks,
Supun..
On Thu, Oct 16, 2014 at 6:46 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:
> Hi Supun,
>
> Considering we are going to provide web base GUI support and JavaScript is
> a well-known client side language, I would select JavaScript over Python. I
> am not much familiar with Python, so would like to know the advantages we
> get by selecting python here.
>
>
> Thanks,
> Shameera.
>
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
> > Hi Shameera,
> >
> > Why you prefer JavaScript over a language like Python?
> >
> > Thanks,
> > Supun..
> >
> > On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > shameerainfo@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> First of all thanks everyone for giving valuable inputs. After doing
> some
> >> background search and talking to different people in the University who
> has
> >> used different workflow languages, I myself convinced that introducing
> an
> >> another workflow language is not what actually they need. By changing
> >> exiting workflow language to another will not solve problems. What they
> >> asking is a easy way to construct the workflows. Indirectly what they
> >> asking for a sort of API which they can use to generate the workflows
> and
> >> run it. Correct me if i am wrong here.
> >>
> >> As most of above replies depict, if we can get a simple API, as an
> >> example, for a web based application, JavaScript API would be a good
> >> solution, and probably JSON would be a good candidate for language,
> instead
> >> of XML.
> >>
> >> Airavata community already have started to implement web base GUI. Hence
> >> introducing a JSON base JavaScript API would be great help. WDYT?
> >>
> >> Thanks,
> >> Shameera.
> >>
> >>
> >> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >> alekny7@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> it is not dataflow instead focused on orchestrating REST services but
> >>> you may find it useful datapoint - we created worfklow service that
> uses
> >>> natively JavaScript and JSON to describe what happens during workflow
> >>> execution:
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>
> >>> HTH,
> >>>
> >>> Alek
> >>>
> >>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> wrote:
> >>>
> >>>> Hi Chris,
> >>>>
> >>>> Great to hear OODT community will be interested in adopting a JSON
> >>>> based workflow language and potentially a web based composer as well.
> >>>> Airavata previously had BPEL support initially through a home grown
> >>>> implementation [1] by Alek Slominski and later through Apache ODE
> [2]. Also
> >>>> a white paper [3] by Alek on this topic is an interesting read.
> >>>>
> >>>> I am of the same opinion that we should adopt something more modern as
> >>>> the challenges from scientific workflows seems to be converging with
> the
> >>>> data flow patterns in business workflows.
> >>>>
> >>>> It will be great if we can all compile a list of potential candidates
> >>>> and hack them through.
> >>>>
> >>>> Suresh
> >>>> [1] -
> >>>>
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>>> [2] -
> >>>>
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
> >>>> [3] -
> >>>>
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
> >>>>
> >>>>
> >>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>>>
> >>>> > Hi Guys,
> >>>> >
> >>>> > I've been interested in this too - we don't per have a specific
> >>>> > OODT workflow language, but we specific workflows using XML, and
> >>>> > other configuration (we are also thinking of moving to JSON for
> >>>> > this).
> >>>> >
> >>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> >>>> > to me.
> >>>> >
> >>>> > I wonder at the end of the day if we should adopt something more
> >>>> > modern like PIG or some other data flow type of language (PIG
> >>>> > is really neat).
> >>>> >
> >>>> > Cheers,
> >>>> > Chris
> >>>> >
> >>>> >
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Chris Mattmann, Ph.D.
> >>>> > Chief Architect
> >>>> > Instrument Software and Science Data Systems Section (398)
> >>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>>> > Office: 168-519, Mailstop: 168-527
> >>>> > Email: chris.a.mattmann@nasa.gov
> >>>> > WWW: http://sunset.usc.edu/~mattmann/
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Adjunct Associate Professor, Computer Science Department
> >>>> > University of Southern California, Los Angeles, CA 90089 USA
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > -----Original Message-----
> >>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>>> > Reply-To: "architecture@airavata.apache.org"
> >>>> > <ar...@airavata.apache.org>
> >>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>>> > To: "architecture@airavata.apache.org" <
> >>>> architecture@airavata.apache.org>,
> >>>> > dev <de...@airavata.apache.org>
> >>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> Airavata.
> >>>> >
> >>>> >> Hi All,
> >>>> >>
> >>>> >> As we all know Airavata has its own workflow language call XWF.
> When
> >>>> XWF
> >>>> >> was introduced, main focus points are interoperability and
> >>>> convertibility.
> >>>> >> But with years of experience it is convinced that above
> requirements
> >>>> are
> >>>> >> not really useful when we come to real world use cases. And XWF is
> >>>> XML
> >>>> >> based bulky language where we attache WSDLs and Workflow image it
> >>>> self.
> >>>> >> But
> >>>> >> with the recent changes WSDL part is being removed from XWF.
> >>>> >>
> >>>> >> It is worth to evaluate handy Scientific workflow languages in
> >>>> industry
> >>>> >> and
> >>>> >> find out pros and cons, at the end of this evaluation we need to
> >>>> come up
> >>>> >> with idea how we should improve Airavata workflow language, either
> >>>> we can
> >>>> >> improve existing XWF language, totally change to a new language
> >>>> available
> >>>> >> in industry or write a new light weight language. Basic
> requirements
> >>>> that
> >>>> >> we expect from new improvement are, high usability, flexible, light
> >>>> weight
> >>>> >> and real time monitoring support. As you can see above requirements
> >>>> are
> >>>> >> not
> >>>> >> direct comes with workflow languages but we need workflow language
> >>>> which
> >>>> >> help to support above requirements.
> >>>> >>
> >>>> >> After reading few papers and googling, initially i have come up
> with
> >>>> >> following three existing languages,
> >>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>>> >> 2. WS-BPEL
> >>>> >> 3. SIDL
> >>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
> >>>> >>
> >>>> >> In my opinion SIDL is more familiar with scientific domain,
> >>>> Radical-SAGA
> >>>> >> also uses slightly modified version of SIDL. Other than above three
> >>>> >> languages we can come up with simple workflow language base on
> >>>> json(or
> >>>> >> yaml) which support all our requirements for some extends.
> >>>> >>
> >>>> >> It would be grate if I can get more input regarding the $Subject
> >>>> form the
> >>>> >> airavata community. You all are more than welcome to provide any
> >>>> type of
> >>>> >> suggestions.
> >>>> >>
> >>>> >> Thanks,
> >>>> >> Shameera.
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> --
> >>>> >> Best Regards,
> >>>> >> Shameera Rathnayaka.
> >>>> >>
> >>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> The best way to predict the future is to invent it - Alan Kay
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Shameera Rathnayaka.
> >>
> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Nipurn Doshi <ni...@umail.iu.edu>.
Hi,
How about I suggest a third option where I have the current code setup
which has access to thrift classes, airavata interface classes and a
complete framework ( Laravel ) that supports rest apis to generate json
data that we are looking at?
I am suggesting this only because I have done this in previous projects I
have worked on using Laravel.
Although, it's in PHP.
On 16-Oct-2014 6:47 pm, "Shameera Rathnayaka" <sh...@gmail.com>
wrote:
> Hi Supun,
>
> Considering we are going to provide web base GUI support and JavaScript
> is a well-known client side language, I would select JavaScript over
> Python. I am not much familiar with Python, so would like to know the
> advantages we get by selecting python here.
>
>
> Thanks,
> Shameera.
>
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
>> Hi Shameera,
>>
>> Why you prefer JavaScript over a language like Python?
>>
>> Thanks,
>> Supun..
>>
>> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> First of all thanks everyone for giving valuable inputs. After doing
>>> some background search and talking to different people in the University
>>> who has used different workflow languages, I myself convinced that
>>> introducing an another workflow language is not what actually they need. By
>>> changing exiting workflow language to another will not solve problems. What
>>> they asking is a easy way to construct the workflows. Indirectly what they
>>> asking for a sort of API which they can use to generate the workflows and
>>> run it. Correct me if i am wrong here.
>>>
>>> As most of above replies depict, if we can get a simple API, as an
>>> example, for a web based application, JavaScript API would be a good
>>> solution, and probably JSON would be a good candidate for language, instead
>>> of XML.
>>>
>>> Airavata community already have started to implement web base GUI. Hence
>>> introducing a JSON base JavaScript API would be great help. WDYT?
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> alekny7@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> it is not dataflow instead focused on orchestrating REST services but
>>>> you may find it useful datapoint - we created worfklow service that uses
>>>> natively JavaScript and JSON to describe what happens during workflow
>>>> execution:
>>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>>
>>>> HTH,
>>>>
>>>> Alek
>>>>
>>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>> Great to hear OODT community will be interested in adopting a JSON
>>>>> based workflow language and potentially a web based composer as well.
>>>>> Airavata previously had BPEL support initially through a home grown
>>>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
>>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>>
>>>>> I am of the same opinion that we should adopt something more modern as
>>>>> the challenges from scientific workflows seems to be converging with the
>>>>> data flow patterns in business workflows.
>>>>>
>>>>> It will be great if we can all compile a list of potential candidates
>>>>> and hack them through.
>>>>>
>>>>> Suresh
>>>>> [1] -
>>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>>> [2] -
>>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>>>> [3] -
>>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>>>
>>>>>
>>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>>
>>>>> > Hi Guys,
>>>>> >
>>>>> > I've been interested in this too - we don't per have a specific
>>>>> > OODT workflow language, but we specific workflows using XML, and
>>>>> > other configuration (we are also thinking of moving to JSON for
>>>>> > this).
>>>>> >
>>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>>> > to me.
>>>>> >
>>>>> > I wonder at the end of the day if we should adopt something more
>>>>> > modern like PIG or some other data flow type of language (PIG
>>>>> > is really neat).
>>>>> >
>>>>> > Cheers,
>>>>> > Chris
>>>>> >
>>>>> >
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Chris Mattmann, Ph.D.
>>>>> > Chief Architect
>>>>> > Instrument Software and Science Data Systems Section (398)
>>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>> > Office: 168-519, Mailstop: 168-527
>>>>> > Email: chris.a.mattmann@nasa.gov
>>>>> > WWW: http://sunset.usc.edu/~mattmann/
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Adjunct Associate Professor, Computer Science Department
>>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > -----Original Message-----
>>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>>> > Reply-To: "architecture@airavata.apache.org"
>>>>> > <ar...@airavata.apache.org>
>>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>>> > To: "architecture@airavata.apache.org" <
>>>>> architecture@airavata.apache.org>,
>>>>> > dev <de...@airavata.apache.org>
>>>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>>>> >
>>>>> >> Hi All,
>>>>> >>
>>>>> >> As we all know Airavata has its own workflow language call XWF.
>>>>> When XWF
>>>>> >> was introduced, main focus points are interoperability and
>>>>> convertibility.
>>>>> >> But with years of experience it is convinced that above
>>>>> requirements are
>>>>> >> not really useful when we come to real world use cases. And XWF is
>>>>> XML
>>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>>> self.
>>>>> >> But
>>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>>> >>
>>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>>> industry
>>>>> >> and
>>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>>> come up
>>>>> >> with idea how we should improve Airavata workflow language, either
>>>>> we can
>>>>> >> improve existing XWF language, totally change to a new language
>>>>> available
>>>>> >> in industry or write a new light weight language. Basic
>>>>> requirements that
>>>>> >> we expect from new improvement are, high usability, flexible, light
>>>>> weight
>>>>> >> and real time monitoring support. As you can see above requirements
>>>>> are
>>>>> >> not
>>>>> >> direct comes with workflow languages but we need workflow language
>>>>> which
>>>>> >> help to support above requirements.
>>>>> >>
>>>>> >> After reading few papers and googling, initially i have come up with
>>>>> >> following three existing languages,
>>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>>> >> 2. WS-BPEL
>>>>> >> 3. SIDL
>>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>>> >>
>>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>>> Radical-SAGA
>>>>> >> also uses slightly modified version of SIDL. Other than above three
>>>>> >> languages we can come up with simple workflow language base on
>>>>> json(or
>>>>> >> yaml) which support all our requirements for some extends.
>>>>> >>
>>>>> >> It would be grate if I can get more input regarding the $Subject
>>>>> form the
>>>>> >> airavata community. You all are more than welcome to provide any
>>>>> type of
>>>>> >> suggestions.
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Shameera.
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Best Regards,
>>>>> >> Shameera Rathnayaka.
>>>>> >>
>>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> The best way to predict the future is to invent it - Alan Kay
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>>
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by K Yoshimoto <ke...@sdsc.edu>.
I prefer python myself, but if the API is clean, it should be
usable from a large number of languages on the client side.
On Thu, Oct 16, 2014 at 06:46:36PM -0400, Shameera Rathnayaka wrote:
> Hi Supun,
>
> Considering we are going to provide web base GUI support and JavaScript is
> a well-known client side language, I would select JavaScript over Python. I
> am not much familiar with Python, so would like to know the advantages we
> get by selecting python here.
>
>
> Thanks,
> Shameera.
>
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
> > Hi Shameera,
> >
> > Why you prefer JavaScript over a language like Python?
> >
> > Thanks,
> > Supun..
> >
> > On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > shameerainfo@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> First of all thanks everyone for giving valuable inputs. After doing some
> >> background search and talking to different people in the University who has
> >> used different workflow languages, I myself convinced that introducing an
> >> another workflow language is not what actually they need. By changing
> >> exiting workflow language to another will not solve problems. What they
> >> asking is a easy way to construct the workflows. Indirectly what they
> >> asking for a sort of API which they can use to generate the workflows and
> >> run it. Correct me if i am wrong here.
> >>
> >> As most of above replies depict, if we can get a simple API, as an
> >> example, for a web based application, JavaScript API would be a good
> >> solution, and probably JSON would be a good candidate for language, instead
> >> of XML.
> >>
> >> Airavata community already have started to implement web base GUI. Hence
> >> introducing a JSON base JavaScript API would be great help. WDYT?
> >>
> >> Thanks,
> >> Shameera.
> >>
> >>
> >> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >> alekny7@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> it is not dataflow instead focused on orchestrating REST services but
> >>> you may find it useful datapoint - we created worfklow service that uses
> >>> natively JavaScript and JSON to describe what happens during workflow
> >>> execution:
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>
> >>> HTH,
> >>>
> >>> Alek
> >>>
> >>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
> >>>
> >>>> Hi Chris,
> >>>>
> >>>> Great to hear OODT community will be interested in adopting a JSON
> >>>> based workflow language and potentially a web based composer as well.
> >>>> Airavata previously had BPEL support initially through a home grown
> >>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
> >>>> a white paper [3] by Alek on this topic is an interesting read.
> >>>>
> >>>> I am of the same opinion that we should adopt something more modern as
> >>>> the challenges from scientific workflows seems to be converging with the
> >>>> data flow patterns in business workflows.
> >>>>
> >>>> It will be great if we can all compile a list of potential candidates
> >>>> and hack them through.
> >>>>
> >>>> Suresh
> >>>> [1] -
> >>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>>> [2] -
> >>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
> >>>> [3] -
> >>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
> >>>>
> >>>>
> >>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>>>
> >>>> > Hi Guys,
> >>>> >
> >>>> > I've been interested in this too - we don't per have a specific
> >>>> > OODT workflow language, but we specific workflows using XML, and
> >>>> > other configuration (we are also thinking of moving to JSON for
> >>>> > this).
> >>>> >
> >>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> >>>> > to me.
> >>>> >
> >>>> > I wonder at the end of the day if we should adopt something more
> >>>> > modern like PIG or some other data flow type of language (PIG
> >>>> > is really neat).
> >>>> >
> >>>> > Cheers,
> >>>> > Chris
> >>>> >
> >>>> >
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Chris Mattmann, Ph.D.
> >>>> > Chief Architect
> >>>> > Instrument Software and Science Data Systems Section (398)
> >>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>>> > Office: 168-519, Mailstop: 168-527
> >>>> > Email: chris.a.mattmann@nasa.gov
> >>>> > WWW: http://sunset.usc.edu/~mattmann/
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Adjunct Associate Professor, Computer Science Department
> >>>> > University of Southern California, Los Angeles, CA 90089 USA
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > -----Original Message-----
> >>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>>> > Reply-To: "architecture@airavata.apache.org"
> >>>> > <ar...@airavata.apache.org>
> >>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>>> > To: "architecture@airavata.apache.org" <
> >>>> architecture@airavata.apache.org>,
> >>>> > dev <de...@airavata.apache.org>
> >>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> >>>> >
> >>>> >> Hi All,
> >>>> >>
> >>>> >> As we all know Airavata has its own workflow language call XWF. When
> >>>> XWF
> >>>> >> was introduced, main focus points are interoperability and
> >>>> convertibility.
> >>>> >> But with years of experience it is convinced that above requirements
> >>>> are
> >>>> >> not really useful when we come to real world use cases. And XWF is
> >>>> XML
> >>>> >> based bulky language where we attache WSDLs and Workflow image it
> >>>> self.
> >>>> >> But
> >>>> >> with the recent changes WSDL part is being removed from XWF.
> >>>> >>
> >>>> >> It is worth to evaluate handy Scientific workflow languages in
> >>>> industry
> >>>> >> and
> >>>> >> find out pros and cons, at the end of this evaluation we need to
> >>>> come up
> >>>> >> with idea how we should improve Airavata workflow language, either
> >>>> we can
> >>>> >> improve existing XWF language, totally change to a new language
> >>>> available
> >>>> >> in industry or write a new light weight language. Basic requirements
> >>>> that
> >>>> >> we expect from new improvement are, high usability, flexible, light
> >>>> weight
> >>>> >> and real time monitoring support. As you can see above requirements
> >>>> are
> >>>> >> not
> >>>> >> direct comes with workflow languages but we need workflow language
> >>>> which
> >>>> >> help to support above requirements.
> >>>> >>
> >>>> >> After reading few papers and googling, initially i have come up with
> >>>> >> following three existing languages,
> >>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>>> >> 2. WS-BPEL
> >>>> >> 3. SIDL
> >>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
> >>>> >>
> >>>> >> In my opinion SIDL is more familiar with scientific domain,
> >>>> Radical-SAGA
> >>>> >> also uses slightly modified version of SIDL. Other than above three
> >>>> >> languages we can come up with simple workflow language base on
> >>>> json(or
> >>>> >> yaml) which support all our requirements for some extends.
> >>>> >>
> >>>> >> It would be grate if I can get more input regarding the $Subject
> >>>> form the
> >>>> >> airavata community. You all are more than welcome to provide any
> >>>> type of
> >>>> >> suggestions.
> >>>> >>
> >>>> >> Thanks,
> >>>> >> Shameera.
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> --
> >>>> >> Best Regards,
> >>>> >> Shameera Rathnayaka.
> >>>> >>
> >>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> The best way to predict the future is to invent it - Alan Kay
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Shameera Rathnayaka.
> >>
> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Supun Kamburugamuva <su...@gmail.com>.
Hi Shameera,
To create a web based UI I don't see an immediate advantage of using
JavaScript over a Python API. Because Airavata's API is Thrift (not any
language based API) you can generate code for any language (including
JavaScript). Anyway most probably you will not use direct Airavata API
calls for creating a web UI. I'm also not sure what you exactly mean by
JavaScript language as well. If you explain in more detail that would be
really helpful.
If you just give a Python API to create workflows, you really don't even
need XBaya to create/run workflows. You can create them as scripts and run
them from command line. You can debug your workflows written in Python
using normal python debuggers. You can generate the XBaya workflow by using
the structure you create in Python as well. Then you can use XBaya to
monitor the workflow. These things are hard to do with an JavaScript API.
As a language also I would pick Python over JavaScript.
Thanks,
Supun..
On Thu, Oct 16, 2014 at 6:46 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:
> Hi Supun,
>
> Considering we are going to provide web base GUI support and JavaScript is
> a well-known client side language, I would select JavaScript over Python. I
> am not much familiar with Python, so would like to know the advantages we
> get by selecting python here.
>
>
> Thanks,
> Shameera.
>
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
> > Hi Shameera,
> >
> > Why you prefer JavaScript over a language like Python?
> >
> > Thanks,
> > Supun..
> >
> > On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > shameerainfo@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> First of all thanks everyone for giving valuable inputs. After doing
> some
> >> background search and talking to different people in the University who
> has
> >> used different workflow languages, I myself convinced that introducing
> an
> >> another workflow language is not what actually they need. By changing
> >> exiting workflow language to another will not solve problems. What they
> >> asking is a easy way to construct the workflows. Indirectly what they
> >> asking for a sort of API which they can use to generate the workflows
> and
> >> run it. Correct me if i am wrong here.
> >>
> >> As most of above replies depict, if we can get a simple API, as an
> >> example, for a web based application, JavaScript API would be a good
> >> solution, and probably JSON would be a good candidate for language,
> instead
> >> of XML.
> >>
> >> Airavata community already have started to implement web base GUI. Hence
> >> introducing a JSON base JavaScript API would be great help. WDYT?
> >>
> >> Thanks,
> >> Shameera.
> >>
> >>
> >> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >> alekny7@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> it is not dataflow instead focused on orchestrating REST services but
> >>> you may find it useful datapoint - we created worfklow service that
> uses
> >>> natively JavaScript and JSON to describe what happens during workflow
> >>> execution:
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>
> >>> HTH,
> >>>
> >>> Alek
> >>>
> >>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> wrote:
> >>>
> >>>> Hi Chris,
> >>>>
> >>>> Great to hear OODT community will be interested in adopting a JSON
> >>>> based workflow language and potentially a web based composer as well.
> >>>> Airavata previously had BPEL support initially through a home grown
> >>>> implementation [1] by Alek Slominski and later through Apache ODE
> [2]. Also
> >>>> a white paper [3] by Alek on this topic is an interesting read.
> >>>>
> >>>> I am of the same opinion that we should adopt something more modern as
> >>>> the challenges from scientific workflows seems to be converging with
> the
> >>>> data flow patterns in business workflows.
> >>>>
> >>>> It will be great if we can all compile a list of potential candidates
> >>>> and hack them through.
> >>>>
> >>>> Suresh
> >>>> [1] -
> >>>>
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>>> [2] -
> >>>>
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
> >>>> [3] -
> >>>>
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
> >>>>
> >>>>
> >>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>>>
> >>>> > Hi Guys,
> >>>> >
> >>>> > I've been interested in this too - we don't per have a specific
> >>>> > OODT workflow language, but we specific workflows using XML, and
> >>>> > other configuration (we are also thinking of moving to JSON for
> >>>> > this).
> >>>> >
> >>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> >>>> > to me.
> >>>> >
> >>>> > I wonder at the end of the day if we should adopt something more
> >>>> > modern like PIG or some other data flow type of language (PIG
> >>>> > is really neat).
> >>>> >
> >>>> > Cheers,
> >>>> > Chris
> >>>> >
> >>>> >
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Chris Mattmann, Ph.D.
> >>>> > Chief Architect
> >>>> > Instrument Software and Science Data Systems Section (398)
> >>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>>> > Office: 168-519, Mailstop: 168-527
> >>>> > Email: chris.a.mattmann@nasa.gov
> >>>> > WWW: http://sunset.usc.edu/~mattmann/
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Adjunct Associate Professor, Computer Science Department
> >>>> > University of Southern California, Los Angeles, CA 90089 USA
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > -----Original Message-----
> >>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>>> > Reply-To: "architecture@airavata.apache.org"
> >>>> > <ar...@airavata.apache.org>
> >>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>>> > To: "architecture@airavata.apache.org" <
> >>>> architecture@airavata.apache.org>,
> >>>> > dev <de...@airavata.apache.org>
> >>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> Airavata.
> >>>> >
> >>>> >> Hi All,
> >>>> >>
> >>>> >> As we all know Airavata has its own workflow language call XWF.
> When
> >>>> XWF
> >>>> >> was introduced, main focus points are interoperability and
> >>>> convertibility.
> >>>> >> But with years of experience it is convinced that above
> requirements
> >>>> are
> >>>> >> not really useful when we come to real world use cases. And XWF is
> >>>> XML
> >>>> >> based bulky language where we attache WSDLs and Workflow image it
> >>>> self.
> >>>> >> But
> >>>> >> with the recent changes WSDL part is being removed from XWF.
> >>>> >>
> >>>> >> It is worth to evaluate handy Scientific workflow languages in
> >>>> industry
> >>>> >> and
> >>>> >> find out pros and cons, at the end of this evaluation we need to
> >>>> come up
> >>>> >> with idea how we should improve Airavata workflow language, either
> >>>> we can
> >>>> >> improve existing XWF language, totally change to a new language
> >>>> available
> >>>> >> in industry or write a new light weight language. Basic
> requirements
> >>>> that
> >>>> >> we expect from new improvement are, high usability, flexible, light
> >>>> weight
> >>>> >> and real time monitoring support. As you can see above requirements
> >>>> are
> >>>> >> not
> >>>> >> direct comes with workflow languages but we need workflow language
> >>>> which
> >>>> >> help to support above requirements.
> >>>> >>
> >>>> >> After reading few papers and googling, initially i have come up
> with
> >>>> >> following three existing languages,
> >>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>>> >> 2. WS-BPEL
> >>>> >> 3. SIDL
> >>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
> >>>> >>
> >>>> >> In my opinion SIDL is more familiar with scientific domain,
> >>>> Radical-SAGA
> >>>> >> also uses slightly modified version of SIDL. Other than above three
> >>>> >> languages we can come up with simple workflow language base on
> >>>> json(or
> >>>> >> yaml) which support all our requirements for some extends.
> >>>> >>
> >>>> >> It would be grate if I can get more input regarding the $Subject
> >>>> form the
> >>>> >> airavata community. You all are more than welcome to provide any
> >>>> type of
> >>>> >> suggestions.
> >>>> >>
> >>>> >> Thanks,
> >>>> >> Shameera.
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> --
> >>>> >> Best Regards,
> >>>> >> Shameera Rathnayaka.
> >>>> >>
> >>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> The best way to predict the future is to invent it - Alan Kay
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Shameera Rathnayaka.
> >>
> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Nipurn Doshi <ni...@umail.iu.edu>.
Hi,
How about I suggest a third option where I have the current code setup
which has access to thrift classes, airavata interface classes and a
complete framework ( Laravel ) that supports rest apis to generate json
data that we are looking at?
I am suggesting this only because I have done this in previous projects I
have worked on using Laravel.
Although, it's in PHP.
On 16-Oct-2014 6:47 pm, "Shameera Rathnayaka" <sh...@gmail.com>
wrote:
> Hi Supun,
>
> Considering we are going to provide web base GUI support and JavaScript
> is a well-known client side language, I would select JavaScript over
> Python. I am not much familiar with Python, so would like to know the
> advantages we get by selecting python here.
>
>
> Thanks,
> Shameera.
>
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
>> Hi Shameera,
>>
>> Why you prefer JavaScript over a language like Python?
>>
>> Thanks,
>> Supun..
>>
>> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> First of all thanks everyone for giving valuable inputs. After doing
>>> some background search and talking to different people in the University
>>> who has used different workflow languages, I myself convinced that
>>> introducing an another workflow language is not what actually they need. By
>>> changing exiting workflow language to another will not solve problems. What
>>> they asking is a easy way to construct the workflows. Indirectly what they
>>> asking for a sort of API which they can use to generate the workflows and
>>> run it. Correct me if i am wrong here.
>>>
>>> As most of above replies depict, if we can get a simple API, as an
>>> example, for a web based application, JavaScript API would be a good
>>> solution, and probably JSON would be a good candidate for language, instead
>>> of XML.
>>>
>>> Airavata community already have started to implement web base GUI. Hence
>>> introducing a JSON base JavaScript API would be great help. WDYT?
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> alekny7@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> it is not dataflow instead focused on orchestrating REST services but
>>>> you may find it useful datapoint - we created worfklow service that uses
>>>> natively JavaScript and JSON to describe what happens during workflow
>>>> execution:
>>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>>
>>>> HTH,
>>>>
>>>> Alek
>>>>
>>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>> Great to hear OODT community will be interested in adopting a JSON
>>>>> based workflow language and potentially a web based composer as well.
>>>>> Airavata previously had BPEL support initially through a home grown
>>>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
>>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>>
>>>>> I am of the same opinion that we should adopt something more modern as
>>>>> the challenges from scientific workflows seems to be converging with the
>>>>> data flow patterns in business workflows.
>>>>>
>>>>> It will be great if we can all compile a list of potential candidates
>>>>> and hack them through.
>>>>>
>>>>> Suresh
>>>>> [1] -
>>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>>> [2] -
>>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>>>> [3] -
>>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>>>
>>>>>
>>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>>
>>>>> > Hi Guys,
>>>>> >
>>>>> > I've been interested in this too - we don't per have a specific
>>>>> > OODT workflow language, but we specific workflows using XML, and
>>>>> > other configuration (we are also thinking of moving to JSON for
>>>>> > this).
>>>>> >
>>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>>> > to me.
>>>>> >
>>>>> > I wonder at the end of the day if we should adopt something more
>>>>> > modern like PIG or some other data flow type of language (PIG
>>>>> > is really neat).
>>>>> >
>>>>> > Cheers,
>>>>> > Chris
>>>>> >
>>>>> >
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Chris Mattmann, Ph.D.
>>>>> > Chief Architect
>>>>> > Instrument Software and Science Data Systems Section (398)
>>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>> > Office: 168-519, Mailstop: 168-527
>>>>> > Email: chris.a.mattmann@nasa.gov
>>>>> > WWW: http://sunset.usc.edu/~mattmann/
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Adjunct Associate Professor, Computer Science Department
>>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > -----Original Message-----
>>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>>> > Reply-To: "architecture@airavata.apache.org"
>>>>> > <ar...@airavata.apache.org>
>>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>>> > To: "architecture@airavata.apache.org" <
>>>>> architecture@airavata.apache.org>,
>>>>> > dev <de...@airavata.apache.org>
>>>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>>>> >
>>>>> >> Hi All,
>>>>> >>
>>>>> >> As we all know Airavata has its own workflow language call XWF.
>>>>> When XWF
>>>>> >> was introduced, main focus points are interoperability and
>>>>> convertibility.
>>>>> >> But with years of experience it is convinced that above
>>>>> requirements are
>>>>> >> not really useful when we come to real world use cases. And XWF is
>>>>> XML
>>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>>> self.
>>>>> >> But
>>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>>> >>
>>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>>> industry
>>>>> >> and
>>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>>> come up
>>>>> >> with idea how we should improve Airavata workflow language, either
>>>>> we can
>>>>> >> improve existing XWF language, totally change to a new language
>>>>> available
>>>>> >> in industry or write a new light weight language. Basic
>>>>> requirements that
>>>>> >> we expect from new improvement are, high usability, flexible, light
>>>>> weight
>>>>> >> and real time monitoring support. As you can see above requirements
>>>>> are
>>>>> >> not
>>>>> >> direct comes with workflow languages but we need workflow language
>>>>> which
>>>>> >> help to support above requirements.
>>>>> >>
>>>>> >> After reading few papers and googling, initially i have come up with
>>>>> >> following three existing languages,
>>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>>> >> 2. WS-BPEL
>>>>> >> 3. SIDL
>>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>>> >>
>>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>>> Radical-SAGA
>>>>> >> also uses slightly modified version of SIDL. Other than above three
>>>>> >> languages we can come up with simple workflow language base on
>>>>> json(or
>>>>> >> yaml) which support all our requirements for some extends.
>>>>> >>
>>>>> >> It would be grate if I can get more input regarding the $Subject
>>>>> form the
>>>>> >> airavata community. You all are more than welcome to provide any
>>>>> type of
>>>>> >> suggestions.
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Shameera.
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Best Regards,
>>>>> >> Shameera Rathnayaka.
>>>>> >>
>>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> The best way to predict the future is to invent it - Alan Kay
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>>
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi Supun,
Considering we are going to provide web base GUI support and JavaScript is
a well-known client side language, I would select JavaScript over Python. I
am not much familiar with Python, so would like to know the advantages we
get by selecting python here.
Thanks,
Shameera.
On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:
> Hi Shameera,
>
> Why you prefer JavaScript over a language like Python?
>
> Thanks,
> Supun..
>
> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi,
>>
>> First of all thanks everyone for giving valuable inputs. After doing some
>> background search and talking to different people in the University who has
>> used different workflow languages, I myself convinced that introducing an
>> another workflow language is not what actually they need. By changing
>> exiting workflow language to another will not solve problems. What they
>> asking is a easy way to construct the workflows. Indirectly what they
>> asking for a sort of API which they can use to generate the workflows and
>> run it. Correct me if i am wrong here.
>>
>> As most of above replies depict, if we can get a simple API, as an
>> example, for a web based application, JavaScript API would be a good
>> solution, and probably JSON would be a good candidate for language, instead
>> of XML.
>>
>> Airavata community already have started to implement web base GUI. Hence
>> introducing a JSON base JavaScript API would be great help. WDYT?
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> alekny7@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> it is not dataflow instead focused on orchestrating REST services but
>>> you may find it useful datapoint - we created worfklow service that uses
>>> natively JavaScript and JSON to describe what happens during workflow
>>> execution:
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>
>>> HTH,
>>>
>>> Alek
>>>
>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>>>
>>>> Hi Chris,
>>>>
>>>> Great to hear OODT community will be interested in adopting a JSON
>>>> based workflow language and potentially a web based composer as well.
>>>> Airavata previously had BPEL support initially through a home grown
>>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>
>>>> I am of the same opinion that we should adopt something more modern as
>>>> the challenges from scientific workflows seems to be converging with the
>>>> data flow patterns in business workflows.
>>>>
>>>> It will be great if we can all compile a list of potential candidates
>>>> and hack them through.
>>>>
>>>> Suresh
>>>> [1] -
>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>> [2] -
>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>>> [3] -
>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>>
>>>>
>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>
>>>> > Hi Guys,
>>>> >
>>>> > I've been interested in this too - we don't per have a specific
>>>> > OODT workflow language, but we specific workflows using XML, and
>>>> > other configuration (we are also thinking of moving to JSON for
>>>> > this).
>>>> >
>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>> > to me.
>>>> >
>>>> > I wonder at the end of the day if we should adopt something more
>>>> > modern like PIG or some other data flow type of language (PIG
>>>> > is really neat).
>>>> >
>>>> > Cheers,
>>>> > Chris
>>>> >
>>>> >
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Chris Mattmann, Ph.D.
>>>> > Chief Architect
>>>> > Instrument Software and Science Data Systems Section (398)
>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> > Office: 168-519, Mailstop: 168-527
>>>> > Email: chris.a.mattmann@nasa.gov
>>>> > WWW: http://sunset.usc.edu/~mattmann/
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Adjunct Associate Professor, Computer Science Department
>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > -----Original Message-----
>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>> > Reply-To: "architecture@airavata.apache.org"
>>>> > <ar...@airavata.apache.org>
>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>> > To: "architecture@airavata.apache.org" <
>>>> architecture@airavata.apache.org>,
>>>> > dev <de...@airavata.apache.org>
>>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>>> >
>>>> >> Hi All,
>>>> >>
>>>> >> As we all know Airavata has its own workflow language call XWF. When
>>>> XWF
>>>> >> was introduced, main focus points are interoperability and
>>>> convertibility.
>>>> >> But with years of experience it is convinced that above requirements
>>>> are
>>>> >> not really useful when we come to real world use cases. And XWF is
>>>> XML
>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>> self.
>>>> >> But
>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>> >>
>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>> industry
>>>> >> and
>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>> come up
>>>> >> with idea how we should improve Airavata workflow language, either
>>>> we can
>>>> >> improve existing XWF language, totally change to a new language
>>>> available
>>>> >> in industry or write a new light weight language. Basic requirements
>>>> that
>>>> >> we expect from new improvement are, high usability, flexible, light
>>>> weight
>>>> >> and real time monitoring support. As you can see above requirements
>>>> are
>>>> >> not
>>>> >> direct comes with workflow languages but we need workflow language
>>>> which
>>>> >> help to support above requirements.
>>>> >>
>>>> >> After reading few papers and googling, initially i have come up with
>>>> >> following three existing languages,
>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>> >> 2. WS-BPEL
>>>> >> 3. SIDL
>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>> >>
>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>> Radical-SAGA
>>>> >> also uses slightly modified version of SIDL. Other than above three
>>>> >> languages we can come up with simple workflow language base on
>>>> json(or
>>>> >> yaml) which support all our requirements for some extends.
>>>> >>
>>>> >> It would be grate if I can get more input regarding the $Subject
>>>> form the
>>>> >> airavata community. You all are more than welcome to provide any
>>>> type of
>>>> >> suggestions.
>>>> >>
>>>> >> Thanks,
>>>> >> Shameera.
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Best Regards,
>>>> >> Shameera Rathnayaka.
>>>> >>
>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>
>>>>
>>>
>>>
>>> --
>>> The best way to predict the future is to invent it - Alan Kay
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>
--
Best Regards,
Shameera Rathnayaka.
email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Supun Kamburugamuva <su...@gmail.com>.
Once we had an offline discussion about the Airavata Workflow language
(with Milinda, Saliya and Shameera). In that discussion one thing came out
was why we need to invent a different language when a simple library like
Python will full fill of Airavata requirements.
There are many benefits in using a Python library as the API for
controlling Airavata workflows.
1. It is a library, gives the ultimate control over the execution and it
can be simpler than any domain specific language that we can come with like
XML, JSON etc
2. Most people use python and can learn it easily than any Domain specific
language
3. You can easily create a Python library for Airavata because all the APIs
are thrift based.
4. If you design the constructs correctly you can plug an XBaya.
Any thoughts?
Thanks,
Supun..
On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:
> Hi Shameera,
>
> Why you prefer JavaScript over a language like Python?
>
> Thanks,
> Supun..
>
> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi,
>>
>> First of all thanks everyone for giving valuable inputs. After doing some
>> background search and talking to different people in the University who has
>> used different workflow languages, I myself convinced that introducing an
>> another workflow language is not what actually they need. By changing
>> exiting workflow language to another will not solve problems. What they
>> asking is a easy way to construct the workflows. Indirectly what they
>> asking for a sort of API which they can use to generate the workflows and
>> run it. Correct me if i am wrong here.
>>
>> As most of above replies depict, if we can get a simple API, as an
>> example, for a web based application, JavaScript API would be a good
>> solution, and probably JSON would be a good candidate for language, instead
>> of XML.
>>
>> Airavata community already have started to implement web base GUI. Hence
>> introducing a JSON base JavaScript API would be great help. WDYT?
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> alekny7@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> it is not dataflow instead focused on orchestrating REST services but
>>> you may find it useful datapoint - we created worfklow service that uses
>>> natively JavaScript and JSON to describe what happens during workflow
>>> execution:
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>
>>> HTH,
>>>
>>> Alek
>>>
>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>>>
>>>> Hi Chris,
>>>>
>>>> Great to hear OODT community will be interested in adopting a JSON
>>>> based workflow language and potentially a web based composer as well.
>>>> Airavata previously had BPEL support initially through a home grown
>>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>
>>>> I am of the same opinion that we should adopt something more modern as
>>>> the challenges from scientific workflows seems to be converging with the
>>>> data flow patterns in business workflows.
>>>>
>>>> It will be great if we can all compile a list of potential candidates
>>>> and hack them through.
>>>>
>>>> Suresh
>>>> [1] -
>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>> [2] -
>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>>> [3] -
>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>>
>>>>
>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>
>>>> > Hi Guys,
>>>> >
>>>> > I've been interested in this too - we don't per have a specific
>>>> > OODT workflow language, but we specific workflows using XML, and
>>>> > other configuration (we are also thinking of moving to JSON for
>>>> > this).
>>>> >
>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>> > to me.
>>>> >
>>>> > I wonder at the end of the day if we should adopt something more
>>>> > modern like PIG or some other data flow type of language (PIG
>>>> > is really neat).
>>>> >
>>>> > Cheers,
>>>> > Chris
>>>> >
>>>> >
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Chris Mattmann, Ph.D.
>>>> > Chief Architect
>>>> > Instrument Software and Science Data Systems Section (398)
>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> > Office: 168-519, Mailstop: 168-527
>>>> > Email: chris.a.mattmann@nasa.gov
>>>> > WWW: http://sunset.usc.edu/~mattmann/
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Adjunct Associate Professor, Computer Science Department
>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > -----Original Message-----
>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>> > Reply-To: "architecture@airavata.apache.org"
>>>> > <ar...@airavata.apache.org>
>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>> > To: "architecture@airavata.apache.org" <
>>>> architecture@airavata.apache.org>,
>>>> > dev <de...@airavata.apache.org>
>>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>>> >
>>>> >> Hi All,
>>>> >>
>>>> >> As we all know Airavata has its own workflow language call XWF. When
>>>> XWF
>>>> >> was introduced, main focus points are interoperability and
>>>> convertibility.
>>>> >> But with years of experience it is convinced that above requirements
>>>> are
>>>> >> not really useful when we come to real world use cases. And XWF is
>>>> XML
>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>> self.
>>>> >> But
>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>> >>
>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>> industry
>>>> >> and
>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>> come up
>>>> >> with idea how we should improve Airavata workflow language, either
>>>> we can
>>>> >> improve existing XWF language, totally change to a new language
>>>> available
>>>> >> in industry or write a new light weight language. Basic requirements
>>>> that
>>>> >> we expect from new improvement are, high usability, flexible, light
>>>> weight
>>>> >> and real time monitoring support. As you can see above requirements
>>>> are
>>>> >> not
>>>> >> direct comes with workflow languages but we need workflow language
>>>> which
>>>> >> help to support above requirements.
>>>> >>
>>>> >> After reading few papers and googling, initially i have come up with
>>>> >> following three existing languages,
>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>> >> 2. WS-BPEL
>>>> >> 3. SIDL
>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>> >>
>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>> Radical-SAGA
>>>> >> also uses slightly modified version of SIDL. Other than above three
>>>> >> languages we can come up with simple workflow language base on
>>>> json(or
>>>> >> yaml) which support all our requirements for some extends.
>>>> >>
>>>> >> It would be grate if I can get more input regarding the $Subject
>>>> form the
>>>> >> airavata community. You all are more than welcome to provide any
>>>> type of
>>>> >> suggestions.
>>>> >>
>>>> >> Thanks,
>>>> >> Shameera.
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Best Regards,
>>>> >> Shameera Rathnayaka.
>>>> >>
>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>
>>>>
>>>
>>>
>>> --
>>> The best way to predict the future is to invent it - Alan Kay
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi Supun,
Considering we are going to provide web base GUI support and JavaScript is
a well-known client side language, I would select JavaScript over Python. I
am not much familiar with Python, so would like to know the advantages we
get by selecting python here.
Thanks,
Shameera.
On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:
> Hi Shameera,
>
> Why you prefer JavaScript over a language like Python?
>
> Thanks,
> Supun..
>
> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi,
>>
>> First of all thanks everyone for giving valuable inputs. After doing some
>> background search and talking to different people in the University who has
>> used different workflow languages, I myself convinced that introducing an
>> another workflow language is not what actually they need. By changing
>> exiting workflow language to another will not solve problems. What they
>> asking is a easy way to construct the workflows. Indirectly what they
>> asking for a sort of API which they can use to generate the workflows and
>> run it. Correct me if i am wrong here.
>>
>> As most of above replies depict, if we can get a simple API, as an
>> example, for a web based application, JavaScript API would be a good
>> solution, and probably JSON would be a good candidate for language, instead
>> of XML.
>>
>> Airavata community already have started to implement web base GUI. Hence
>> introducing a JSON base JavaScript API would be great help. WDYT?
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> alekny7@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> it is not dataflow instead focused on orchestrating REST services but
>>> you may find it useful datapoint - we created worfklow service that uses
>>> natively JavaScript and JSON to describe what happens during workflow
>>> execution:
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>
>>> HTH,
>>>
>>> Alek
>>>
>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>>>
>>>> Hi Chris,
>>>>
>>>> Great to hear OODT community will be interested in adopting a JSON
>>>> based workflow language and potentially a web based composer as well.
>>>> Airavata previously had BPEL support initially through a home grown
>>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>
>>>> I am of the same opinion that we should adopt something more modern as
>>>> the challenges from scientific workflows seems to be converging with the
>>>> data flow patterns in business workflows.
>>>>
>>>> It will be great if we can all compile a list of potential candidates
>>>> and hack them through.
>>>>
>>>> Suresh
>>>> [1] -
>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>> [2] -
>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>>> [3] -
>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>>
>>>>
>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>
>>>> > Hi Guys,
>>>> >
>>>> > I've been interested in this too - we don't per have a specific
>>>> > OODT workflow language, but we specific workflows using XML, and
>>>> > other configuration (we are also thinking of moving to JSON for
>>>> > this).
>>>> >
>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>> > to me.
>>>> >
>>>> > I wonder at the end of the day if we should adopt something more
>>>> > modern like PIG or some other data flow type of language (PIG
>>>> > is really neat).
>>>> >
>>>> > Cheers,
>>>> > Chris
>>>> >
>>>> >
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Chris Mattmann, Ph.D.
>>>> > Chief Architect
>>>> > Instrument Software and Science Data Systems Section (398)
>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> > Office: 168-519, Mailstop: 168-527
>>>> > Email: chris.a.mattmann@nasa.gov
>>>> > WWW: http://sunset.usc.edu/~mattmann/
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Adjunct Associate Professor, Computer Science Department
>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > -----Original Message-----
>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>> > Reply-To: "architecture@airavata.apache.org"
>>>> > <ar...@airavata.apache.org>
>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>> > To: "architecture@airavata.apache.org" <
>>>> architecture@airavata.apache.org>,
>>>> > dev <de...@airavata.apache.org>
>>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>>> >
>>>> >> Hi All,
>>>> >>
>>>> >> As we all know Airavata has its own workflow language call XWF. When
>>>> XWF
>>>> >> was introduced, main focus points are interoperability and
>>>> convertibility.
>>>> >> But with years of experience it is convinced that above requirements
>>>> are
>>>> >> not really useful when we come to real world use cases. And XWF is
>>>> XML
>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>> self.
>>>> >> But
>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>> >>
>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>> industry
>>>> >> and
>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>> come up
>>>> >> with idea how we should improve Airavata workflow language, either
>>>> we can
>>>> >> improve existing XWF language, totally change to a new language
>>>> available
>>>> >> in industry or write a new light weight language. Basic requirements
>>>> that
>>>> >> we expect from new improvement are, high usability, flexible, light
>>>> weight
>>>> >> and real time monitoring support. As you can see above requirements
>>>> are
>>>> >> not
>>>> >> direct comes with workflow languages but we need workflow language
>>>> which
>>>> >> help to support above requirements.
>>>> >>
>>>> >> After reading few papers and googling, initially i have come up with
>>>> >> following three existing languages,
>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>> >> 2. WS-BPEL
>>>> >> 3. SIDL
>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>> >>
>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>> Radical-SAGA
>>>> >> also uses slightly modified version of SIDL. Other than above three
>>>> >> languages we can come up with simple workflow language base on
>>>> json(or
>>>> >> yaml) which support all our requirements for some extends.
>>>> >>
>>>> >> It would be grate if I can get more input regarding the $Subject
>>>> form the
>>>> >> airavata community. You all are more than welcome to provide any
>>>> type of
>>>> >> suggestions.
>>>> >>
>>>> >> Thanks,
>>>> >> Shameera.
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Best Regards,
>>>> >> Shameera Rathnayaka.
>>>> >>
>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>
>>>>
>>>
>>>
>>> --
>>> The best way to predict the future is to invent it - Alan Kay
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>
--
Best Regards,
Shameera Rathnayaka.
email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi Supun,
Considering we are going to provide web base GUI support and JavaScript is
a well-known client side language, I would select JavaScript over Python. I
am not much familiar with Python, so would like to know the advantages we
get by selecting python here.
Thanks,
Shameera.
On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:
> Hi Shameera,
>
> Why you prefer JavaScript over a language like Python?
>
> Thanks,
> Supun..
>
> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi,
>>
>> First of all thanks everyone for giving valuable inputs. After doing some
>> background search and talking to different people in the University who has
>> used different workflow languages, I myself convinced that introducing an
>> another workflow language is not what actually they need. By changing
>> exiting workflow language to another will not solve problems. What they
>> asking is a easy way to construct the workflows. Indirectly what they
>> asking for a sort of API which they can use to generate the workflows and
>> run it. Correct me if i am wrong here.
>>
>> As most of above replies depict, if we can get a simple API, as an
>> example, for a web based application, JavaScript API would be a good
>> solution, and probably JSON would be a good candidate for language, instead
>> of XML.
>>
>> Airavata community already have started to implement web base GUI. Hence
>> introducing a JSON base JavaScript API would be great help. WDYT?
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> alekny7@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> it is not dataflow instead focused on orchestrating REST services but
>>> you may find it useful datapoint - we created worfklow service that uses
>>> natively JavaScript and JSON to describe what happens during workflow
>>> execution:
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>
>>> HTH,
>>>
>>> Alek
>>>
>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>>>
>>>> Hi Chris,
>>>>
>>>> Great to hear OODT community will be interested in adopting a JSON
>>>> based workflow language and potentially a web based composer as well.
>>>> Airavata previously had BPEL support initially through a home grown
>>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>
>>>> I am of the same opinion that we should adopt something more modern as
>>>> the challenges from scientific workflows seems to be converging with the
>>>> data flow patterns in business workflows.
>>>>
>>>> It will be great if we can all compile a list of potential candidates
>>>> and hack them through.
>>>>
>>>> Suresh
>>>> [1] -
>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>> [2] -
>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>>> [3] -
>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>>
>>>>
>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>
>>>> > Hi Guys,
>>>> >
>>>> > I've been interested in this too - we don't per have a specific
>>>> > OODT workflow language, but we specific workflows using XML, and
>>>> > other configuration (we are also thinking of moving to JSON for
>>>> > this).
>>>> >
>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>> > to me.
>>>> >
>>>> > I wonder at the end of the day if we should adopt something more
>>>> > modern like PIG or some other data flow type of language (PIG
>>>> > is really neat).
>>>> >
>>>> > Cheers,
>>>> > Chris
>>>> >
>>>> >
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Chris Mattmann, Ph.D.
>>>> > Chief Architect
>>>> > Instrument Software and Science Data Systems Section (398)
>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> > Office: 168-519, Mailstop: 168-527
>>>> > Email: chris.a.mattmann@nasa.gov
>>>> > WWW: http://sunset.usc.edu/~mattmann/
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Adjunct Associate Professor, Computer Science Department
>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > -----Original Message-----
>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>> > Reply-To: "architecture@airavata.apache.org"
>>>> > <ar...@airavata.apache.org>
>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>> > To: "architecture@airavata.apache.org" <
>>>> architecture@airavata.apache.org>,
>>>> > dev <de...@airavata.apache.org>
>>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>>> >
>>>> >> Hi All,
>>>> >>
>>>> >> As we all know Airavata has its own workflow language call XWF. When
>>>> XWF
>>>> >> was introduced, main focus points are interoperability and
>>>> convertibility.
>>>> >> But with years of experience it is convinced that above requirements
>>>> are
>>>> >> not really useful when we come to real world use cases. And XWF is
>>>> XML
>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>> self.
>>>> >> But
>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>> >>
>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>> industry
>>>> >> and
>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>> come up
>>>> >> with idea how we should improve Airavata workflow language, either
>>>> we can
>>>> >> improve existing XWF language, totally change to a new language
>>>> available
>>>> >> in industry or write a new light weight language. Basic requirements
>>>> that
>>>> >> we expect from new improvement are, high usability, flexible, light
>>>> weight
>>>> >> and real time monitoring support. As you can see above requirements
>>>> are
>>>> >> not
>>>> >> direct comes with workflow languages but we need workflow language
>>>> which
>>>> >> help to support above requirements.
>>>> >>
>>>> >> After reading few papers and googling, initially i have come up with
>>>> >> following three existing languages,
>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>> >> 2. WS-BPEL
>>>> >> 3. SIDL
>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>> >>
>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>> Radical-SAGA
>>>> >> also uses slightly modified version of SIDL. Other than above three
>>>> >> languages we can come up with simple workflow language base on
>>>> json(or
>>>> >> yaml) which support all our requirements for some extends.
>>>> >>
>>>> >> It would be grate if I can get more input regarding the $Subject
>>>> form the
>>>> >> airavata community. You all are more than welcome to provide any
>>>> type of
>>>> >> suggestions.
>>>> >>
>>>> >> Thanks,
>>>> >> Shameera.
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Best Regards,
>>>> >> Shameera Rathnayaka.
>>>> >>
>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>
>>>>
>>>
>>>
>>> --
>>> The best way to predict the future is to invent it - Alan Kay
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>
--
Best Regards,
Shameera Rathnayaka.
email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Supun Kamburugamuva <su...@gmail.com>.
Once we had an offline discussion about the Airavata Workflow language
(with Milinda, Saliya and Shameera). In that discussion one thing came out
was why we need to invent a different language when a simple library like
Python will full fill of Airavata requirements.
There are many benefits in using a Python library as the API for
controlling Airavata workflows.
1. It is a library, gives the ultimate control over the execution and it
can be simpler than any domain specific language that we can come with like
XML, JSON etc
2. Most people use python and can learn it easily than any Domain specific
language
3. You can easily create a Python library for Airavata because all the APIs
are thrift based.
4. If you design the constructs correctly you can plug an XBaya.
Any thoughts?
Thanks,
Supun..
On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:
> Hi Shameera,
>
> Why you prefer JavaScript over a language like Python?
>
> Thanks,
> Supun..
>
> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi,
>>
>> First of all thanks everyone for giving valuable inputs. After doing some
>> background search and talking to different people in the University who has
>> used different workflow languages, I myself convinced that introducing an
>> another workflow language is not what actually they need. By changing
>> exiting workflow language to another will not solve problems. What they
>> asking is a easy way to construct the workflows. Indirectly what they
>> asking for a sort of API which they can use to generate the workflows and
>> run it. Correct me if i am wrong here.
>>
>> As most of above replies depict, if we can get a simple API, as an
>> example, for a web based application, JavaScript API would be a good
>> solution, and probably JSON would be a good candidate for language, instead
>> of XML.
>>
>> Airavata community already have started to implement web base GUI. Hence
>> introducing a JSON base JavaScript API would be great help. WDYT?
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> alekny7@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> it is not dataflow instead focused on orchestrating REST services but
>>> you may find it useful datapoint - we created worfklow service that uses
>>> natively JavaScript and JSON to describe what happens during workflow
>>> execution:
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>
>>> HTH,
>>>
>>> Alek
>>>
>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>>>
>>>> Hi Chris,
>>>>
>>>> Great to hear OODT community will be interested in adopting a JSON
>>>> based workflow language and potentially a web based composer as well.
>>>> Airavata previously had BPEL support initially through a home grown
>>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>
>>>> I am of the same opinion that we should adopt something more modern as
>>>> the challenges from scientific workflows seems to be converging with the
>>>> data flow patterns in business workflows.
>>>>
>>>> It will be great if we can all compile a list of potential candidates
>>>> and hack them through.
>>>>
>>>> Suresh
>>>> [1] -
>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>> [2] -
>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>>> [3] -
>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>>
>>>>
>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>
>>>> > Hi Guys,
>>>> >
>>>> > I've been interested in this too - we don't per have a specific
>>>> > OODT workflow language, but we specific workflows using XML, and
>>>> > other configuration (we are also thinking of moving to JSON for
>>>> > this).
>>>> >
>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>> > to me.
>>>> >
>>>> > I wonder at the end of the day if we should adopt something more
>>>> > modern like PIG or some other data flow type of language (PIG
>>>> > is really neat).
>>>> >
>>>> > Cheers,
>>>> > Chris
>>>> >
>>>> >
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Chris Mattmann, Ph.D.
>>>> > Chief Architect
>>>> > Instrument Software and Science Data Systems Section (398)
>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> > Office: 168-519, Mailstop: 168-527
>>>> > Email: chris.a.mattmann@nasa.gov
>>>> > WWW: http://sunset.usc.edu/~mattmann/
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Adjunct Associate Professor, Computer Science Department
>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > -----Original Message-----
>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>> > Reply-To: "architecture@airavata.apache.org"
>>>> > <ar...@airavata.apache.org>
>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>> > To: "architecture@airavata.apache.org" <
>>>> architecture@airavata.apache.org>,
>>>> > dev <de...@airavata.apache.org>
>>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>>> >
>>>> >> Hi All,
>>>> >>
>>>> >> As we all know Airavata has its own workflow language call XWF. When
>>>> XWF
>>>> >> was introduced, main focus points are interoperability and
>>>> convertibility.
>>>> >> But with years of experience it is convinced that above requirements
>>>> are
>>>> >> not really useful when we come to real world use cases. And XWF is
>>>> XML
>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>> self.
>>>> >> But
>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>> >>
>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>> industry
>>>> >> and
>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>> come up
>>>> >> with idea how we should improve Airavata workflow language, either
>>>> we can
>>>> >> improve existing XWF language, totally change to a new language
>>>> available
>>>> >> in industry or write a new light weight language. Basic requirements
>>>> that
>>>> >> we expect from new improvement are, high usability, flexible, light
>>>> weight
>>>> >> and real time monitoring support. As you can see above requirements
>>>> are
>>>> >> not
>>>> >> direct comes with workflow languages but we need workflow language
>>>> which
>>>> >> help to support above requirements.
>>>> >>
>>>> >> After reading few papers and googling, initially i have come up with
>>>> >> following three existing languages,
>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>> >> 2. WS-BPEL
>>>> >> 3. SIDL
>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>> >>
>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>> Radical-SAGA
>>>> >> also uses slightly modified version of SIDL. Other than above three
>>>> >> languages we can come up with simple workflow language base on
>>>> json(or
>>>> >> yaml) which support all our requirements for some extends.
>>>> >>
>>>> >> It would be grate if I can get more input regarding the $Subject
>>>> form the
>>>> >> airavata community. You all are more than welcome to provide any
>>>> type of
>>>> >> suggestions.
>>>> >>
>>>> >> Thanks,
>>>> >> Shameera.
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Best Regards,
>>>> >> Shameera Rathnayaka.
>>>> >>
>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>
>>>>
>>>
>>>
>>> --
>>> The best way to predict the future is to invent it - Alan Kay
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Supun Kamburugamuva <su...@gmail.com>.
Hi Shameera,
Why you prefer JavaScript over a language like Python?
Thanks,
Supun..
On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:
> Hi,
>
> First of all thanks everyone for giving valuable inputs. After doing some
> background search and talking to different people in the University who has
> used different workflow languages, I myself convinced that introducing an
> another workflow language is not what actually they need. By changing
> exiting workflow language to another will not solve problems. What they
> asking is a easy way to construct the workflows. Indirectly what they
> asking for a sort of API which they can use to generate the workflows and
> run it. Correct me if i am wrong here.
>
> As most of above replies depict, if we can get a simple API, as an
> example, for a web based application, JavaScript API would be a good
> solution, and probably JSON would be a good candidate for language, instead
> of XML.
>
> Airavata community already have started to implement web base GUI. Hence
> introducing a JSON base JavaScript API would be great help. WDYT?
>
> Thanks,
> Shameera.
>
>
> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> alekny7@gmail.com> wrote:
>
>> Hi,
>>
>> it is not dataflow instead focused on orchestrating REST services but you
>> may find it useful datapoint - we created worfklow service that uses
>> natively JavaScript and JSON to describe what happens during workflow
>> execution:
>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>
>> HTH,
>>
>> Alek
>>
>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>>
>>> Hi Chris,
>>>
>>> Great to hear OODT community will be interested in adopting a JSON based
>>> workflow language and potentially a web based composer as well. Airavata
>>> previously had BPEL support initially through a home grown implementation
>>> [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper
>>> [3] by Alek on this topic is an interesting read.
>>>
>>> I am of the same opinion that we should adopt something more modern as
>>> the challenges from scientific workflows seems to be converging with the
>>> data flow patterns in business workflows.
>>>
>>> It will be great if we can all compile a list of potential candidates
>>> and hack them through.
>>>
>>> Suresh
>>> [1] -
>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>> [2] -
>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>> [3] -
>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>
>>>
>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>
>>> > Hi Guys,
>>> >
>>> > I've been interested in this too - we don't per have a specific
>>> > OODT workflow language, but we specific workflows using XML, and
>>> > other configuration (we are also thinking of moving to JSON for
>>> > this).
>>> >
>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>> > to me.
>>> >
>>> > I wonder at the end of the day if we should adopt something more
>>> > modern like PIG or some other data flow type of language (PIG
>>> > is really neat).
>>> >
>>> > Cheers,
>>> > Chris
>>> >
>>> >
>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > Chris Mattmann, Ph.D.
>>> > Chief Architect
>>> > Instrument Software and Science Data Systems Section (398)
>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> > Office: 168-519, Mailstop: 168-527
>>> > Email: chris.a.mattmann@nasa.gov
>>> > WWW: http://sunset.usc.edu/~mattmann/
>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > Adjunct Associate Professor, Computer Science Department
>>> > University of Southern California, Los Angeles, CA 90089 USA
>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>> > Reply-To: "architecture@airavata.apache.org"
>>> > <ar...@airavata.apache.org>
>>> > Date: Thursday, September 18, 2014 8:26 AM
>>> > To: "architecture@airavata.apache.org" <
>>> architecture@airavata.apache.org>,
>>> > dev <de...@airavata.apache.org>
>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>> >
>>> >> Hi All,
>>> >>
>>> >> As we all know Airavata has its own workflow language call XWF. When
>>> XWF
>>> >> was introduced, main focus points are interoperability and
>>> convertibility.
>>> >> But with years of experience it is convinced that above requirements
>>> are
>>> >> not really useful when we come to real world use cases. And XWF is XML
>>> >> based bulky language where we attache WSDLs and Workflow image it
>>> self.
>>> >> But
>>> >> with the recent changes WSDL part is being removed from XWF.
>>> >>
>>> >> It is worth to evaluate handy Scientific workflow languages in
>>> industry
>>> >> and
>>> >> find out pros and cons, at the end of this evaluation we need to come
>>> up
>>> >> with idea how we should improve Airavata workflow language, either we
>>> can
>>> >> improve existing XWF language, totally change to a new language
>>> available
>>> >> in industry or write a new light weight language. Basic requirements
>>> that
>>> >> we expect from new improvement are, high usability, flexible, light
>>> weight
>>> >> and real time monitoring support. As you can see above requirements
>>> are
>>> >> not
>>> >> direct comes with workflow languages but we need workflow language
>>> which
>>> >> help to support above requirements.
>>> >>
>>> >> After reading few papers and googling, initially i have come up with
>>> >> following three existing languages,
>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>> >> 2. WS-BPEL
>>> >> 3. SIDL
>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>> >>
>>> >> In my opinion SIDL is more familiar with scientific domain,
>>> Radical-SAGA
>>> >> also uses slightly modified version of SIDL. Other than above three
>>> >> languages we can come up with simple workflow language base on json(or
>>> >> yaml) which support all our requirements for some extends.
>>> >>
>>> >> It would be grate if I can get more input regarding the $Subject form
>>> the
>>> >> airavata community. You all are more than welcome to provide any type
>>> of
>>> >> suggestions.
>>> >>
>>> >> Thanks,
>>> >> Shameera.
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Best Regards,
>>> >> Shameera Rathnayaka.
>>> >>
>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>>
>>
>>
>> --
>> The best way to predict the future is to invent it - Alan Kay
>>
>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Supun Kamburugamuva <su...@gmail.com>.
Hi Shameera,
Why you prefer JavaScript over a language like Python?
Thanks,
Supun..
On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:
> Hi,
>
> First of all thanks everyone for giving valuable inputs. After doing some
> background search and talking to different people in the University who has
> used different workflow languages, I myself convinced that introducing an
> another workflow language is not what actually they need. By changing
> exiting workflow language to another will not solve problems. What they
> asking is a easy way to construct the workflows. Indirectly what they
> asking for a sort of API which they can use to generate the workflows and
> run it. Correct me if i am wrong here.
>
> As most of above replies depict, if we can get a simple API, as an
> example, for a web based application, JavaScript API would be a good
> solution, and probably JSON would be a good candidate for language, instead
> of XML.
>
> Airavata community already have started to implement web base GUI. Hence
> introducing a JSON base JavaScript API would be great help. WDYT?
>
> Thanks,
> Shameera.
>
>
> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> alekny7@gmail.com> wrote:
>
>> Hi,
>>
>> it is not dataflow instead focused on orchestrating REST services but you
>> may find it useful datapoint - we created worfklow service that uses
>> natively JavaScript and JSON to describe what happens during workflow
>> execution:
>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>
>> HTH,
>>
>> Alek
>>
>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>>
>>> Hi Chris,
>>>
>>> Great to hear OODT community will be interested in adopting a JSON based
>>> workflow language and potentially a web based composer as well. Airavata
>>> previously had BPEL support initially through a home grown implementation
>>> [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper
>>> [3] by Alek on this topic is an interesting read.
>>>
>>> I am of the same opinion that we should adopt something more modern as
>>> the challenges from scientific workflows seems to be converging with the
>>> data flow patterns in business workflows.
>>>
>>> It will be great if we can all compile a list of potential candidates
>>> and hack them through.
>>>
>>> Suresh
>>> [1] -
>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>> [2] -
>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>> [3] -
>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>
>>>
>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>
>>> > Hi Guys,
>>> >
>>> > I've been interested in this too - we don't per have a specific
>>> > OODT workflow language, but we specific workflows using XML, and
>>> > other configuration (we are also thinking of moving to JSON for
>>> > this).
>>> >
>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>> > to me.
>>> >
>>> > I wonder at the end of the day if we should adopt something more
>>> > modern like PIG or some other data flow type of language (PIG
>>> > is really neat).
>>> >
>>> > Cheers,
>>> > Chris
>>> >
>>> >
>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > Chris Mattmann, Ph.D.
>>> > Chief Architect
>>> > Instrument Software and Science Data Systems Section (398)
>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> > Office: 168-519, Mailstop: 168-527
>>> > Email: chris.a.mattmann@nasa.gov
>>> > WWW: http://sunset.usc.edu/~mattmann/
>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > Adjunct Associate Professor, Computer Science Department
>>> > University of Southern California, Los Angeles, CA 90089 USA
>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>> > Reply-To: "architecture@airavata.apache.org"
>>> > <ar...@airavata.apache.org>
>>> > Date: Thursday, September 18, 2014 8:26 AM
>>> > To: "architecture@airavata.apache.org" <
>>> architecture@airavata.apache.org>,
>>> > dev <de...@airavata.apache.org>
>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>> >
>>> >> Hi All,
>>> >>
>>> >> As we all know Airavata has its own workflow language call XWF. When
>>> XWF
>>> >> was introduced, main focus points are interoperability and
>>> convertibility.
>>> >> But with years of experience it is convinced that above requirements
>>> are
>>> >> not really useful when we come to real world use cases. And XWF is XML
>>> >> based bulky language where we attache WSDLs and Workflow image it
>>> self.
>>> >> But
>>> >> with the recent changes WSDL part is being removed from XWF.
>>> >>
>>> >> It is worth to evaluate handy Scientific workflow languages in
>>> industry
>>> >> and
>>> >> find out pros and cons, at the end of this evaluation we need to come
>>> up
>>> >> with idea how we should improve Airavata workflow language, either we
>>> can
>>> >> improve existing XWF language, totally change to a new language
>>> available
>>> >> in industry or write a new light weight language. Basic requirements
>>> that
>>> >> we expect from new improvement are, high usability, flexible, light
>>> weight
>>> >> and real time monitoring support. As you can see above requirements
>>> are
>>> >> not
>>> >> direct comes with workflow languages but we need workflow language
>>> which
>>> >> help to support above requirements.
>>> >>
>>> >> After reading few papers and googling, initially i have come up with
>>> >> following three existing languages,
>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>> >> 2. WS-BPEL
>>> >> 3. SIDL
>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>> >>
>>> >> In my opinion SIDL is more familiar with scientific domain,
>>> Radical-SAGA
>>> >> also uses slightly modified version of SIDL. Other than above three
>>> >> languages we can come up with simple workflow language base on json(or
>>> >> yaml) which support all our requirements for some extends.
>>> >>
>>> >> It would be grate if I can get more input regarding the $Subject form
>>> the
>>> >> airavata community. You all are more than welcome to provide any type
>>> of
>>> >> suggestions.
>>> >>
>>> >> Thanks,
>>> >> Shameera.
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Best Regards,
>>> >> Shameera Rathnayaka.
>>> >>
>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>>
>>
>>
>> --
>> The best way to predict the future is to invent it - Alan Kay
>>
>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Supun Kamburugamuva <su...@gmail.com>.
Hi Shameera,
Why you prefer JavaScript over a language like Python?
Thanks,
Supun..
On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:
> Hi,
>
> First of all thanks everyone for giving valuable inputs. After doing some
> background search and talking to different people in the University who has
> used different workflow languages, I myself convinced that introducing an
> another workflow language is not what actually they need. By changing
> exiting workflow language to another will not solve problems. What they
> asking is a easy way to construct the workflows. Indirectly what they
> asking for a sort of API which they can use to generate the workflows and
> run it. Correct me if i am wrong here.
>
> As most of above replies depict, if we can get a simple API, as an
> example, for a web based application, JavaScript API would be a good
> solution, and probably JSON would be a good candidate for language, instead
> of XML.
>
> Airavata community already have started to implement web base GUI. Hence
> introducing a JSON base JavaScript API would be great help. WDYT?
>
> Thanks,
> Shameera.
>
>
> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> alekny7@gmail.com> wrote:
>
>> Hi,
>>
>> it is not dataflow instead focused on orchestrating REST services but you
>> may find it useful datapoint - we created worfklow service that uses
>> natively JavaScript and JSON to describe what happens during workflow
>> execution:
>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>
>> HTH,
>>
>> Alek
>>
>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>>
>>> Hi Chris,
>>>
>>> Great to hear OODT community will be interested in adopting a JSON based
>>> workflow language and potentially a web based composer as well. Airavata
>>> previously had BPEL support initially through a home grown implementation
>>> [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper
>>> [3] by Alek on this topic is an interesting read.
>>>
>>> I am of the same opinion that we should adopt something more modern as
>>> the challenges from scientific workflows seems to be converging with the
>>> data flow patterns in business workflows.
>>>
>>> It will be great if we can all compile a list of potential candidates
>>> and hack them through.
>>>
>>> Suresh
>>> [1] -
>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>> [2] -
>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>> [3] -
>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>
>>>
>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>
>>> > Hi Guys,
>>> >
>>> > I've been interested in this too - we don't per have a specific
>>> > OODT workflow language, but we specific workflows using XML, and
>>> > other configuration (we are also thinking of moving to JSON for
>>> > this).
>>> >
>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>> > to me.
>>> >
>>> > I wonder at the end of the day if we should adopt something more
>>> > modern like PIG or some other data flow type of language (PIG
>>> > is really neat).
>>> >
>>> > Cheers,
>>> > Chris
>>> >
>>> >
>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > Chris Mattmann, Ph.D.
>>> > Chief Architect
>>> > Instrument Software and Science Data Systems Section (398)
>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> > Office: 168-519, Mailstop: 168-527
>>> > Email: chris.a.mattmann@nasa.gov
>>> > WWW: http://sunset.usc.edu/~mattmann/
>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > Adjunct Associate Professor, Computer Science Department
>>> > University of Southern California, Los Angeles, CA 90089 USA
>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>> > Reply-To: "architecture@airavata.apache.org"
>>> > <ar...@airavata.apache.org>
>>> > Date: Thursday, September 18, 2014 8:26 AM
>>> > To: "architecture@airavata.apache.org" <
>>> architecture@airavata.apache.org>,
>>> > dev <de...@airavata.apache.org>
>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>> >
>>> >> Hi All,
>>> >>
>>> >> As we all know Airavata has its own workflow language call XWF. When
>>> XWF
>>> >> was introduced, main focus points are interoperability and
>>> convertibility.
>>> >> But with years of experience it is convinced that above requirements
>>> are
>>> >> not really useful when we come to real world use cases. And XWF is XML
>>> >> based bulky language where we attache WSDLs and Workflow image it
>>> self.
>>> >> But
>>> >> with the recent changes WSDL part is being removed from XWF.
>>> >>
>>> >> It is worth to evaluate handy Scientific workflow languages in
>>> industry
>>> >> and
>>> >> find out pros and cons, at the end of this evaluation we need to come
>>> up
>>> >> with idea how we should improve Airavata workflow language, either we
>>> can
>>> >> improve existing XWF language, totally change to a new language
>>> available
>>> >> in industry or write a new light weight language. Basic requirements
>>> that
>>> >> we expect from new improvement are, high usability, flexible, light
>>> weight
>>> >> and real time monitoring support. As you can see above requirements
>>> are
>>> >> not
>>> >> direct comes with workflow languages but we need workflow language
>>> which
>>> >> help to support above requirements.
>>> >>
>>> >> After reading few papers and googling, initially i have come up with
>>> >> following three existing languages,
>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>> >> 2. WS-BPEL
>>> >> 3. SIDL
>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>> >>
>>> >> In my opinion SIDL is more familiar with scientific domain,
>>> Radical-SAGA
>>> >> also uses slightly modified version of SIDL. Other than above three
>>> >> languages we can come up with simple workflow language base on json(or
>>> >> yaml) which support all our requirements for some extends.
>>> >>
>>> >> It would be grate if I can get more input regarding the $Subject form
>>> the
>>> >> airavata community. You all are more than welcome to provide any type
>>> of
>>> >> suggestions.
>>> >>
>>> >> Thanks,
>>> >> Shameera.
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Best Regards,
>>> >> Shameera Rathnayaka.
>>> >>
>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>>
>>
>>
>> --
>> The best way to predict the future is to invent it - Alan Kay
>>
>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi,
First of all thanks everyone for giving valuable inputs. After doing some
background search and talking to different people in the University who has
used different workflow languages, I myself convinced that introducing an
another workflow language is not what actually they need. By changing
exiting workflow language to another will not solve problems. What they
asking is a easy way to construct the workflows. Indirectly what they
asking for a sort of API which they can use to generate the workflows and
run it. Correct me if i am wrong here.
As most of above replies depict, if we can get a simple API, as an
example, for a web based application, JavaScript API would be a good
solution, and probably JSON would be a good candidate for language, instead
of XML.
Airavata community already have started to implement web base GUI. Hence
introducing a JSON base JavaScript API would be great help. WDYT?
Thanks,
Shameera.
On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
alekny7@gmail.com> wrote:
> Hi,
>
> it is not dataflow instead focused on orchestrating REST services but you
> may find it useful datapoint - we created worfklow service that uses
> natively JavaScript and JSON to describe what happens during workflow
> execution:
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>
> HTH,
>
> Alek
>
> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>
>> Hi Chris,
>>
>> Great to hear OODT community will be interested in adopting a JSON based
>> workflow language and potentially a web based composer as well. Airavata
>> previously had BPEL support initially through a home grown implementation
>> [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper
>> [3] by Alek on this topic is an interesting read.
>>
>> I am of the same opinion that we should adopt something more modern as
>> the challenges from scientific workflows seems to be converging with the
>> data flow patterns in business workflows.
>>
>> It will be great if we can all compile a list of potential candidates and
>> hack them through.
>>
>> Suresh
>> [1] -
>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>> [2] -
>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>> [3] -
>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>
>>
>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>
>> > Hi Guys,
>> >
>> > I've been interested in this too - we don't per have a specific
>> > OODT workflow language, but we specific workflows using XML, and
>> > other configuration (we are also thinking of moving to JSON for
>> > this).
>> >
>> > In the past I've also looked at YAWL and BPEL - both seem complex
>> > to me.
>> >
>> > I wonder at the end of the day if we should adopt something more
>> > modern like PIG or some other data flow type of language (PIG
>> > is really neat).
>> >
>> > Cheers,
>> > Chris
>> >
>> >
>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > Chris Mattmann, Ph.D.
>> > Chief Architect
>> > Instrument Software and Science Data Systems Section (398)
>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > Office: 168-519, Mailstop: 168-527
>> > Email: chris.a.mattmann@nasa.gov
>> > WWW: http://sunset.usc.edu/~mattmann/
>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > Adjunct Associate Professor, Computer Science Department
>> > University of Southern California, Los Angeles, CA 90089 USA
>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >
>> >
>> >
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: Shameera Rathnayaka <sh...@gmail.com>
>> > Reply-To: "architecture@airavata.apache.org"
>> > <ar...@airavata.apache.org>
>> > Date: Thursday, September 18, 2014 8:26 AM
>> > To: "architecture@airavata.apache.org" <
>> architecture@airavata.apache.org>,
>> > dev <de...@airavata.apache.org>
>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>> >
>> >> Hi All,
>> >>
>> >> As we all know Airavata has its own workflow language call XWF. When
>> XWF
>> >> was introduced, main focus points are interoperability and
>> convertibility.
>> >> But with years of experience it is convinced that above requirements
>> are
>> >> not really useful when we come to real world use cases. And XWF is XML
>> >> based bulky language where we attache WSDLs and Workflow image it self.
>> >> But
>> >> with the recent changes WSDL part is being removed from XWF.
>> >>
>> >> It is worth to evaluate handy Scientific workflow languages in industry
>> >> and
>> >> find out pros and cons, at the end of this evaluation we need to come
>> up
>> >> with idea how we should improve Airavata workflow language, either we
>> can
>> >> improve existing XWF language, totally change to a new language
>> available
>> >> in industry or write a new light weight language. Basic requirements
>> that
>> >> we expect from new improvement are, high usability, flexible, light
>> weight
>> >> and real time monitoring support. As you can see above requirements are
>> >> not
>> >> direct comes with workflow languages but we need workflow language
>> which
>> >> help to support above requirements.
>> >>
>> >> After reading few papers and googling, initially i have come up with
>> >> following three existing languages,
>> >> 1. YAWL <http://www.yawlfoundation.org/>
>> >> 2. WS-BPEL
>> >> 3. SIDL
>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>> >>
>> >> In my opinion SIDL is more familiar with scientific domain,
>> Radical-SAGA
>> >> also uses slightly modified version of SIDL. Other than above three
>> >> languages we can come up with simple workflow language base on json(or
>> >> yaml) which support all our requirements for some extends.
>> >>
>> >> It would be grate if I can get more input regarding the $Subject form
>> the
>> >> airavata community. You all are more than welcome to provide any type
>> of
>> >> suggestions.
>> >>
>> >> Thanks,
>> >> Shameera.
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards,
>> >> Shameera Rathnayaka.
>> >>
>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>
>>
>
>
> --
> The best way to predict the future is to invent it - Alan Kay
>
--
Best Regards,
Shameera Rathnayaka.
email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi,
First of all thanks everyone for giving valuable inputs. After doing some
background search and talking to different people in the University who has
used different workflow languages, I myself convinced that introducing an
another workflow language is not what actually they need. By changing
exiting workflow language to another will not solve problems. What they
asking is a easy way to construct the workflows. Indirectly what they
asking for a sort of API which they can use to generate the workflows and
run it. Correct me if i am wrong here.
As most of above replies depict, if we can get a simple API, as an
example, for a web based application, JavaScript API would be a good
solution, and probably JSON would be a good candidate for language, instead
of XML.
Airavata community already have started to implement web base GUI. Hence
introducing a JSON base JavaScript API would be great help. WDYT?
Thanks,
Shameera.
On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
alekny7@gmail.com> wrote:
> Hi,
>
> it is not dataflow instead focused on orchestrating REST services but you
> may find it useful datapoint - we created worfklow service that uses
> natively JavaScript and JSON to describe what happens during workflow
> execution:
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>
> HTH,
>
> Alek
>
> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>
>> Hi Chris,
>>
>> Great to hear OODT community will be interested in adopting a JSON based
>> workflow language and potentially a web based composer as well. Airavata
>> previously had BPEL support initially through a home grown implementation
>> [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper
>> [3] by Alek on this topic is an interesting read.
>>
>> I am of the same opinion that we should adopt something more modern as
>> the challenges from scientific workflows seems to be converging with the
>> data flow patterns in business workflows.
>>
>> It will be great if we can all compile a list of potential candidates and
>> hack them through.
>>
>> Suresh
>> [1] -
>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>> [2] -
>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>> [3] -
>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>
>>
>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>
>> > Hi Guys,
>> >
>> > I've been interested in this too - we don't per have a specific
>> > OODT workflow language, but we specific workflows using XML, and
>> > other configuration (we are also thinking of moving to JSON for
>> > this).
>> >
>> > In the past I've also looked at YAWL and BPEL - both seem complex
>> > to me.
>> >
>> > I wonder at the end of the day if we should adopt something more
>> > modern like PIG or some other data flow type of language (PIG
>> > is really neat).
>> >
>> > Cheers,
>> > Chris
>> >
>> >
>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > Chris Mattmann, Ph.D.
>> > Chief Architect
>> > Instrument Software and Science Data Systems Section (398)
>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > Office: 168-519, Mailstop: 168-527
>> > Email: chris.a.mattmann@nasa.gov
>> > WWW: http://sunset.usc.edu/~mattmann/
>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > Adjunct Associate Professor, Computer Science Department
>> > University of Southern California, Los Angeles, CA 90089 USA
>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >
>> >
>> >
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: Shameera Rathnayaka <sh...@gmail.com>
>> > Reply-To: "architecture@airavata.apache.org"
>> > <ar...@airavata.apache.org>
>> > Date: Thursday, September 18, 2014 8:26 AM
>> > To: "architecture@airavata.apache.org" <
>> architecture@airavata.apache.org>,
>> > dev <de...@airavata.apache.org>
>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>> >
>> >> Hi All,
>> >>
>> >> As we all know Airavata has its own workflow language call XWF. When
>> XWF
>> >> was introduced, main focus points are interoperability and
>> convertibility.
>> >> But with years of experience it is convinced that above requirements
>> are
>> >> not really useful when we come to real world use cases. And XWF is XML
>> >> based bulky language where we attache WSDLs and Workflow image it self.
>> >> But
>> >> with the recent changes WSDL part is being removed from XWF.
>> >>
>> >> It is worth to evaluate handy Scientific workflow languages in industry
>> >> and
>> >> find out pros and cons, at the end of this evaluation we need to come
>> up
>> >> with idea how we should improve Airavata workflow language, either we
>> can
>> >> improve existing XWF language, totally change to a new language
>> available
>> >> in industry or write a new light weight language. Basic requirements
>> that
>> >> we expect from new improvement are, high usability, flexible, light
>> weight
>> >> and real time monitoring support. As you can see above requirements are
>> >> not
>> >> direct comes with workflow languages but we need workflow language
>> which
>> >> help to support above requirements.
>> >>
>> >> After reading few papers and googling, initially i have come up with
>> >> following three existing languages,
>> >> 1. YAWL <http://www.yawlfoundation.org/>
>> >> 2. WS-BPEL
>> >> 3. SIDL
>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>> >>
>> >> In my opinion SIDL is more familiar with scientific domain,
>> Radical-SAGA
>> >> also uses slightly modified version of SIDL. Other than above three
>> >> languages we can come up with simple workflow language base on json(or
>> >> yaml) which support all our requirements for some extends.
>> >>
>> >> It would be grate if I can get more input regarding the $Subject form
>> the
>> >> airavata community. You all are more than welcome to provide any type
>> of
>> >> suggestions.
>> >>
>> >> Thanks,
>> >> Shameera.
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards,
>> >> Shameera Rathnayaka.
>> >>
>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>
>>
>
>
> --
> The best way to predict the future is to invent it - Alan Kay
>
--
Best Regards,
Shameera Rathnayaka.
email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Bruce Barkstrom <br...@gmail.com>.
Thanks very much for the documentation.
Just glancing through it, I'm not sure I'd describe production
by embedding a particular language in the basic description.
In the large-scale production systems I architected for NASA's
ERBE and CERES production, I started with a Data Flow Diagram.
Now, I'd probably move to UML before getting close to a language-
specific design.
A key part of our work was two years getting an agreement on
the naming convention of the files that tied in to the multiple
instruments, multiple products, and multiple versions of each
element in the provenance. I did a formal description of those
structures in an ESI paper (I guess 2009) on configuration management
in large-scale satellite data production. That paper contains
some example figures of the provenance graphs - which are
probably easiest to visualize if you get the pdf version and
view the graphs with a zoom of 2800% or so.
One point that was important is that our production flow was
not completely automated - and humans had to invoke the
proper connection between input files, various pieces of
production executables, and output files - including variations
from one month to the next a as well as match*ing *which satellite
was involved (plus some other configuration issues). In the
long run, we also needed to build production schedules so
the archive could check quality control on the proper running
of the process. That work required interaction between objects
and staff on the archive and the science teams, which makes
for the usual schedule complications that are an intrinsic
part of production planning on big projects.
Also, in some work I'm doing currently, I am quite concerned
about user search states (or production states) that come from
a Markov model that allows simulation of user loadings on
an archive. I haven't gone into full detail on this yet - but it
may mean that the interaction between the user and the archive
needs to keep track of the state of both agents. The basic
notion for this work is that the archive and the user are
playing a two agent cooperative game where the archive
knows what it contains and the user knows what his or her
target is. I think that requires maintaining agent states across
multiple sessions. I don't recall whether that violates the
original source of the RESTFUL design, so I'll need to go
back to the original to check.
I'll try to look at what you've sent in more detail and provide
some more comments. Regardless, I appreciate the information.
Bruce B.
On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
alekny7@gmail.com> wrote:
> Hi,
>
> it is not dataflow instead focused on orchestrating REST services but you
> may find it useful datapoint - we created worfklow service that uses
> natively JavaScript and JSON to describe what happens during workflow
> execution:
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>
> HTH,
>
> Alek
>
> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>
> > Hi Chris,
> >
> > Great to hear OODT community will be interested in adopting a JSON based
> > workflow language and potentially a web based composer as well. Airavata
> > previously had BPEL support initially through a home grown implementation
> > [1] by Alek Slominski and later through Apache ODE [2]. Also a white
> paper
> > [3] by Alek on this topic is an interesting read.
> >
> > I am of the same opinion that we should adopt something more modern as
> the
> > challenges from scientific workflows seems to be converging with the data
> > flow patterns in business workflows.
> >
> > It will be great if we can all compile a list of potential candidates and
> > hack them through.
> >
> > Suresh
> > [1] -
> > http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > [2] -
> >
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
> > [3] -
> >
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
> >
> >
> > On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> > > Hi Guys,
> > >
> > > I've been interested in this too - we don't per have a specific
> > > OODT workflow language, but we specific workflows using XML, and
> > > other configuration (we are also thinking of moving to JSON for
> > > this).
> > >
> > > In the past I've also looked at YAWL and BPEL - both seem complex
> > > to me.
> > >
> > > I wonder at the end of the day if we should adopt something more
> > > modern like PIG or some other data flow type of language (PIG
> > > is really neat).
> > >
> > > Cheers,
> > > Chris
> > >
> > >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > Chris Mattmann, Ph.D.
> > > Chief Architect
> > > Instrument Software and Science Data Systems Section (398)
> > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > Office: 168-519, Mailstop: 168-527
> > > Email: chris.a.mattmann@nasa.gov
> > > WWW: http://sunset.usc.edu/~mattmann/
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > Adjunct Associate Professor, Computer Science Department
> > > University of Southern California, Los Angeles, CA 90089 USA
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Shameera Rathnayaka <sh...@gmail.com>
> > > Reply-To: "architecture@airavata.apache.org"
> > > <ar...@airavata.apache.org>
> > > Date: Thursday, September 18, 2014 8:26 AM
> > > To: "architecture@airavata.apache.org" <
> architecture@airavata.apache.org
> > >,
> > > dev <de...@airavata.apache.org>
> > > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> > >
> > >> Hi All,
> > >>
> > >> As we all know Airavata has its own workflow language call XWF. When
> XWF
> > >> was introduced, main focus points are interoperability and
> > convertibility.
> > >> But with years of experience it is convinced that above requirements
> are
> > >> not really useful when we come to real world use cases. And XWF is XML
> > >> based bulky language where we attache WSDLs and Workflow image it
> self.
> > >> But
> > >> with the recent changes WSDL part is being removed from XWF.
> > >>
> > >> It is worth to evaluate handy Scientific workflow languages in
> industry
> > >> and
> > >> find out pros and cons, at the end of this evaluation we need to come
> up
> > >> with idea how we should improve Airavata workflow language, either we
> > can
> > >> improve existing XWF language, totally change to a new language
> > available
> > >> in industry or write a new light weight language. Basic requirements
> > that
> > >> we expect from new improvement are, high usability, flexible, light
> > weight
> > >> and real time monitoring support. As you can see above requirements
> are
> > >> not
> > >> direct comes with workflow languages but we need workflow language
> which
> > >> help to support above requirements.
> > >>
> > >> After reading few papers and googling, initially i have come up with
> > >> following three existing languages,
> > >> 1. YAWL <http://www.yawlfoundation.org/>
> > >> 2. WS-BPEL
> > >> 3. SIDL
> > >> <http://computation.llnl.gov/casc/components/index.html#page=home>
> > >>
> > >> In my opinion SIDL is more familiar with scientific domain,
> Radical-SAGA
> > >> also uses slightly modified version of SIDL. Other than above three
> > >> languages we can come up with simple workflow language base on json(or
> > >> yaml) which support all our requirements for some extends.
> > >>
> > >> It would be grate if I can get more input regarding the $Subject form
> > the
> > >> airavata community. You all are more than welcome to provide any type
> of
> > >> suggestions.
> > >>
> > >> Thanks,
> > >> Shameera.
> > >>
> > >>
> > >>
> > >> --
> > >> Best Regards,
> > >> Shameera Rathnayaka.
> > >>
> > >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > >> Blog : http://shameerarathnayaka.blogspot.com/
> >
> >
>
>
> --
> The best way to predict the future is to invent it - Alan Kay
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi,
First of all thanks everyone for giving valuable inputs. After doing some
background search and talking to different people in the University who has
used different workflow languages, I myself convinced that introducing an
another workflow language is not what actually they need. By changing
exiting workflow language to another will not solve problems. What they
asking is a easy way to construct the workflows. Indirectly what they
asking for a sort of API which they can use to generate the workflows and
run it. Correct me if i am wrong here.
As most of above replies depict, if we can get a simple API, as an
example, for a web based application, JavaScript API would be a good
solution, and probably JSON would be a good candidate for language, instead
of XML.
Airavata community already have started to implement web base GUI. Hence
introducing a JSON base JavaScript API would be great help. WDYT?
Thanks,
Shameera.
On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
alekny7@gmail.com> wrote:
> Hi,
>
> it is not dataflow instead focused on orchestrating REST services but you
> may find it useful datapoint - we created worfklow service that uses
> natively JavaScript and JSON to describe what happens during workflow
> execution:
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>
> HTH,
>
> Alek
>
> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>
>> Hi Chris,
>>
>> Great to hear OODT community will be interested in adopting a JSON based
>> workflow language and potentially a web based composer as well. Airavata
>> previously had BPEL support initially through a home grown implementation
>> [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper
>> [3] by Alek on this topic is an interesting read.
>>
>> I am of the same opinion that we should adopt something more modern as
>> the challenges from scientific workflows seems to be converging with the
>> data flow patterns in business workflows.
>>
>> It will be great if we can all compile a list of potential candidates and
>> hack them through.
>>
>> Suresh
>> [1] -
>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>> [2] -
>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>> [3] -
>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>
>>
>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>
>> > Hi Guys,
>> >
>> > I've been interested in this too - we don't per have a specific
>> > OODT workflow language, but we specific workflows using XML, and
>> > other configuration (we are also thinking of moving to JSON for
>> > this).
>> >
>> > In the past I've also looked at YAWL and BPEL - both seem complex
>> > to me.
>> >
>> > I wonder at the end of the day if we should adopt something more
>> > modern like PIG or some other data flow type of language (PIG
>> > is really neat).
>> >
>> > Cheers,
>> > Chris
>> >
>> >
>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > Chris Mattmann, Ph.D.
>> > Chief Architect
>> > Instrument Software and Science Data Systems Section (398)
>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > Office: 168-519, Mailstop: 168-527
>> > Email: chris.a.mattmann@nasa.gov
>> > WWW: http://sunset.usc.edu/~mattmann/
>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > Adjunct Associate Professor, Computer Science Department
>> > University of Southern California, Los Angeles, CA 90089 USA
>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >
>> >
>> >
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: Shameera Rathnayaka <sh...@gmail.com>
>> > Reply-To: "architecture@airavata.apache.org"
>> > <ar...@airavata.apache.org>
>> > Date: Thursday, September 18, 2014 8:26 AM
>> > To: "architecture@airavata.apache.org" <
>> architecture@airavata.apache.org>,
>> > dev <de...@airavata.apache.org>
>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>> >
>> >> Hi All,
>> >>
>> >> As we all know Airavata has its own workflow language call XWF. When
>> XWF
>> >> was introduced, main focus points are interoperability and
>> convertibility.
>> >> But with years of experience it is convinced that above requirements
>> are
>> >> not really useful when we come to real world use cases. And XWF is XML
>> >> based bulky language where we attache WSDLs and Workflow image it self.
>> >> But
>> >> with the recent changes WSDL part is being removed from XWF.
>> >>
>> >> It is worth to evaluate handy Scientific workflow languages in industry
>> >> and
>> >> find out pros and cons, at the end of this evaluation we need to come
>> up
>> >> with idea how we should improve Airavata workflow language, either we
>> can
>> >> improve existing XWF language, totally change to a new language
>> available
>> >> in industry or write a new light weight language. Basic requirements
>> that
>> >> we expect from new improvement are, high usability, flexible, light
>> weight
>> >> and real time monitoring support. As you can see above requirements are
>> >> not
>> >> direct comes with workflow languages but we need workflow language
>> which
>> >> help to support above requirements.
>> >>
>> >> After reading few papers and googling, initially i have come up with
>> >> following three existing languages,
>> >> 1. YAWL <http://www.yawlfoundation.org/>
>> >> 2. WS-BPEL
>> >> 3. SIDL
>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>> >>
>> >> In my opinion SIDL is more familiar with scientific domain,
>> Radical-SAGA
>> >> also uses slightly modified version of SIDL. Other than above three
>> >> languages we can come up with simple workflow language base on json(or
>> >> yaml) which support all our requirements for some extends.
>> >>
>> >> It would be grate if I can get more input regarding the $Subject form
>> the
>> >> airavata community. You all are more than welcome to provide any type
>> of
>> >> suggestions.
>> >>
>> >> Thanks,
>> >> Shameera.
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards,
>> >> Shameera Rathnayaka.
>> >>
>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>
>>
>
>
> --
> The best way to predict the future is to invent it - Alan Kay
>
--
Best Regards,
Shameera Rathnayaka.
email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by "Aleksander Slominski (NY)" <al...@gmail.com>.
Hi,
it is not dataflow instead focused on orchestrating REST services but you
may find it useful datapoint - we created worfklow service that uses
natively JavaScript and JSON to describe what happens during workflow
execution:
https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
HTH,
Alek
On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
> Hi Chris,
>
> Great to hear OODT community will be interested in adopting a JSON based
> workflow language and potentially a web based composer as well. Airavata
> previously had BPEL support initially through a home grown implementation
> [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper
> [3] by Alek on this topic is an interesting read.
>
> I am of the same opinion that we should adopt something more modern as the
> challenges from scientific workflows seems to be converging with the data
> flow patterns in business workflows.
>
> It will be great if we can all compile a list of potential candidates and
> hack them through.
>
> Suresh
> [1] -
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> [2] -
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
> [3] -
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>
>
> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
> > Hi Guys,
> >
> > I've been interested in this too - we don't per have a specific
> > OODT workflow language, but we specific workflows using XML, and
> > other configuration (we are also thinking of moving to JSON for
> > this).
> >
> > In the past I've also looked at YAWL and BPEL - both seem complex
> > to me.
> >
> > I wonder at the end of the day if we should adopt something more
> > modern like PIG or some other data flow type of language (PIG
> > is really neat).
> >
> > Cheers,
> > Chris
> >
> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Chris Mattmann, Ph.D.
> > Chief Architect
> > Instrument Software and Science Data Systems Section (398)
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 168-519, Mailstop: 168-527
> > Email: chris.a.mattmann@nasa.gov
> > WWW: http://sunset.usc.edu/~mattmann/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Adjunct Associate Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Shameera Rathnayaka <sh...@gmail.com>
> > Reply-To: "architecture@airavata.apache.org"
> > <ar...@airavata.apache.org>
> > Date: Thursday, September 18, 2014 8:26 AM
> > To: "architecture@airavata.apache.org" <architecture@airavata.apache.org
> >,
> > dev <de...@airavata.apache.org>
> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> >
> >> Hi All,
> >>
> >> As we all know Airavata has its own workflow language call XWF. When XWF
> >> was introduced, main focus points are interoperability and
> convertibility.
> >> But with years of experience it is convinced that above requirements are
> >> not really useful when we come to real world use cases. And XWF is XML
> >> based bulky language where we attache WSDLs and Workflow image it self.
> >> But
> >> with the recent changes WSDL part is being removed from XWF.
> >>
> >> It is worth to evaluate handy Scientific workflow languages in industry
> >> and
> >> find out pros and cons, at the end of this evaluation we need to come up
> >> with idea how we should improve Airavata workflow language, either we
> can
> >> improve existing XWF language, totally change to a new language
> available
> >> in industry or write a new light weight language. Basic requirements
> that
> >> we expect from new improvement are, high usability, flexible, light
> weight
> >> and real time monitoring support. As you can see above requirements are
> >> not
> >> direct comes with workflow languages but we need workflow language which
> >> help to support above requirements.
> >>
> >> After reading few papers and googling, initially i have come up with
> >> following three existing languages,
> >> 1. YAWL <http://www.yawlfoundation.org/>
> >> 2. WS-BPEL
> >> 3. SIDL
> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
> >>
> >> In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
> >> also uses slightly modified version of SIDL. Other than above three
> >> languages we can come up with simple workflow language base on json(or
> >> yaml) which support all our requirements for some extends.
> >>
> >> It would be grate if I can get more input regarding the $Subject form
> the
> >> airavata community. You all are more than welcome to provide any type of
> >> suggestions.
> >>
> >> Thanks,
> >> Shameera.
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Shameera Rathnayaka.
> >>
> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >> Blog : http://shameerarathnayaka.blogspot.com/
>
>
--
The best way to predict the future is to invent it - Alan Kay
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by "Aleksander Slominski (NY)" <al...@gmail.com>.
Hi,
it is not dataflow instead focused on orchestrating REST services but you
may find it useful datapoint - we created worfklow service that uses
natively JavaScript and JSON to describe what happens during workflow
execution:
https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
HTH,
Alek
On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
> Hi Chris,
>
> Great to hear OODT community will be interested in adopting a JSON based
> workflow language and potentially a web based composer as well. Airavata
> previously had BPEL support initially through a home grown implementation
> [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper
> [3] by Alek on this topic is an interesting read.
>
> I am of the same opinion that we should adopt something more modern as the
> challenges from scientific workflows seems to be converging with the data
> flow patterns in business workflows.
>
> It will be great if we can all compile a list of potential candidates and
> hack them through.
>
> Suresh
> [1] -
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> [2] -
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
> [3] -
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>
>
> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
> > Hi Guys,
> >
> > I've been interested in this too - we don't per have a specific
> > OODT workflow language, but we specific workflows using XML, and
> > other configuration (we are also thinking of moving to JSON for
> > this).
> >
> > In the past I've also looked at YAWL and BPEL - both seem complex
> > to me.
> >
> > I wonder at the end of the day if we should adopt something more
> > modern like PIG or some other data flow type of language (PIG
> > is really neat).
> >
> > Cheers,
> > Chris
> >
> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Chris Mattmann, Ph.D.
> > Chief Architect
> > Instrument Software and Science Data Systems Section (398)
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 168-519, Mailstop: 168-527
> > Email: chris.a.mattmann@nasa.gov
> > WWW: http://sunset.usc.edu/~mattmann/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Adjunct Associate Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Shameera Rathnayaka <sh...@gmail.com>
> > Reply-To: "architecture@airavata.apache.org"
> > <ar...@airavata.apache.org>
> > Date: Thursday, September 18, 2014 8:26 AM
> > To: "architecture@airavata.apache.org" <architecture@airavata.apache.org
> >,
> > dev <de...@airavata.apache.org>
> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> >
> >> Hi All,
> >>
> >> As we all know Airavata has its own workflow language call XWF. When XWF
> >> was introduced, main focus points are interoperability and
> convertibility.
> >> But with years of experience it is convinced that above requirements are
> >> not really useful when we come to real world use cases. And XWF is XML
> >> based bulky language where we attache WSDLs and Workflow image it self.
> >> But
> >> with the recent changes WSDL part is being removed from XWF.
> >>
> >> It is worth to evaluate handy Scientific workflow languages in industry
> >> and
> >> find out pros and cons, at the end of this evaluation we need to come up
> >> with idea how we should improve Airavata workflow language, either we
> can
> >> improve existing XWF language, totally change to a new language
> available
> >> in industry or write a new light weight language. Basic requirements
> that
> >> we expect from new improvement are, high usability, flexible, light
> weight
> >> and real time monitoring support. As you can see above requirements are
> >> not
> >> direct comes with workflow languages but we need workflow language which
> >> help to support above requirements.
> >>
> >> After reading few papers and googling, initially i have come up with
> >> following three existing languages,
> >> 1. YAWL <http://www.yawlfoundation.org/>
> >> 2. WS-BPEL
> >> 3. SIDL
> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
> >>
> >> In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
> >> also uses slightly modified version of SIDL. Other than above three
> >> languages we can come up with simple workflow language base on json(or
> >> yaml) which support all our requirements for some extends.
> >>
> >> It would be grate if I can get more input regarding the $Subject form
> the
> >> airavata community. You all are more than welcome to provide any type of
> >> suggestions.
> >>
> >> Thanks,
> >> Shameera.
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Shameera Rathnayaka.
> >>
> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >> Blog : http://shameerarathnayaka.blogspot.com/
>
>
--
The best way to predict the future is to invent it - Alan Kay
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by "Aleksander Slominski (NY)" <al...@gmail.com>.
Hi,
it is not dataflow instead focused on orchestrating REST services but you
may find it useful datapoint - we created worfklow service that uses
natively JavaScript and JSON to describe what happens during workflow
execution:
https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
HTH,
Alek
On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
> Hi Chris,
>
> Great to hear OODT community will be interested in adopting a JSON based
> workflow language and potentially a web based composer as well. Airavata
> previously had BPEL support initially through a home grown implementation
> [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper
> [3] by Alek on this topic is an interesting read.
>
> I am of the same opinion that we should adopt something more modern as the
> challenges from scientific workflows seems to be converging with the data
> flow patterns in business workflows.
>
> It will be great if we can all compile a list of potential candidates and
> hack them through.
>
> Suresh
> [1] -
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> [2] -
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
> [3] -
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>
>
> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
> > Hi Guys,
> >
> > I've been interested in this too - we don't per have a specific
> > OODT workflow language, but we specific workflows using XML, and
> > other configuration (we are also thinking of moving to JSON for
> > this).
> >
> > In the past I've also looked at YAWL and BPEL - both seem complex
> > to me.
> >
> > I wonder at the end of the day if we should adopt something more
> > modern like PIG or some other data flow type of language (PIG
> > is really neat).
> >
> > Cheers,
> > Chris
> >
> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Chris Mattmann, Ph.D.
> > Chief Architect
> > Instrument Software and Science Data Systems Section (398)
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 168-519, Mailstop: 168-527
> > Email: chris.a.mattmann@nasa.gov
> > WWW: http://sunset.usc.edu/~mattmann/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Adjunct Associate Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Shameera Rathnayaka <sh...@gmail.com>
> > Reply-To: "architecture@airavata.apache.org"
> > <ar...@airavata.apache.org>
> > Date: Thursday, September 18, 2014 8:26 AM
> > To: "architecture@airavata.apache.org" <architecture@airavata.apache.org
> >,
> > dev <de...@airavata.apache.org>
> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> >
> >> Hi All,
> >>
> >> As we all know Airavata has its own workflow language call XWF. When XWF
> >> was introduced, main focus points are interoperability and
> convertibility.
> >> But with years of experience it is convinced that above requirements are
> >> not really useful when we come to real world use cases. And XWF is XML
> >> based bulky language where we attache WSDLs and Workflow image it self.
> >> But
> >> with the recent changes WSDL part is being removed from XWF.
> >>
> >> It is worth to evaluate handy Scientific workflow languages in industry
> >> and
> >> find out pros and cons, at the end of this evaluation we need to come up
> >> with idea how we should improve Airavata workflow language, either we
> can
> >> improve existing XWF language, totally change to a new language
> available
> >> in industry or write a new light weight language. Basic requirements
> that
> >> we expect from new improvement are, high usability, flexible, light
> weight
> >> and real time monitoring support. As you can see above requirements are
> >> not
> >> direct comes with workflow languages but we need workflow language which
> >> help to support above requirements.
> >>
> >> After reading few papers and googling, initially i have come up with
> >> following three existing languages,
> >> 1. YAWL <http://www.yawlfoundation.org/>
> >> 2. WS-BPEL
> >> 3. SIDL
> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
> >>
> >> In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
> >> also uses slightly modified version of SIDL. Other than above three
> >> languages we can come up with simple workflow language base on json(or
> >> yaml) which support all our requirements for some extends.
> >>
> >> It would be grate if I can get more input regarding the $Subject form
> the
> >> airavata community. You all are more than welcome to provide any type of
> >> suggestions.
> >>
> >> Thanks,
> >> Shameera.
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Shameera Rathnayaka.
> >>
> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >> Blog : http://shameerarathnayaka.blogspot.com/
>
>
--
The best way to predict the future is to invent it - Alan Kay
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Suresh Marru <sm...@apache.org>.
Hi Chris,
Great to hear OODT community will be interested in adopting a JSON based workflow language and potentially a web based composer as well. Airavata previously had BPEL support initially through a home grown implementation [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper [3] by Alek on this topic is an interesting read.
I am of the same opinion that we should adopt something more modern as the challenges from scientific workflows seems to be converging with the data flow patterns in business workflows.
It will be great if we can all compile a list of potential candidates and hack them through.
Suresh
[1] - http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
[2] - http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
[3] - http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <ch...@jpl.nasa.gov> wrote:
> Hi Guys,
>
> I've been interested in this too - we don't per have a specific
> OODT workflow language, but we specific workflows using XML, and
> other configuration (we are also thinking of moving to JSON for
> this).
>
> In the past I've also looked at YAWL and BPEL - both seem complex
> to me.
>
> I wonder at the end of the day if we should adopt something more
> modern like PIG or some other data flow type of language (PIG
> is really neat).
>
> Cheers,
> Chris
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Shameera Rathnayaka <sh...@gmail.com>
> Reply-To: "architecture@airavata.apache.org"
> <ar...@airavata.apache.org>
> Date: Thursday, September 18, 2014 8:26 AM
> To: "architecture@airavata.apache.org" <ar...@airavata.apache.org>,
> dev <de...@airavata.apache.org>
> Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>
>> Hi All,
>>
>> As we all know Airavata has its own workflow language call XWF. When XWF
>> was introduced, main focus points are interoperability and convertibility.
>> But with years of experience it is convinced that above requirements are
>> not really useful when we come to real world use cases. And XWF is XML
>> based bulky language where we attache WSDLs and Workflow image it self.
>> But
>> with the recent changes WSDL part is being removed from XWF.
>>
>> It is worth to evaluate handy Scientific workflow languages in industry
>> and
>> find out pros and cons, at the end of this evaluation we need to come up
>> with idea how we should improve Airavata workflow language, either we can
>> improve existing XWF language, totally change to a new language available
>> in industry or write a new light weight language. Basic requirements that
>> we expect from new improvement are, high usability, flexible, light weight
>> and real time monitoring support. As you can see above requirements are
>> not
>> direct comes with workflow languages but we need workflow language which
>> help to support above requirements.
>>
>> After reading few papers and googling, initially i have come up with
>> following three existing languages,
>> 1. YAWL <http://www.yawlfoundation.org/>
>> 2. WS-BPEL
>> 3. SIDL
>> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>
>> In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
>> also uses slightly modified version of SIDL. Other than above three
>> languages we can come up with simple workflow language base on json(or
>> yaml) which support all our requirements for some extends.
>>
>> It would be grate if I can get more input regarding the $Subject form the
>> airavata community. You all are more than welcome to provide any type of
>> suggestions.
>>
>> Thanks,
>> Shameera.
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Suresh Marru <sm...@apache.org>.
Hi Chris,
Great to hear OODT community will be interested in adopting a JSON based workflow language and potentially a web based composer as well. Airavata previously had BPEL support initially through a home grown implementation [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper [3] by Alek on this topic is an interesting read.
I am of the same opinion that we should adopt something more modern as the challenges from scientific workflows seems to be converging with the data flow patterns in business workflows.
It will be great if we can all compile a list of potential candidates and hack them through.
Suresh
[1] - http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
[2] - http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
[3] - http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <ch...@jpl.nasa.gov> wrote:
> Hi Guys,
>
> I've been interested in this too - we don't per have a specific
> OODT workflow language, but we specific workflows using XML, and
> other configuration (we are also thinking of moving to JSON for
> this).
>
> In the past I've also looked at YAWL and BPEL - both seem complex
> to me.
>
> I wonder at the end of the day if we should adopt something more
> modern like PIG or some other data flow type of language (PIG
> is really neat).
>
> Cheers,
> Chris
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Shameera Rathnayaka <sh...@gmail.com>
> Reply-To: "architecture@airavata.apache.org"
> <ar...@airavata.apache.org>
> Date: Thursday, September 18, 2014 8:26 AM
> To: "architecture@airavata.apache.org" <ar...@airavata.apache.org>,
> dev <de...@airavata.apache.org>
> Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>
>> Hi All,
>>
>> As we all know Airavata has its own workflow language call XWF. When XWF
>> was introduced, main focus points are interoperability and convertibility.
>> But with years of experience it is convinced that above requirements are
>> not really useful when we come to real world use cases. And XWF is XML
>> based bulky language where we attache WSDLs and Workflow image it self.
>> But
>> with the recent changes WSDL part is being removed from XWF.
>>
>> It is worth to evaluate handy Scientific workflow languages in industry
>> and
>> find out pros and cons, at the end of this evaluation we need to come up
>> with idea how we should improve Airavata workflow language, either we can
>> improve existing XWF language, totally change to a new language available
>> in industry or write a new light weight language. Basic requirements that
>> we expect from new improvement are, high usability, flexible, light weight
>> and real time monitoring support. As you can see above requirements are
>> not
>> direct comes with workflow languages but we need workflow language which
>> help to support above requirements.
>>
>> After reading few papers and googling, initially i have come up with
>> following three existing languages,
>> 1. YAWL <http://www.yawlfoundation.org/>
>> 2. WS-BPEL
>> 3. SIDL
>> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>
>> In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
>> also uses slightly modified version of SIDL. Other than above three
>> languages we can come up with simple workflow language base on json(or
>> yaml) which support all our requirements for some extends.
>>
>> It would be grate if I can get more input regarding the $Subject form the
>> airavata community. You all are more than welcome to provide any type of
>> suggestions.
>>
>> Thanks,
>> Shameera.
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Bruce Barkstrom <br...@gmail.com>.
Workflows for either producing or using scientific data
may be very simple (observer looks at a thermometer
and records a temperature using a browser form - and
used to be writes down the temperature in a log book)
to very complex (cf the article in the newest CACM:
Greengard, S., 2014: Weathering a New Era of Big
Data, CACM, 57, No. 9, 12-14 which provides some
statistics on the number of data sources and such
that weather forecasting uses).
Unfortunately, the social groupings of practitioners
are often isolated - leading to misleading views of
how simple the workflows are. One of the critical
issues in dealing with this realistically is to figure
out when the production workflows create hierarchical
collections of data, usually because of different
versions in the production code. Bioinformatics
has almost no reprocessing and not much in the
way of versioning. Operational data collection
(weather and related things) has intermittent version
"glitches" - but not much reporcessing (reanalysis
projects take years). Climate data records involving
satellite instruments can have VERY complex
collection processes and versioning.
It's also important to be careful about the disciplinary
groupings of different kinds of data users. For example,
geological data archives can have rock cores turned
in by drilling teams in response to legal requirements
(which means prospectors may want property records
stated in English units and with local political jurisdictions),
while geologists may expect English units (if they're from
American petroleum firms), whereas academic geologists
may want MKS.
I expect choosing a language is going to be a long
and very complex social negotiation process.
Bruce B.
On Thu, Sep 18, 2014 at 1:15 PM, Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hi Guys,
>
> I've been interested in this too - we don't per have a specific
> OODT workflow language, but we specific workflows using XML, and
> other configuration (we are also thinking of moving to JSON for
> this).
>
> In the past I've also looked at YAWL and BPEL - both seem complex
> to me.
>
> I wonder at the end of the day if we should adopt something more
> modern like PIG or some other data flow type of language (PIG
> is really neat).
>
> Cheers,
> Chris
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Shameera Rathnayaka <sh...@gmail.com>
> Reply-To: "architecture@airavata.apache.org"
> <ar...@airavata.apache.org>
> Date: Thursday, September 18, 2014 8:26 AM
> To: "architecture@airavata.apache.org" <ar...@airavata.apache.org>,
> dev <de...@airavata.apache.org>
> Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>
> >Hi All,
> >
> >As we all know Airavata has its own workflow language call XWF. When XWF
> >was introduced, main focus points are interoperability and convertibility.
> >But with years of experience it is convinced that above requirements are
> >not really useful when we come to real world use cases. And XWF is XML
> >based bulky language where we attache WSDLs and Workflow image it self.
> >But
> >with the recent changes WSDL part is being removed from XWF.
> >
> >It is worth to evaluate handy Scientific workflow languages in industry
> >and
> >find out pros and cons, at the end of this evaluation we need to come up
> >with idea how we should improve Airavata workflow language, either we can
> >improve existing XWF language, totally change to a new language available
> >in industry or write a new light weight language. Basic requirements that
> >we expect from new improvement are, high usability, flexible, light weight
> >and real time monitoring support. As you can see above requirements are
> >not
> >direct comes with workflow languages but we need workflow language which
> >help to support above requirements.
> >
> >After reading few papers and googling, initially i have come up with
> >following three existing languages,
> >1. YAWL <http://www.yawlfoundation.org/>
> >2. WS-BPEL
> >3. SIDL
> ><http://computation.llnl.gov/casc/components/index.html#page=home>
> >
> >In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
> >also uses slightly modified version of SIDL. Other than above three
> >languages we can come up with simple workflow language base on json(or
> >yaml) which support all our requirements for some extends.
> >
> >It would be grate if I can get more input regarding the $Subject form the
> >airavata community. You all are more than welcome to provide any type of
> >suggestions.
> >
> >Thanks,
> >Shameera.
> >
> >
> >
> >--
> >Best Regards,
> >Shameera Rathnayaka.
> >
> >email: shameera AT apache.org , shameerainfo AT gmail.com
> >Blog : http://shameerarathnayaka.blogspot.com/
>
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by BW <bw...@mysoftcloud.com>.
Is there a list of graphical BEL workflow tools?
On Thursday, September 18, 2014, Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hi Guys,
>
> I've been interested in this too - we don't per have a specific
> OODT workflow language, but we specific workflows using XML, and
> other configuration (we are also thinking of moving to JSON for
> this).
>
> In the past I've also looked at YAWL and BPEL - both seem complex
> to me.
>
> I wonder at the end of the day if we should adopt something more
> modern like PIG or some other data flow type of language (PIG
> is really neat).
>
> Cheers,
> Chris
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov <javascript:;>
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Shameera Rathnayaka <shameerainfo@gmail.com <javascript:;>>
> Reply-To: "architecture@airavata.apache.org <javascript:;>"
> <architecture@airavata.apache.org <javascript:;>>
> Date: Thursday, September 18, 2014 8:26 AM
> To: "architecture@airavata.apache.org <javascript:;>" <
> architecture@airavata.apache.org <javascript:;>>,
> dev <dev@airavata.apache.org <javascript:;>>
> Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>
> >Hi All,
> >
> >As we all know Airavata has its own workflow language call XWF. When XWF
> >was introduced, main focus points are interoperability and convertibility.
> >But with years of experience it is convinced that above requirements are
> >not really useful when we come to real world use cases. And XWF is XML
> >based bulky language where we attache WSDLs and Workflow image it self.
> >But
> >with the recent changes WSDL part is being removed from XWF.
> >
> >It is worth to evaluate handy Scientific workflow languages in industry
> >and
> >find out pros and cons, at the end of this evaluation we need to come up
> >with idea how we should improve Airavata workflow language, either we can
> >improve existing XWF language, totally change to a new language available
> >in industry or write a new light weight language. Basic requirements that
> >we expect from new improvement are, high usability, flexible, light weight
> >and real time monitoring support. As you can see above requirements are
> >not
> >direct comes with workflow languages but we need workflow language which
> >help to support above requirements.
> >
> >After reading few papers and googling, initially i have come up with
> >following three existing languages,
> >1. YAWL <http://www.yawlfoundation.org/>
> >2. WS-BPEL
> >3. SIDL
> ><http://computation.llnl.gov/casc/components/index.html#page=home>
> >
> >In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
> >also uses slightly modified version of SIDL. Other than above three
> >languages we can come up with simple workflow language base on json(or
> >yaml) which support all our requirements for some extends.
> >
> >It would be grate if I can get more input regarding the $Subject form the
> >airavata community. You all are more than welcome to provide any type of
> >suggestions.
> >
> >Thanks,
> >Shameera.
> >
> >
> >
> >--
> >Best Regards,
> >Shameera Rathnayaka.
> >
> >email: shameera AT apache.org , shameerainfo AT gmail.com
> >Blog : http://shameerarathnayaka.blogspot.com/
>
>
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Posted by Suresh Marru <sm...@apache.org>.
Hi Chris,
Great to hear OODT community will be interested in adopting a JSON based workflow language and potentially a web based composer as well. Airavata previously had BPEL support initially through a home grown implementation [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper [3] by Alek on this topic is an interesting read.
I am of the same opinion that we should adopt something more modern as the challenges from scientific workflows seems to be converging with the data flow patterns in business workflows.
It will be great if we can all compile a list of potential candidates and hack them through.
Suresh
[1] - http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
[2] - http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
[3] - http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <ch...@jpl.nasa.gov> wrote:
> Hi Guys,
>
> I've been interested in this too - we don't per have a specific
> OODT workflow language, but we specific workflows using XML, and
> other configuration (we are also thinking of moving to JSON for
> this).
>
> In the past I've also looked at YAWL and BPEL - both seem complex
> to me.
>
> I wonder at the end of the day if we should adopt something more
> modern like PIG or some other data flow type of language (PIG
> is really neat).
>
> Cheers,
> Chris
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Shameera Rathnayaka <sh...@gmail.com>
> Reply-To: "architecture@airavata.apache.org"
> <ar...@airavata.apache.org>
> Date: Thursday, September 18, 2014 8:26 AM
> To: "architecture@airavata.apache.org" <ar...@airavata.apache.org>,
> dev <de...@airavata.apache.org>
> Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>
>> Hi All,
>>
>> As we all know Airavata has its own workflow language call XWF. When XWF
>> was introduced, main focus points are interoperability and convertibility.
>> But with years of experience it is convinced that above requirements are
>> not really useful when we come to real world use cases. And XWF is XML
>> based bulky language where we attache WSDLs and Workflow image it self.
>> But
>> with the recent changes WSDL part is being removed from XWF.
>>
>> It is worth to evaluate handy Scientific workflow languages in industry
>> and
>> find out pros and cons, at the end of this evaluation we need to come up
>> with idea how we should improve Airavata workflow language, either we can
>> improve existing XWF language, totally change to a new language available
>> in industry or write a new light weight language. Basic requirements that
>> we expect from new improvement are, high usability, flexible, light weight
>> and real time monitoring support. As you can see above requirements are
>> not
>> direct comes with workflow languages but we need workflow language which
>> help to support above requirements.
>>
>> After reading few papers and googling, initially i have come up with
>> following three existing languages,
>> 1. YAWL <http://www.yawlfoundation.org/>
>> 2. WS-BPEL
>> 3. SIDL
>> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>
>> In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
>> also uses slightly modified version of SIDL. Other than above three
>> languages we can come up with simple workflow language base on json(or
>> yaml) which support all our requirements for some extends.
>>
>> It would be grate if I can get more input regarding the $Subject form the
>> airavata community. You all are more than welcome to provide any type of
>> suggestions.
>>
>> Thanks,
>> Shameera.
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/