You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@airavata.apache.org by Shameera Rathnayaka <sh...@gmail.com> on 2014/10/17 00:25:20 UTC

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Hi,

First of all thanks everyone for giving valuable inputs. After doing some
background search and talking to different people in the University who has
used different workflow languages, I myself convinced that introducing an
another workflow language is not what actually they need. By changing
exiting workflow language to another will not solve problems. What they
asking is a easy way to construct the workflows. Indirectly what they
asking for a sort of API which they can use to generate the workflows and
run it. Correct me if i am wrong here.

As most of above replies depict,  if we can get a simple API, as an
example, for a web based application, JavaScript API would be a good
solution, and probably JSON would be a good candidate for language, instead
of XML.

Airavata community already have started to implement web base GUI. Hence
introducing a JSON base JavaScript API would be great help. WDYT?

Thanks,
Shameera.


On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
alekny7@gmail.com> wrote:

> Hi,
>
> it is not dataflow instead focused on orchestrating REST services but you
> may find it useful datapoint - we created worfklow service that uses
> natively JavaScript and JSON to describe what happens during workflow
> execution:
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>
> HTH,
>
> Alek
>
> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>
>> Hi Chris,
>>
>> Great to hear OODT community will be interested in adopting a JSON based
>> workflow language and potentially a web based composer as well. Airavata
>> previously had BPEL support initially through a home grown implementation
>> [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper
>> [3] by Alek on this topic is an interesting read.
>>
>> I am of the same opinion that we should adopt something more modern as
>> the challenges from scientific workflows seems to be converging with the
>> data flow patterns in business workflows.
>>
>> It will be great if we can all compile a list of potential candidates and
>> hack them through.
>>
>> Suresh
>> [1] -
>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>> [2] -
>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>> [3] -
>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>
>>
>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>
>> > Hi Guys,
>> >
>> > I've been interested in this too - we don't per have a specific
>> > OODT workflow language, but we specific workflows using XML, and
>> > other configuration (we are also thinking of moving to JSON for
>> > this).
>> >
>> > In the past I've also looked at YAWL and BPEL - both seem complex
>> > to me.
>> >
>> > I wonder at the end of the day if we should adopt something more
>> > modern like PIG or some other data flow type of language (PIG
>> > is really neat).
>> >
>> > Cheers,
>> > Chris
>> >
>> >
>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > Chris Mattmann, Ph.D.
>> > Chief Architect
>> > Instrument Software and Science Data Systems Section (398)
>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > Office: 168-519, Mailstop: 168-527
>> > Email: chris.a.mattmann@nasa.gov
>> > WWW:  http://sunset.usc.edu/~mattmann/
>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > Adjunct Associate Professor, Computer Science Department
>> > University of Southern California, Los Angeles, CA 90089 USA
>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >
>> >
>> >
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: Shameera Rathnayaka <sh...@gmail.com>
>> > Reply-To: "architecture@airavata.apache.org"
>> > <ar...@airavata.apache.org>
>> > Date: Thursday, September 18, 2014 8:26 AM
>> > To: "architecture@airavata.apache.org" <
>> architecture@airavata.apache.org>,
>> > dev <de...@airavata.apache.org>
>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>> >
>> >> Hi All,
>> >>
>> >> As we all know Airavata has its own workflow language call XWF. When
>> XWF
>> >> was introduced, main focus points are interoperability and
>> convertibility.
>> >> But with years of experience it is convinced that above requirements
>> are
>> >> not really useful when we come to real world use cases. And XWF is XML
>> >> based bulky language where we attache WSDLs and Workflow image it self.
>> >> But
>> >> with the recent changes WSDL part is being removed from XWF.
>> >>
>> >> It is worth to evaluate handy Scientific workflow languages in industry
>> >> and
>> >> find out pros and cons, at the end of this evaluation we need to come
>> up
>> >> with idea how we should improve Airavata workflow language, either we
>> can
>> >> improve existing XWF language, totally change to a new language
>> available
>> >> in industry or write a new light weight language. Basic requirements
>> that
>> >> we expect from new improvement are, high usability, flexible, light
>> weight
>> >> and real time monitoring support. As you can see above requirements are
>> >> not
>> >> direct comes with workflow languages but we need workflow language
>> which
>> >> help to support above requirements.
>> >>
>> >> After reading few papers and googling, initially i have come up with
>> >> following three existing languages,
>> >> 1. YAWL <http://www.yawlfoundation.org/>
>> >> 2. WS-BPEL
>> >> 3. SIDL
>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>> >>
>> >> In my opinion SIDL is more familiar with scientific domain,
>> Radical-SAGA
>> >> also uses slightly modified version of SIDL. Other than above three
>> >> languages we can come up with simple workflow language base on json(or
>> >> yaml) which support all our requirements for some extends.
>> >>
>> >> It would be grate if I can get more input regarding the $Subject form
>> the
>> >> airavata community. You all are more than welcome to provide any type
>> of
>> >> suggestions.
>> >>
>> >> Thanks,
>> >> Shameera.
>> >>
>> >> 
>> >>
>> >> --
>> >> Best Regards,
>> >> Shameera Rathnayaka.
>> >>
>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>
>>
>
>
> --
> The best way to predict the future is to invent it - Alan Kay
>



-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Amila Jayasekara <th...@gmail.com>.

Hi Shameera,

Thanks for the reply.

I wouldnt call XWF a "language". It is not a language in any sense but an
intermediate representation of a workflow graph.
To my understanding what you need is not a language but an efficient
portable intermediate representation. Seems you also have realized it. JSON
might be a good candidate provided most of the frontends are web based.

Thanks
-Thejaka

On Mon, Oct 20, 2014 at 10:35 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:

> Hi Amila,
>
> Please see my comments inline.
>
> On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara <thejaka.amila@gmail.com
> >
> wrote:
>
> > Hello,
> >
> > I am sorry, I am bit late on this thread. But when reading through this
> > thread I simply got lost, what this thread is discussing. I have few
> > questions.
> >
> > 1. @Shameera : Is XWF actually a language to define workflow ? To my
> > understanding it was an intermediate representation to convert workflow
> > defined in UI to java object model. Was XWF ever used by any airavata
> user
> > to define a workflow graph ?
> >
>
> Yes, XWF is the language defined and used by Airavata  to explain the
> workflows but it is not well documented. Both server and client sides read
> this description language to generate runtime java representation.  so
> when you used XBaya to create a workflow with multiple applications, under
> the hood XBaya generate xwf which describe that workflow to server.
>
>
>
> >
> > From initial description what I understood is we are looking for a
> > improved "intermediate representation", not a language which describes
> > workflows.
> > 
> >
> >
>
> > 2. So what is the exact purpose of this proposed language ?
> >         - Is it to hide parallelism in workflows ?
> >         - Is it to replace the XBAYA functionalities ? (i hope not)
> >
>
>
> Actually initial idea was to identify good, well-defined and scientific
> domain friendly language. Whole purpose of this effort is reduce the entry
> barrier of the end user. But later it is understood that introducing a new
> language won't fix our issue.
>
>
> >
> > 3. What are we trying to achieve by this proposed language which we
> cannot
> > achieved through workflow UI tool ?
> >
> > 4. Who is going to use this language ?
> >
>
> As I explained, our direction has been changed.By introducing a new
> language we are get nothing but nice description file. No functional
> improvements etc ... The current language should use all airavata
> client(Currently we have only XBaya) side applications to explain the
> workflow to the server side.
>
>
>
> >
> > 5. Why would a user prefer (assuming intended audience of proposed
> > language is end users) a language over a Work Flow UI tool such as XBAYA
> ?
> > (In other words what are the things we can do with language which we
> cannot
> > do with UI)
> >
>
> Let's say you are going to write a new web base client for Airavata which
> generate workflows and launch it. What you need to do is do some magic
> with UI and finally come up with description language and send it to server
> side. Here you need to learn how to write a valid XWF file and write your
> own JS code to generate it. But if airavata has a JavaScript API which can
> be used to generate XWF for you by getting JSON objects as input then it
> would be great help for client side developers.
>
> Thank,
> Shameera.
>
>
>
> >
> > Sorry, if above questions are in-appropriate, just wanted to understand
> > what exactly needed.
> >
> > Thanks
> > -Thejaka Amila
> >
> >
> >
> > On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
> > wrote:
> >
> >> I think I'm not suggesting to create a Workflow interpreter using Python
> >> etc. What I'm suggesting is to remove the Worflow aspect from core
> Airavata
> >> and move it to a more higher level component. The more I think about it,
> >> the model I'm suggesting is similar to what Hadoop, Storm etc has done
> for
> >> distributed system computations. This model is proven to be successful
> over
> >> the years.
> >>
> >> Keeping what Airavata does at its core can help you to build a more
> >> robust system. If we look at Airavata as middleware to execute
> applications
> >> on computing resources we can simplify what Airavata does and focus on
> >> improving the core functionality. All the successful systems have
> thrived
> >> on defining what it does at its core and keeping it simple and being
> >> excellent at what it does. In that regard keeping workflow aspect out of
> >> Airavata can help you to focus on the core problem. That is to execute
> an
> >> application on a remote computing resource in a fault tolerant and
> scalable
> >> manner.
> >>
> >> What I'm suggesting is to give the Orchestrator the capability to
> execute
> >> a Driver program that is specified by the user. (This program can be
> >> written in Python, Java or any other language). This driver program is
> >> similar to what you define in a Hadoop or Storm configuration. The
> driver
> >> program specifies the flow of the computation. It specifies what are the
> >> applications needs to be executed, how to manipulate input output. The
> >> driver program is the workflow for the user. Because the user specifies
> the
> >> program he can program it to handle workflow steering etc. Every time
> the
> >> user wants to execute this program he can tell Airavata to execute the
> >> Driver program.
> >>
> >> I'm also not 100% sure about all the details. But this can be a new way
> >> to look at how systems like Airavata should behave. Your thoughts and
> >> suggestions are more than welcome.
> >>
> >> Thanks,
> >> Supun..
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
> >> shameerainfo@gmail.com> wrote:
> >>
> >>> Hi Supun,
> >>>
> >>> I think we were in two context, because I as suggesting a way to
> >>> serialize
> >>> and deserialize the workflow description while you are suggesting to
> >>> implement
> >>> some kind of workflow interpreter using Python, where Python client can
> >>> send
> >>> thrift calls to Airavata server to run the application. I can see with
> >>> your
> >>> suggested
> >>> approach we can control the workflow execution process from client side
> >>> which
> >>> make it easy to implement workflow debugger.As you mentioned this is a
> >>> major change
> >>> to Airavata. So we should neatly think as this will change our existing
> >>> architecture.
> >>>
> >>> Still if someone need to use different language java, php, JS etc ...
> to
> >>> run the same
> >>> workflow which generated by Python, we need a language independent
> >>> workflow
> >>> description.
> >>> My initial question was what is the best language for this?. But as I
> >>> have
> >>> explained in
> >>> one of my previous reply, It is not matter what language we used Either
> >>> we
> >>> can use
> >>> XML or JSON to write this description, what matters is how easy to
> >>> generate
> >>> workflow with the provided API. Hence it would be great to have set of
> >>> neat
> >>> APIs in
> >>> different languages.
> >>>
> >>> Thanks,
> >>> Shameera.
> >>>
> >>>
> >>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <
> supun06@gmail.com
> >>> >
> >>> wrote:
> >>>
> >>> > Hi Shameera,
> >>> >
> >>> > Using python is a radical approach for workflow I think. But I
> believe
> >>> this
> >>> > is a very beautiful and new approach.Here is a possible scenario for
> >>> > implementation and running using a Python based library.
> >>> >
> >>> > The Python library facilitates the creation of Applications and
> >>> submitting
> >>> > them to Airavata for execution. The underlying functionality is
> exactly
> >>> > similar to what Java clients provides.The only difference is that,
> >>> Python
> >>> > library should have a more fluent API than Java for easy creation of
> >>> > workflows. We can generate the Python clients that talk to Airavata
> >>> server
> >>> > using Thrift.
> >>> >
> >>> > Here is an example off the top of my head to a Python script created
> by
> >>> > user for a Workflow. This is a very crude example and we need to come
> >>> up
> >>> > with a much better API if we are going to go along this path. First
> we
> >>> need
> >>> > to write a Python script that can execute a workflow using Airavata.
> >>> >
> >>> > import airavata
> >>> >
> >>> > host = Host("localhost", ....)
> >>> > app1 = Application(host, ....)
> >>> > app2 = Application(host, ....)
> >>> >
> >>> > # we will connect these applications as a workflow using some
> topology
> >>> > builder or other constructs
> >>> >
> >>> > wb = WorkFlowBuilder()
> >>> > wb.setApp("name1", app1)
> >>> > # you can do a simple output transformation here etc
> >>> >
> >>> > # connects the input of app1 to app2 etc
> >>> > wb.setApp("name2", app2).connectInput("name1")
> >>> >
> >>> > wb.submit()
> >>> >
> >>> > Now we can load this Python script from XBaya. When XBaya loads this
> >>> script
> >>> > the Python script can output an XML configuration of the topology,
> >>> XBaya
> >>> > can render. There are other ways like directly executing the Python
> >>> script
> >>> > from command line and connecting XBaya indirectly as well. Now you
> can
> >>> run
> >>> > the workflow from XBaya. Running the Workflow means just executing
> the
> >>> > Python script.
> >>> >
> >>> > XBaya gets the notifications through messaging and update the UI
> >>> > accordingly.
> >>> >
> >>> > The users need to write the Python script by hand. XBaya cannot
> create
> >>> the
> >>> > script. Because workflow language is an actual python program the
> >>> benefits
> >>> > are immense. For example user can do workflow steering in the
> workflow
> >>> > itself by subscribing to messages from Airavata.
> >>> >
> >>> > Thanks,
> >>> > Supun..
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> >>> > shameerainfo@gmail.com> wrote:
> >>> >
> >>> > > Hi Supun,
> >>> > >
> >>> > > I meant to say JS is a well-known client side scripting language i
> >>> have
> >>> > > messed scripting part. Even we use Python, ultimately we should
> >>> convert
> >>> > > this to java model in server side, somehow we need to serialized
> >>> python
> >>> > > representation to the language which java can parse and generate
> that
> >>> > > model. In this case we need to parse python script in java isn't
> it?
> >>> I am
> >>> > > not exactly clear how you suggesting to use python here. More
> >>> details on
> >>> > > how end system works if we use Python would be great help to
> clearly
> >>> > > understand your points.
> >>> > >
> >>> > > Thanks,
> >>> > > Shameera.
> >>> > >
> >>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> >>> > chris.mattmann@gmail.com>
> >>> > > wrote:
> >>> > >
> >>> > > > Have you guys considered using JCC [1] as a means
> >>> > > > to expose the workflow API currently in Java as a
> >>> > > > Python API?
> >>> > > >
> >>> > > > We are exploring its use in OODT, and we have already
> >>> > > > created a Tika [2] JCC-based python API.
> >>> > > >
> >>> > > > Cheers,
> >>> > > > Chris
> >>> > > >
> >>> > > > [1] http://lucene.apache.org/pylucene/jcc/
> >>> > > > [2] http://github.com/chrismattmann/tika-python/
> >>> > > >
> >>> > > > ------------------------
> >>> > > > Chris Mattmann
> >>> > > > chris.mattmann@gmail.com
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > -----Original Message-----
> >>> > > > From: Supun Kamburugamuva <su...@gmail.com>
> >>> > > > Reply-To: <de...@oodt.apache.org>
> >>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
> >>> > > > To: dev <de...@airavata.apache.org>
> >>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> >>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
> >>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> >>> > > > <de...@oodt.apache.org>
> >>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> >>> > Airavata.
> >>> > > >
> >>> > > > >Once we had an offline discussion about the Airavata Workflow
> >>> language
> >>> > > > >(with Milinda, Saliya and Shameera). In that discussion one
> thing
> >>> came
> >>> > > out
> >>> > > > >was why we need to invent a different language when a simple
> >>> library
> >>> > > like
> >>> > > > >Python will full fill of Airavata requirements.
> >>> > > > >
> >>> > > > >There are many benefits in using a Python library as the API for
> >>> > > > >controlling Airavata workflows.
> >>> > > > >
> >>> > > > >1. It is a library, gives the ultimate control over the
> execution
> >>> and
> >>> > it
> >>> > > > >can be simpler than any domain specific language that we can
> come
> >>> with
> >>> > > > >like
> >>> > > > >XML, JSON etc
> >>> > > > >2. Most people use python and can learn it easily than any
> Domain
> >>> > > specific
> >>> > > > >language
> >>> > > > >3. You can easily create a Python library for Airavata because
> >>> all the
> >>> > > > >APIs
> >>> > > > >are thrift based.
> >>> > > > >4. If you design the constructs correctly you can plug an XBaya.
> >>> > > > >
> >>> > > > >Any thoughts?
> >>> > > > >
> >>> > > > >Thanks,
> >>> > > > >Supun..
> >>> > > > >
> >>> > > > >
> >>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> >>> > supun06@gmail.com
> >>> > > >
> >>> > > > >wrote:
> >>> > > > >
> >>> > > > >> Hi Shameera,
> >>> > > > >>
> >>> > > > >> Why you prefer JavaScript over a language like Python?
> >>> > > > >>
> >>> > > > >> Thanks,
> >>> > > > >> Supun..
> >>> > > > >>
> >>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> >>> > > > >> shameerainfo@gmail.com> wrote:
> >>> > > > >>
> >>> > > > >>> Hi,
> >>> > > > >>>
> >>> > > > >>> First of all thanks everyone for giving valuable inputs.
> After
> >>> > doing
> >>> > > > >>>some
> >>> > > > >>> background search and talking to different people in the
> >>> University
> >>> > > > >>>who has
> >>> > > > >>> used different workflow languages, I myself convinced that
> >>> > > introducing
> >>> > > > >>>an
> >>> > > > >>> another workflow language is not what actually they need. By
> >>> > changing
> >>> > > > >>> exiting workflow language to another will not solve problems.
> >>> What
> >>> > > they
> >>> > > > >>> asking is a easy way to construct the workflows. Indirectly
> >>> what
> >>> > they
> >>> > > > >>> asking for a sort of API which they can use to generate the
> >>> > workflows
> >>> > > > >>>and
> >>> > > > >>> run it. Correct me if i am wrong here.
> >>> > > > >>>
> >>> > > > >>> As most of above replies depict,  if we can get a simple API,
> >>> as an
> >>> > > > >>> example, for a web based application, JavaScript API would
> be a
> >>> > good
> >>> > > > >>> solution, and probably JSON would be a good candidate for
> >>> language,
> >>> > > > >>>instead
> >>> > > > >>> of XML.
> >>> > > > >>>
> >>> > > > >>> Airavata community already have started to implement web base
> >>> GUI.
> >>> > > > >>>Hence
> >>> > > > >>> introducing a JSON base JavaScript API would be great help.
> >>> WDYT?
> >>> > > > >>>
> >>> > > > >>> Thanks,
> >>> > > > >>> Shameera.
> >>> > > > >>>
> >>> > > > >>>
> >>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >>> > > > >>> alekny7@gmail.com> wrote:
> >>> > > > >>>
> >>> > > > >>>> Hi,
> >>> > > > >>>>
> >>> > > > >>>> it is not dataflow instead focused on orchestrating REST
> >>> services
> >>> > > but
> >>> > > > >>>> you may find it useful datapoint - we created worfklow
> service
> >>> > that
> >>> > > > >>>>uses
> >>> > > > >>>> natively JavaScript and JSON to describe what happens during
> >>> > > workflow
> >>> > > > >>>> execution:
> >>> > > > >>>>
> >>> > > >
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>> > > > >>>>
> >>> > > > >>>> HTH,
> >>> > > > >>>>
> >>> > > > >>>> Alek
> >>> > > > >>>>
> >>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
> >>> smarru@apache.org>
> >>> > > > >>>>wrote:
> >>> > > > >>>>
> >>> > > > >>>>> Hi Chris,
> >>> > > > >>>>>
> >>> > > > >>>>> Great to hear OODT community will be interested in
> adopting a
> >>> > JSON
> >>> > > > >>>>> based workflow language and potentially a web based
> composer
> >>> as
> >>> > > well.
> >>> > > > >>>>> Airavata previously had BPEL support initially through a
> home
> >>> > grown
> >>> > > > >>>>> implementation [1] by Alek Slominski and later through
> >>> Apache ODE
> >>> > > > >>>>>[2]. Also
> >>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
> >>> read.
> >>> > > > >>>>>
> >>> > > > >>>>> I am of the same opinion that we should adopt something
> more
> >>> > modern
> >>> > > > >>>>>as
> >>> > > > >>>>> the challenges from scientific workflows seems to be
> >>> converging
> >>> > > with
> >>> > > > >>>>>the
> >>> > > > >>>>> data flow patterns in business workflows.
> >>> > > > >>>>>
> >>> > > > >>>>> It will be great if we can all compile a list of potential
> >>> > > candidates
> >>> > > > >>>>> and hack them through.
> >>> > > > >>>>>
> >>> > > > >>>>> Suresh
> >>> > > > >>>>> [1] -
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > >
> >>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>> > > > >>>>> [2] -
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > >
> >>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> >>> > > > >>>>>time_for_eScience_workflows
> >>> > > > >>>>> [3] -
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > >
> >>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> >>> > > > >>>>>6.pdf
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>> > > > >>>>>
> >>> > > > >>>>> > Hi Guys,
> >>> > > > >>>>> >
> >>> > > > >>>>> > I've been interested in this too - we don't per have a
> >>> specific
> >>> > > > >>>>> > OODT workflow language, but we specific workflows using
> >>> XML,
> >>> > and
> >>> > > > >>>>> > other configuration (we are also thinking of moving to
> >>> JSON for
> >>> > > > >>>>> > this).
> >>> > > > >>>>> >
> >>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
> >>> > complex
> >>> > > > >>>>> > to me.
> >>> > > > >>>>> >
> >>> > > > >>>>> > I wonder at the end of the day if we should adopt
> something
> >>> > more
> >>> > > > >>>>> > modern like PIG or some other data flow type of language
> >>> (PIG
> >>> > > > >>>>> > is really neat).
> >>> > > > >>>>> >
> >>> > > > >>>>> > Cheers,
> >>> > > > >>>>> > Chris
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> > > > >>>>> > Chris Mattmann, Ph.D.
> >>> > > > >>>>> > Chief Architect
> >>> > > > >>>>> > Instrument Software and Science Data Systems Section
> (398)
> >>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
> >>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
> >>> > > > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
> >>> > > > >>>>> >
> >>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
> >>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089
> >>> USA
> >>> > > > >>>>> >
> >>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> > -----Original Message-----
> >>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
> >>> > > > >>>>> > <ar...@airavata.apache.org>
> >>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>> > > > >>>>> > To: "architecture@airavata.apache.org" <
> >>> > > > >>>>> architecture@airavata.apache.org>,
> >>> > > > >>>>> > dev <de...@airavata.apache.org>
> >>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language
> for
> >>> > > > >>>>>Airavata.
> >>> > > > >>>>> >
> >>> > > > >>>>> >> Hi All,
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> As we all know Airavata has its own workflow language
> call
> >>> > XWF.
> >>> > > > >>>>>When
> >>> > > > >>>>> XWF
> >>> > > > >>>>> >> was introduced, main focus points are interoperability
> and
> >>> > > > >>>>> convertibility.
> >>> > > > >>>>> >> But with years of experience it is convinced that above
> >>> > > > >>>>>requirements
> >>> > > > >>>>> are
> >>> > > > >>>>> >> not really useful when we come to real world use cases.
> >>> And
> >>> > XWF
> >>> > > is
> >>> > > > >>>>> XML
> >>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
> >>> image
> >>> > > it
> >>> > > > >>>>> self.
> >>> > > > >>>>> >> But
> >>> > > > >>>>> >> with the recent changes WSDL part is being removed from
> >>> XWF.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
> >>> languages in
> >>> > > > >>>>> industry
> >>> > > > >>>>> >> and
> >>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
> >>> need
> >>> > to
> >>> > > > >>>>> come up
> >>> > > > >>>>> >> with idea how we should improve Airavata workflow
> >>> language,
> >>> > > either
> >>> > > > >>>>> we can
> >>> > > > >>>>> >> improve existing XWF language, totally change to a new
> >>> > language
> >>> > > > >>>>> available
> >>> > > > >>>>> >> in industry or write a new light weight language. Basic
> >>> > > > >>>>>requirements
> >>> > > > >>>>> that
> >>> > > > >>>>> >> we expect from new improvement are, high usability,
> >>> flexible,
> >>> > > > >>>>>light
> >>> > > > >>>>> weight
> >>> > > > >>>>> >> and real time monitoring support. As you can see above
> >>> > > > >>>>>requirements
> >>> > > > >>>>> are
> >>> > > > >>>>> >> not
> >>> > > > >>>>> >> direct comes with workflow languages but we need
> workflow
> >>> > > language
> >>> > > > >>>>> which
> >>> > > > >>>>> >> help to support above requirements.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> After reading few papers and googling, initially i have
> >>> come
> >>> > up
> >>> > > > >>>>>with
> >>> > > > >>>>> >> following three existing languages,
> >>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>> > > > >>>>> >> 2. WS-BPEL
> >>> > > > >>>>> >> 3. SIDL
> >>> > > > >>>>> >> <
> >>> > > http://computation.llnl.gov/casc/components/index.html#page=home
> >>> > > > >
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific
> >>> domain,
> >>> > > > >>>>> Radical-SAGA
> >>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
> >>> above
> >>> > > > >>>>>three
> >>> > > > >>>>> >> languages we can come up with simple workflow language
> >>> base on
> >>> > > > >>>>> json(or
> >>> > > > >>>>> >> yaml) which support all our requirements for some
> extends.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> It would be grate if I can get more input regarding the
> >>> > $Subject
> >>> > > > >>>>> form the
> >>> > > > >>>>> >> airavata community. You all are more than welcome to
> >>> provide
> >>> > any
> >>> > > > >>>>> type of
> >>> > > > >>>>> >> suggestions.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> Thanks,
> >>> > > > >>>>> >> Shameera.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> 
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> --
> >>> > > > >>>>> >> Best Regards,
> >>> > > > >>>>> >> Shameera Rathnayaka.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT
> gmail.com
> >>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > > >>>>
> >>> > > > >>>>
> >>> > > > >>>> --
> >>> > > > >>>> The best way to predict the future is to invent it - Alan
> Kay
> >>> > > > >>>>
> >>> > > > >>>
> >>> > > > >>>
> >>> > > > >>>
> >>> > > > >>> --
> >>> > > > >>> Best Regards,
> >>> > > > >>> Shameera Rathnayaka.
> >>> > > > >>>
> >>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
> >>> > > > >>>
> >>> > > > >>
> >>> > > > >>
> >>> > > > >>
> >>> > > > >> --
> >>> > > > >> Supun Kamburugamuva
> >>> > > > >> Member, Apache Software Foundation; http://www.apache.org
> >>> > > > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >>> > > > >> Blog: http://supunk.blogspot.com
> >>> > > > >>
> >>> > > > >>
> >>> > > > >
> >>> > > > >
> >>> > > > >--
> >>> > > > >Supun Kamburugamuva
> >>> > > > >Member, Apache Software Foundation; http://www.apache.org
> >>> > > > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >>> > > > >Blog: http://supunk.blogspot.com
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > >
> >>> > >
> >>> > > --
> >>> > > Best Regards,
> >>> > > Shameera Rathnayaka.
> >>> > >
> >>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> > > Blog : http://shameerarathnayaka.blogspot.com/
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Supun Kamburugamuva
> >>> > Member, Apache Software Foundation; http://www.apache.org
> >>> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >>> > Blog: http://supunk.blogspot.com
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards,
> >>> Shameera Rathnayaka.
> >>>
> >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> Blog : http://shameerarathnayaka.blogspot.com/
> >>>
> >>
> >>
> >>
> >> --
> >> Supun Kamburugamuva
> >> Member, Apache Software Foundation; http://www.apache.org
> >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >> Blog: http://supunk.blogspot.com
> >>
> >>
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Amila Jayasekara <th...@gmail.com>.

Hi Shameera,

Thanks for the reply.

I wouldnt call XWF a "language". It is not a language in any sense but an
intermediate representation of a workflow graph.
To my understanding what you need is not a language but an efficient
portable intermediate representation. Seems you also have realized it. JSON
might be a good candidate provided most of the frontends are web based.

Thanks
-Thejaka

On Mon, Oct 20, 2014 at 10:35 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:

> Hi Amila,
>
> Please see my comments inline.
>
> On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara <thejaka.amila@gmail.com
> >
> wrote:
>
> > Hello,
> >
> > I am sorry, I am bit late on this thread. But when reading through this
> > thread I simply got lost, what this thread is discussing. I have few
> > questions.
> >
> > 1. @Shameera : Is XWF actually a language to define workflow ? To my
> > understanding it was an intermediate representation to convert workflow
> > defined in UI to java object model. Was XWF ever used by any airavata
> user
> > to define a workflow graph ?
> >
>
> Yes, XWF is the language defined and used by Airavata  to explain the
> workflows but it is not well documented. Both server and client sides read
> this description language to generate runtime java representation.  so
> when you used XBaya to create a workflow with multiple applications, under
> the hood XBaya generate xwf which describe that workflow to server.
>
>
>
> >
> > From initial description what I understood is we are looking for a
> > improved "intermediate representation", not a language which describes
> > workflows.
> > 
> >
> >
>
> > 2. So what is the exact purpose of this proposed language ?
> >         - Is it to hide parallelism in workflows ?
> >         - Is it to replace the XBAYA functionalities ? (i hope not)
> >
>
>
> Actually initial idea was to identify good, well-defined and scientific
> domain friendly language. Whole purpose of this effort is reduce the entry
> barrier of the end user. But later it is understood that introducing a new
> language won't fix our issue.
>
>
> >
> > 3. What are we trying to achieve by this proposed language which we
> cannot
> > achieved through workflow UI tool ?
> >
> > 4. Who is going to use this language ?
> >
>
> As I explained, our direction has been changed.By introducing a new
> language we are get nothing but nice description file. No functional
> improvements etc ... The current language should use all airavata
> client(Currently we have only XBaya) side applications to explain the
> workflow to the server side.
>
>
>
> >
> > 5. Why would a user prefer (assuming intended audience of proposed
> > language is end users) a language over a Work Flow UI tool such as XBAYA
> ?
> > (In other words what are the things we can do with language which we
> cannot
> > do with UI)
> >
>
> Let's say you are going to write a new web base client for Airavata which
> generate workflows and launch it. What you need to do is do some magic
> with UI and finally come up with description language and send it to server
> side. Here you need to learn how to write a valid XWF file and write your
> own JS code to generate it. But if airavata has a JavaScript API which can
> be used to generate XWF for you by getting JSON objects as input then it
> would be great help for client side developers.
>
> Thank,
> Shameera.
>
>
>
> >
> > Sorry, if above questions are in-appropriate, just wanted to understand
> > what exactly needed.
> >
> > Thanks
> > -Thejaka Amila
> >
> >
> >
> > On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
> > wrote:
> >
> >> I think I'm not suggesting to create a Workflow interpreter using Python
> >> etc. What I'm suggesting is to remove the Worflow aspect from core
> Airavata
> >> and move it to a more higher level component. The more I think about it,
> >> the model I'm suggesting is similar to what Hadoop, Storm etc has done
> for
> >> distributed system computations. This model is proven to be successful
> over
> >> the years.
> >>
> >> Keeping what Airavata does at its core can help you to build a more
> >> robust system. If we look at Airavata as middleware to execute
> applications
> >> on computing resources we can simplify what Airavata does and focus on
> >> improving the core functionality. All the successful systems have
> thrived
> >> on defining what it does at its core and keeping it simple and being
> >> excellent at what it does. In that regard keeping workflow aspect out of
> >> Airavata can help you to focus on the core problem. That is to execute
> an
> >> application on a remote computing resource in a fault tolerant and
> scalable
> >> manner.
> >>
> >> What I'm suggesting is to give the Orchestrator the capability to
> execute
> >> a Driver program that is specified by the user. (This program can be
> >> written in Python, Java or any other language). This driver program is
> >> similar to what you define in a Hadoop or Storm configuration. The
> driver
> >> program specifies the flow of the computation. It specifies what are the
> >> applications needs to be executed, how to manipulate input output. The
> >> driver program is the workflow for the user. Because the user specifies
> the
> >> program he can program it to handle workflow steering etc. Every time
> the
> >> user wants to execute this program he can tell Airavata to execute the
> >> Driver program.
> >>
> >> I'm also not 100% sure about all the details. But this can be a new way
> >> to look at how systems like Airavata should behave. Your thoughts and
> >> suggestions are more than welcome.
> >>
> >> Thanks,
> >> Supun..
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
> >> shameerainfo@gmail.com> wrote:
> >>
> >>> Hi Supun,
> >>>
> >>> I think we were in two context, because I as suggesting a way to
> >>> serialize
> >>> and deserialize the workflow description while you are suggesting to
> >>> implement
> >>> some kind of workflow interpreter using Python, where Python client can
> >>> send
> >>> thrift calls to Airavata server to run the application. I can see with
> >>> your
> >>> suggested
> >>> approach we can control the workflow execution process from client side
> >>> which
> >>> make it easy to implement workflow debugger.As you mentioned this is a
> >>> major change
> >>> to Airavata. So we should neatly think as this will change our existing
> >>> architecture.
> >>>
> >>> Still if someone need to use different language java, php, JS etc ...
> to
> >>> run the same
> >>> workflow which generated by Python, we need a language independent
> >>> workflow
> >>> description.
> >>> My initial question was what is the best language for this?. But as I
> >>> have
> >>> explained in
> >>> one of my previous reply, It is not matter what language we used Either
> >>> we
> >>> can use
> >>> XML or JSON to write this description, what matters is how easy to
> >>> generate
> >>> workflow with the provided API. Hence it would be great to have set of
> >>> neat
> >>> APIs in
> >>> different languages.
> >>>
> >>> Thanks,
> >>> Shameera.
> >>>
> >>>
> >>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <
> supun06@gmail.com
> >>> >
> >>> wrote:
> >>>
> >>> > Hi Shameera,
> >>> >
> >>> > Using python is a radical approach for workflow I think. But I
> believe
> >>> this
> >>> > is a very beautiful and new approach.Here is a possible scenario for
> >>> > implementation and running using a Python based library.
> >>> >
> >>> > The Python library facilitates the creation of Applications and
> >>> submitting
> >>> > them to Airavata for execution. The underlying functionality is
> exactly
> >>> > similar to what Java clients provides.The only difference is that,
> >>> Python
> >>> > library should have a more fluent API than Java for easy creation of
> >>> > workflows. We can generate the Python clients that talk to Airavata
> >>> server
> >>> > using Thrift.
> >>> >
> >>> > Here is an example off the top of my head to a Python script created
> by
> >>> > user for a Workflow. This is a very crude example and we need to come
> >>> up
> >>> > with a much better API if we are going to go along this path. First
> we
> >>> need
> >>> > to write a Python script that can execute a workflow using Airavata.
> >>> >
> >>> > import airavata
> >>> >
> >>> > host = Host("localhost", ....)
> >>> > app1 = Application(host, ....)
> >>> > app2 = Application(host, ....)
> >>> >
> >>> > # we will connect these applications as a workflow using some
> topology
> >>> > builder or other constructs
> >>> >
> >>> > wb = WorkFlowBuilder()
> >>> > wb.setApp("name1", app1)
> >>> > # you can do a simple output transformation here etc
> >>> >
> >>> > # connects the input of app1 to app2 etc
> >>> > wb.setApp("name2", app2).connectInput("name1")
> >>> >
> >>> > wb.submit()
> >>> >
> >>> > Now we can load this Python script from XBaya. When XBaya loads this
> >>> script
> >>> > the Python script can output an XML configuration of the topology,
> >>> XBaya
> >>> > can render. There are other ways like directly executing the Python
> >>> script
> >>> > from command line and connecting XBaya indirectly as well. Now you
> can
> >>> run
> >>> > the workflow from XBaya. Running the Workflow means just executing
> the
> >>> > Python script.
> >>> >
> >>> > XBaya gets the notifications through messaging and update the UI
> >>> > accordingly.
> >>> >
> >>> > The users need to write the Python script by hand. XBaya cannot
> create
> >>> the
> >>> > script. Because workflow language is an actual python program the
> >>> benefits
> >>> > are immense. For example user can do workflow steering in the
> workflow
> >>> > itself by subscribing to messages from Airavata.
> >>> >
> >>> > Thanks,
> >>> > Supun..
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> >>> > shameerainfo@gmail.com> wrote:
> >>> >
> >>> > > Hi Supun,
> >>> > >
> >>> > > I meant to say JS is a well-known client side scripting language i
> >>> have
> >>> > > messed scripting part. Even we use Python, ultimately we should
> >>> convert
> >>> > > this to java model in server side, somehow we need to serialized
> >>> python
> >>> > > representation to the language which java can parse and generate
> that
> >>> > > model. In this case we need to parse python script in java isn't
> it?
> >>> I am
> >>> > > not exactly clear how you suggesting to use python here. More
> >>> details on
> >>> > > how end system works if we use Python would be great help to
> clearly
> >>> > > understand your points.
> >>> > >
> >>> > > Thanks,
> >>> > > Shameera.
> >>> > >
> >>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> >>> > chris.mattmann@gmail.com>
> >>> > > wrote:
> >>> > >
> >>> > > > Have you guys considered using JCC [1] as a means
> >>> > > > to expose the workflow API currently in Java as a
> >>> > > > Python API?
> >>> > > >
> >>> > > > We are exploring its use in OODT, and we have already
> >>> > > > created a Tika [2] JCC-based python API.
> >>> > > >
> >>> > > > Cheers,
> >>> > > > Chris
> >>> > > >
> >>> > > > [1] http://lucene.apache.org/pylucene/jcc/
> >>> > > > [2] http://github.com/chrismattmann/tika-python/
> >>> > > >
> >>> > > > ------------------------
> >>> > > > Chris Mattmann
> >>> > > > chris.mattmann@gmail.com
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > -----Original Message-----
> >>> > > > From: Supun Kamburugamuva <su...@gmail.com>
> >>> > > > Reply-To: <de...@oodt.apache.org>
> >>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
> >>> > > > To: dev <de...@airavata.apache.org>
> >>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> >>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
> >>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> >>> > > > <de...@oodt.apache.org>
> >>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> >>> > Airavata.
> >>> > > >
> >>> > > > >Once we had an offline discussion about the Airavata Workflow
> >>> language
> >>> > > > >(with Milinda, Saliya and Shameera). In that discussion one
> thing
> >>> came
> >>> > > out
> >>> > > > >was why we need to invent a different language when a simple
> >>> library
> >>> > > like
> >>> > > > >Python will full fill of Airavata requirements.
> >>> > > > >
> >>> > > > >There are many benefits in using a Python library as the API for
> >>> > > > >controlling Airavata workflows.
> >>> > > > >
> >>> > > > >1. It is a library, gives the ultimate control over the
> execution
> >>> and
> >>> > it
> >>> > > > >can be simpler than any domain specific language that we can
> come
> >>> with
> >>> > > > >like
> >>> > > > >XML, JSON etc
> >>> > > > >2. Most people use python and can learn it easily than any
> Domain
> >>> > > specific
> >>> > > > >language
> >>> > > > >3. You can easily create a Python library for Airavata because
> >>> all the
> >>> > > > >APIs
> >>> > > > >are thrift based.
> >>> > > > >4. If you design the constructs correctly you can plug an XBaya.
> >>> > > > >
> >>> > > > >Any thoughts?
> >>> > > > >
> >>> > > > >Thanks,
> >>> > > > >Supun..
> >>> > > > >
> >>> > > > >
> >>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> >>> > supun06@gmail.com
> >>> > > >
> >>> > > > >wrote:
> >>> > > > >
> >>> > > > >> Hi Shameera,
> >>> > > > >>
> >>> > > > >> Why you prefer JavaScript over a language like Python?
> >>> > > > >>
> >>> > > > >> Thanks,
> >>> > > > >> Supun..
> >>> > > > >>
> >>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> >>> > > > >> shameerainfo@gmail.com> wrote:
> >>> > > > >>
> >>> > > > >>> Hi,
> >>> > > > >>>
> >>> > > > >>> First of all thanks everyone for giving valuable inputs.
> After
> >>> > doing
> >>> > > > >>>some
> >>> > > > >>> background search and talking to different people in the
> >>> University
> >>> > > > >>>who has
> >>> > > > >>> used different workflow languages, I myself convinced that
> >>> > > introducing
> >>> > > > >>>an
> >>> > > > >>> another workflow language is not what actually they need. By
> >>> > changing
> >>> > > > >>> exiting workflow language to another will not solve problems.
> >>> What
> >>> > > they
> >>> > > > >>> asking is a easy way to construct the workflows. Indirectly
> >>> what
> >>> > they
> >>> > > > >>> asking for a sort of API which they can use to generate the
> >>> > workflows
> >>> > > > >>>and
> >>> > > > >>> run it. Correct me if i am wrong here.
> >>> > > > >>>
> >>> > > > >>> As most of above replies depict,  if we can get a simple API,
> >>> as an
> >>> > > > >>> example, for a web based application, JavaScript API would
> be a
> >>> > good
> >>> > > > >>> solution, and probably JSON would be a good candidate for
> >>> language,
> >>> > > > >>>instead
> >>> > > > >>> of XML.
> >>> > > > >>>
> >>> > > > >>> Airavata community already have started to implement web base
> >>> GUI.
> >>> > > > >>>Hence
> >>> > > > >>> introducing a JSON base JavaScript API would be great help.
> >>> WDYT?
> >>> > > > >>>
> >>> > > > >>> Thanks,
> >>> > > > >>> Shameera.
> >>> > > > >>>
> >>> > > > >>>
> >>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >>> > > > >>> alekny7@gmail.com> wrote:
> >>> > > > >>>
> >>> > > > >>>> Hi,
> >>> > > > >>>>
> >>> > > > >>>> it is not dataflow instead focused on orchestrating REST
> >>> services
> >>> > > but
> >>> > > > >>>> you may find it useful datapoint - we created worfklow
> service
> >>> > that
> >>> > > > >>>>uses
> >>> > > > >>>> natively JavaScript and JSON to describe what happens during
> >>> > > workflow
> >>> > > > >>>> execution:
> >>> > > > >>>>
> >>> > > >
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>> > > > >>>>
> >>> > > > >>>> HTH,
> >>> > > > >>>>
> >>> > > > >>>> Alek
> >>> > > > >>>>
> >>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
> >>> smarru@apache.org>
> >>> > > > >>>>wrote:
> >>> > > > >>>>
> >>> > > > >>>>> Hi Chris,
> >>> > > > >>>>>
> >>> > > > >>>>> Great to hear OODT community will be interested in
> adopting a
> >>> > JSON
> >>> > > > >>>>> based workflow language and potentially a web based
> composer
> >>> as
> >>> > > well.
> >>> > > > >>>>> Airavata previously had BPEL support initially through a
> home
> >>> > grown
> >>> > > > >>>>> implementation [1] by Alek Slominski and later through
> >>> Apache ODE
> >>> > > > >>>>>[2]. Also
> >>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
> >>> read.
> >>> > > > >>>>>
> >>> > > > >>>>> I am of the same opinion that we should adopt something
> more
> >>> > modern
> >>> > > > >>>>>as
> >>> > > > >>>>> the challenges from scientific workflows seems to be
> >>> converging
> >>> > > with
> >>> > > > >>>>>the
> >>> > > > >>>>> data flow patterns in business workflows.
> >>> > > > >>>>>
> >>> > > > >>>>> It will be great if we can all compile a list of potential
> >>> > > candidates
> >>> > > > >>>>> and hack them through.
> >>> > > > >>>>>
> >>> > > > >>>>> Suresh
> >>> > > > >>>>> [1] -
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > >
> >>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>> > > > >>>>> [2] -
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > >
> >>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> >>> > > > >>>>>time_for_eScience_workflows
> >>> > > > >>>>> [3] -
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > >
> >>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> >>> > > > >>>>>6.pdf
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>> > > > >>>>>
> >>> > > > >>>>> > Hi Guys,
> >>> > > > >>>>> >
> >>> > > > >>>>> > I've been interested in this too - we don't per have a
> >>> specific
> >>> > > > >>>>> > OODT workflow language, but we specific workflows using
> >>> XML,
> >>> > and
> >>> > > > >>>>> > other configuration (we are also thinking of moving to
> >>> JSON for
> >>> > > > >>>>> > this).
> >>> > > > >>>>> >
> >>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
> >>> > complex
> >>> > > > >>>>> > to me.
> >>> > > > >>>>> >
> >>> > > > >>>>> > I wonder at the end of the day if we should adopt
> something
> >>> > more
> >>> > > > >>>>> > modern like PIG or some other data flow type of language
> >>> (PIG
> >>> > > > >>>>> > is really neat).
> >>> > > > >>>>> >
> >>> > > > >>>>> > Cheers,
> >>> > > > >>>>> > Chris
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> > > > >>>>> > Chris Mattmann, Ph.D.
> >>> > > > >>>>> > Chief Architect
> >>> > > > >>>>> > Instrument Software and Science Data Systems Section
> (398)
> >>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
> >>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
> >>> > > > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
> >>> > > > >>>>> >
> >>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
> >>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089
> >>> USA
> >>> > > > >>>>> >
> >>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> > -----Original Message-----
> >>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
> >>> > > > >>>>> > <ar...@airavata.apache.org>
> >>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>> > > > >>>>> > To: "architecture@airavata.apache.org" <
> >>> > > > >>>>> architecture@airavata.apache.org>,
> >>> > > > >>>>> > dev <de...@airavata.apache.org>
> >>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language
> for
> >>> > > > >>>>>Airavata.
> >>> > > > >>>>> >
> >>> > > > >>>>> >> Hi All,
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> As we all know Airavata has its own workflow language
> call
> >>> > XWF.
> >>> > > > >>>>>When
> >>> > > > >>>>> XWF
> >>> > > > >>>>> >> was introduced, main focus points are interoperability
> and
> >>> > > > >>>>> convertibility.
> >>> > > > >>>>> >> But with years of experience it is convinced that above
> >>> > > > >>>>>requirements
> >>> > > > >>>>> are
> >>> > > > >>>>> >> not really useful when we come to real world use cases.
> >>> And
> >>> > XWF
> >>> > > is
> >>> > > > >>>>> XML
> >>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
> >>> image
> >>> > > it
> >>> > > > >>>>> self.
> >>> > > > >>>>> >> But
> >>> > > > >>>>> >> with the recent changes WSDL part is being removed from
> >>> XWF.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
> >>> languages in
> >>> > > > >>>>> industry
> >>> > > > >>>>> >> and
> >>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
> >>> need
> >>> > to
> >>> > > > >>>>> come up
> >>> > > > >>>>> >> with idea how we should improve Airavata workflow
> >>> language,
> >>> > > either
> >>> > > > >>>>> we can
> >>> > > > >>>>> >> improve existing XWF language, totally change to a new
> >>> > language
> >>> > > > >>>>> available
> >>> > > > >>>>> >> in industry or write a new light weight language. Basic
> >>> > > > >>>>>requirements
> >>> > > > >>>>> that
> >>> > > > >>>>> >> we expect from new improvement are, high usability,
> >>> flexible,
> >>> > > > >>>>>light
> >>> > > > >>>>> weight
> >>> > > > >>>>> >> and real time monitoring support. As you can see above
> >>> > > > >>>>>requirements
> >>> > > > >>>>> are
> >>> > > > >>>>> >> not
> >>> > > > >>>>> >> direct comes with workflow languages but we need
> workflow
> >>> > > language
> >>> > > > >>>>> which
> >>> > > > >>>>> >> help to support above requirements.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> After reading few papers and googling, initially i have
> >>> come
> >>> > up
> >>> > > > >>>>>with
> >>> > > > >>>>> >> following three existing languages,
> >>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>> > > > >>>>> >> 2. WS-BPEL
> >>> > > > >>>>> >> 3. SIDL
> >>> > > > >>>>> >> <
> >>> > > http://computation.llnl.gov/casc/components/index.html#page=home
> >>> > > > >
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific
> >>> domain,
> >>> > > > >>>>> Radical-SAGA
> >>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
> >>> above
> >>> > > > >>>>>three
> >>> > > > >>>>> >> languages we can come up with simple workflow language
> >>> base on
> >>> > > > >>>>> json(or
> >>> > > > >>>>> >> yaml) which support all our requirements for some
> extends.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> It would be grate if I can get more input regarding the
> >>> > $Subject
> >>> > > > >>>>> form the
> >>> > > > >>>>> >> airavata community. You all are more than welcome to
> >>> provide
> >>> > any
> >>> > > > >>>>> type of
> >>> > > > >>>>> >> suggestions.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> Thanks,
> >>> > > > >>>>> >> Shameera.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> 
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> --
> >>> > > > >>>>> >> Best Regards,
> >>> > > > >>>>> >> Shameera Rathnayaka.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT
> gmail.com
> >>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > > >>>>
> >>> > > > >>>>
> >>> > > > >>>> --
> >>> > > > >>>> The best way to predict the future is to invent it - Alan
> Kay
> >>> > > > >>>>
> >>> > > > >>>
> >>> > > > >>>
> >>> > > > >>>
> >>> > > > >>> --
> >>> > > > >>> Best Regards,
> >>> > > > >>> Shameera Rathnayaka.
> >>> > > > >>>
> >>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
> >>> > > > >>>
> >>> > > > >>
> >>> > > > >>
> >>> > > > >>
> >>> > > > >> --
> >>> > > > >> Supun Kamburugamuva
> >>> > > > >> Member, Apache Software Foundation; http://www.apache.org
> >>> > > > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >>> > > > >> Blog: http://supunk.blogspot.com
> >>> > > > >>
> >>> > > > >>
> >>> > > > >
> >>> > > > >
> >>> > > > >--
> >>> > > > >Supun Kamburugamuva
> >>> > > > >Member, Apache Software Foundation; http://www.apache.org
> >>> > > > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >>> > > > >Blog: http://supunk.blogspot.com
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > >
> >>> > >
> >>> > > --
> >>> > > Best Regards,
> >>> > > Shameera Rathnayaka.
> >>> > >
> >>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> > > Blog : http://shameerarathnayaka.blogspot.com/
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Supun Kamburugamuva
> >>> > Member, Apache Software Foundation; http://www.apache.org
> >>> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >>> > Blog: http://supunk.blogspot.com
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards,
> >>> Shameera Rathnayaka.
> >>>
> >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> Blog : http://shameerarathnayaka.blogspot.com/
> >>>
> >>
> >>
> >>
> >> --
> >> Supun Kamburugamuva
> >> Member, Apache Software Foundation; http://www.apache.org
> >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >> Blog: http://supunk.blogspot.com
> >>
> >>
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Lahiru Gunathilake <gl...@gmail.com>.

Hi Shameera,

I think the whole point of introducing thrift is to support language
bindings for gateway developers and I do not think we should limit this
freedom by introducing a python client just to build a workflow. I propose
to represent a workflow as a thrift data-structure (I think by using
thrift's list objects we can create an adjacency list and of course it will
be tricky because of thrift unsupport for inheritance and other limitations
but I think it will be worth the effort assuming thrift IDL will be
improved a lot in future). If we can represent a workflow as a thrift
object model in the client side all the workflow related CRUD operations
can be generated using thrift, so we can support all the languages which
thrift support.

To submit a workflow we can introduce another thrift operation which accept
the big workflow thrift object and workflow interpreter can parse this and
execute the workflow. Internally we can use XWF and we can convert thrift
model to XWF and XWF to a thrift object.

Regards
Lahiru

On Mon, Oct 20, 2014 at 10:35 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:

> Hi Amila,
>
> Please see my comments inline.
>
> On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara <thejaka.amila@gmail.com
> > wrote:
>
>> Hello,
>>
>> I am sorry, I am bit late on this thread. But when reading through this
>> thread I simply got lost, what this thread is discussing. I have few
>> questions.
>>
>> 1. @Shameera : Is XWF actually a language to define workflow ? To my
>> understanding it was an intermediate representation to convert workflow
>> defined in UI to java object model. Was XWF ever used by any airavata user
>> to define a workflow graph ?
>>
>
> Yes, XWF is the language defined and used by Airavata  to explain the
> workflows but it is not well documented. Both server and client sides read
> this description language to generate runtime java representation.  so
> when you used XBaya to create a workflow with multiple applications, under
> the hood XBaya generate xwf which describe that workflow to server.
>
>
>
>>
>> From initial description what I understood is we are looking for a
>> improved "intermediate representation", not a language which describes
>> workflows.
>> 
>>
>>
>
>> 2. So what is the exact purpose of this proposed language ?
>>         - Is it to hide parallelism in workflows ?
>>         - Is it to replace the XBAYA functionalities ? (i hope not)
>>
>
>
> Actually initial idea was to identify good, well-defined and scientific
> domain friendly language. Whole purpose of this effort is reduce the entry
> barrier of the end user. But later it is understood that introducing a new
> language won't fix our issue.
>
>
>>
>> 3. What are we trying to achieve by this proposed language which we
>> cannot achieved through workflow UI tool ?
>>
>> 4. Who is going to use this language ?
>>
>
> As I explained, our direction has been changed.By introducing a new
> language we are get nothing but nice description file. No functional
> improvements etc ... The current language should use all airavata
> client(Currently we have only XBaya) side applications to explain the
> workflow to the server side.
>
>
>
>>
>> 5. Why would a user prefer (assuming intended audience of proposed
>> language is end users) a language over a Work Flow UI tool such as XBAYA ?
>> (In other words what are the things we can do with language which we cannot
>> do with UI)
>>
>
> Let's say you are going to write a new web base client for Airavata which
> generate workflows and launch it. What you need to do is do some magic
> with UI and finally come up with description language and send it to server
> side. Here you need to learn how to write a valid XWF file and write your
> own JS code to generate it. But if airavata has a JavaScript API which can
> be used to generate XWF for you by getting JSON objects as input then it
> would be great help for client side developers.
>
> Thank,
> Shameera.
>
>
>
>>
>> Sorry, if above questions are in-appropriate, just wanted to understand
>> what exactly needed.
>>
>> Thanks
>> -Thejaka Amila
>>
>>
>>
>> On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
>> wrote:
>>
>>> I think I'm not suggesting to create a Workflow interpreter using Python
>>> etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
>>> and move it to a more higher level component. The more I think about it,
>>> the model I'm suggesting is similar to what Hadoop, Storm etc has done for
>>> distributed system computations. This model is proven to be successful over
>>> the years.
>>>
>>> Keeping what Airavata does at its core can help you to build a more
>>> robust system. If we look at Airavata as middleware to execute applications
>>> on computing resources we can simplify what Airavata does and focus on
>>> improving the core functionality. All the successful systems have thrived
>>> on defining what it does at its core and keeping it simple and being
>>> excellent at what it does. In that regard keeping workflow aspect out of
>>> Airavata can help you to focus on the core problem. That is to execute an
>>> application on a remote computing resource in a fault tolerant and scalable
>>> manner.
>>>
>>> What I'm suggesting is to give the Orchestrator the capability to
>>> execute a Driver program that is specified by the user. (This program can
>>> be written in Python, Java or any other language). This driver program is
>>> similar to what you define in a Hadoop or Storm configuration. The driver
>>> program specifies the flow of the computation. It specifies what are the
>>> applications needs to be executed, how to manipulate input output. The
>>> driver program is the workflow for the user. Because the user specifies the
>>> program he can program it to handle workflow steering etc. Every time the
>>> user wants to execute this program he can tell Airavata to execute the
>>> Driver program.
>>>
>>> I'm also not 100% sure about all the details. But this can be a new way
>>> to look at how systems like Airavata should behave. Your thoughts and
>>> suggestions are more than welcome.
>>>
>>> Thanks,
>>> Supun..
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
>>> shameerainfo@gmail.com> wrote:
>>>
>>>> Hi Supun,
>>>>
>>>> I think we were in two context, because I as suggesting a way to
>>>> serialize
>>>> and deserialize the workflow description while you are suggesting to
>>>> implement
>>>> some kind of workflow interpreter using Python, where Python client can
>>>> send
>>>> thrift calls to Airavata server to run the application. I can see with
>>>> your
>>>> suggested
>>>> approach we can control the workflow execution process from client side
>>>> which
>>>> make it easy to implement workflow debugger.As you mentioned this is a
>>>> major change
>>>> to Airavata. So we should neatly think as this will change our existing
>>>> architecture.
>>>>
>>>> Still if someone need to use different language java, php, JS etc ... to
>>>> run the same
>>>> workflow which generated by Python, we need a language independent
>>>> workflow
>>>> description.
>>>> My initial question was what is the best language for this?. But as I
>>>> have
>>>> explained in
>>>> one of my previous reply, It is not matter what language we used Either
>>>> we
>>>> can use
>>>> XML or JSON to write this description, what matters is how easy to
>>>> generate
>>>> workflow with the provided API. Hence it would be great to have set of
>>>> neat
>>>> APIs in
>>>> different languages.
>>>>
>>>> Thanks,
>>>> Shameera.
>>>>
>>>>
>>>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <
>>>> supun06@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi Shameera,
>>>> >
>>>> > Using python is a radical approach for workflow I think. But I
>>>> believe this
>>>> > is a very beautiful and new approach.Here is a possible scenario for
>>>> > implementation and running using a Python based library.
>>>> >
>>>> > The Python library facilitates the creation of Applications and
>>>> submitting
>>>> > them to Airavata for execution. The underlying functionality is
>>>> exactly
>>>> > similar to what Java clients provides.The only difference is that,
>>>> Python
>>>> > library should have a more fluent API than Java for easy creation of
>>>> > workflows. We can generate the Python clients that talk to Airavata
>>>> server
>>>> > using Thrift.
>>>> >
>>>> > Here is an example off the top of my head to a Python script created
>>>> by
>>>> > user for a Workflow. This is a very crude example and we need to come
>>>> up
>>>> > with a much better API if we are going to go along this path. First
>>>> we need
>>>> > to write a Python script that can execute a workflow using Airavata.
>>>> >
>>>> > import airavata
>>>> >
>>>> > host = Host("localhost", ....)
>>>> > app1 = Application(host, ....)
>>>> > app2 = Application(host, ....)
>>>> >
>>>> > # we will connect these applications as a workflow using some topology
>>>> > builder or other constructs
>>>> >
>>>> > wb = WorkFlowBuilder()
>>>> > wb.setApp("name1", app1)
>>>> > # you can do a simple output transformation here etc
>>>> >
>>>> > # connects the input of app1 to app2 etc
>>>> > wb.setApp("name2", app2).connectInput("name1")
>>>> >
>>>> > wb.submit()
>>>> >
>>>> > Now we can load this Python script from XBaya. When XBaya loads this
>>>> script
>>>> > the Python script can output an XML configuration of the topology,
>>>> XBaya
>>>> > can render. There are other ways like directly executing the Python
>>>> script
>>>> > from command line and connecting XBaya indirectly as well. Now you
>>>> can run
>>>> > the workflow from XBaya. Running the Workflow means just executing the
>>>> > Python script.
>>>> >
>>>> > XBaya gets the notifications through messaging and update the UI
>>>> > accordingly.
>>>> >
>>>> > The users need to write the Python script by hand. XBaya cannot
>>>> create the
>>>> > script. Because workflow language is an actual python program the
>>>> benefits
>>>> > are immense. For example user can do workflow steering in the workflow
>>>> > itself by subscribing to messages from Airavata.
>>>> >
>>>> > Thanks,
>>>> > Supun..
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>>>> > shameerainfo@gmail.com> wrote:
>>>> >
>>>> > > Hi Supun,
>>>> > >
>>>> > > I meant to say JS is a well-known client side scripting language i
>>>> have
>>>> > > messed scripting part. Even we use Python, ultimately we should
>>>> convert
>>>> > > this to java model in server side, somehow we need to serialized
>>>> python
>>>> > > representation to the language which java can parse and generate
>>>> that
>>>> > > model. In this case we need to parse python script in java isn't
>>>> it? I am
>>>> > > not exactly clear how you suggesting to use python here. More
>>>> details on
>>>> > > how end system works if we use Python would be great help to clearly
>>>> > > understand your points.
>>>> > >
>>>> > > Thanks,
>>>> > > Shameera.
>>>> > >
>>>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>>>> > chris.mattmann@gmail.com>
>>>> > > wrote:
>>>> > >
>>>> > > > Have you guys considered using JCC [1] as a means
>>>> > > > to expose the workflow API currently in Java as a
>>>> > > > Python API?
>>>> > > >
>>>> > > > We are exploring its use in OODT, and we have already
>>>> > > > created a Tika [2] JCC-based python API.
>>>> > > >
>>>> > > > Cheers,
>>>> > > > Chris
>>>> > > >
>>>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>>>> > > > [2] http://github.com/chrismattmann/tika-python/
>>>> > > >
>>>> > > > ------------------------
>>>> > > > Chris Mattmann
>>>> > > > chris.mattmann@gmail.com
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > > -----Original Message-----
>>>> > > > From: Supun Kamburugamuva <su...@gmail.com>
>>>> > > > Reply-To: <de...@oodt.apache.org>
>>>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>>>> > > > To: dev <de...@airavata.apache.org>
>>>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
>>>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
>>>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
>>>> > > > <de...@oodt.apache.org>
>>>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>>>> > Airavata.
>>>> > > >
>>>> > > > >Once we had an offline discussion about the Airavata Workflow
>>>> language
>>>> > > > >(with Milinda, Saliya and Shameera). In that discussion one
>>>> thing came
>>>> > > out
>>>> > > > >was why we need to invent a different language when a simple
>>>> library
>>>> > > like
>>>> > > > >Python will full fill of Airavata requirements.
>>>> > > > >
>>>> > > > >There are many benefits in using a Python library as the API for
>>>> > > > >controlling Airavata workflows.
>>>> > > > >
>>>> > > > >1. It is a library, gives the ultimate control over the
>>>> execution and
>>>> > it
>>>> > > > >can be simpler than any domain specific language that we can
>>>> come with
>>>> > > > >like
>>>> > > > >XML, JSON etc
>>>> > > > >2. Most people use python and can learn it easily than any Domain
>>>> > > specific
>>>> > > > >language
>>>> > > > >3. You can easily create a Python library for Airavata because
>>>> all the
>>>> > > > >APIs
>>>> > > > >are thrift based.
>>>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>>>> > > > >
>>>> > > > >Any thoughts?
>>>> > > > >
>>>> > > > >Thanks,
>>>> > > > >Supun..
>>>> > > > >
>>>> > > > >
>>>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
>>>> > supun06@gmail.com
>>>> > > >
>>>> > > > >wrote:
>>>> > > > >
>>>> > > > >> Hi Shameera,
>>>> > > > >>
>>>> > > > >> Why you prefer JavaScript over a language like Python?
>>>> > > > >>
>>>> > > > >> Thanks,
>>>> > > > >> Supun..
>>>> > > > >>
>>>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>>>> > > > >> shameerainfo@gmail.com> wrote:
>>>> > > > >>
>>>> > > > >>> Hi,
>>>> > > > >>>
>>>> > > > >>> First of all thanks everyone for giving valuable inputs. After
>>>> > doing
>>>> > > > >>>some
>>>> > > > >>> background search and talking to different people in the
>>>> University
>>>> > > > >>>who has
>>>> > > > >>> used different workflow languages, I myself convinced that
>>>> > > introducing
>>>> > > > >>>an
>>>> > > > >>> another workflow language is not what actually they need. By
>>>> > changing
>>>> > > > >>> exiting workflow language to another will not solve problems.
>>>> What
>>>> > > they
>>>> > > > >>> asking is a easy way to construct the workflows. Indirectly
>>>> what
>>>> > they
>>>> > > > >>> asking for a sort of API which they can use to generate the
>>>> > workflows
>>>> > > > >>>and
>>>> > > > >>> run it. Correct me if i am wrong here.
>>>> > > > >>>
>>>> > > > >>> As most of above replies depict,  if we can get a simple API,
>>>> as an
>>>> > > > >>> example, for a web based application, JavaScript API would be
>>>> a
>>>> > good
>>>> > > > >>> solution, and probably JSON would be a good candidate for
>>>> language,
>>>> > > > >>>instead
>>>> > > > >>> of XML.
>>>> > > > >>>
>>>> > > > >>> Airavata community already have started to implement web base
>>>> GUI.
>>>> > > > >>>Hence
>>>> > > > >>> introducing a JSON base JavaScript API would be great help.
>>>> WDYT?
>>>> > > > >>>
>>>> > > > >>> Thanks,
>>>> > > > >>> Shameera.
>>>> > > > >>>
>>>> > > > >>>
>>>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>>> > > > >>> alekny7@gmail.com> wrote:
>>>> > > > >>>
>>>> > > > >>>> Hi,
>>>> > > > >>>>
>>>> > > > >>>> it is not dataflow instead focused on orchestrating REST
>>>> services
>>>> > > but
>>>> > > > >>>> you may find it useful datapoint - we created worfklow
>>>> service
>>>> > that
>>>> > > > >>>>uses
>>>> > > > >>>> natively JavaScript and JSON to describe what happens during
>>>> > > workflow
>>>> > > > >>>> execution:
>>>> > > > >>>>
>>>> > > >
>>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>> > > > >>>>
>>>> > > > >>>> HTH,
>>>> > > > >>>>
>>>> > > > >>>> Alek
>>>> > > > >>>>
>>>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
>>>> smarru@apache.org>
>>>> > > > >>>>wrote:
>>>> > > > >>>>
>>>> > > > >>>>> Hi Chris,
>>>> > > > >>>>>
>>>> > > > >>>>> Great to hear OODT community will be interested in adopting
>>>> a
>>>> > JSON
>>>> > > > >>>>> based workflow language and potentially a web based
>>>> composer as
>>>> > > well.
>>>> > > > >>>>> Airavata previously had BPEL support initially through a
>>>> home
>>>> > grown
>>>> > > > >>>>> implementation [1] by Alek Slominski and later through
>>>> Apache ODE
>>>> > > > >>>>>[2]. Also
>>>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
>>>> read.
>>>> > > > >>>>>
>>>> > > > >>>>> I am of the same opinion that we should adopt something more
>>>> > modern
>>>> > > > >>>>>as
>>>> > > > >>>>> the challenges from scientific workflows seems to be
>>>> converging
>>>> > > with
>>>> > > > >>>>>the
>>>> > > > >>>>> data flow patterns in business workflows.
>>>> > > > >>>>>
>>>> > > > >>>>> It will be great if we can all compile a list of potential
>>>> > > candidates
>>>> > > > >>>>> and hack them through.
>>>> > > > >>>>>
>>>> > > > >>>>> Suresh
>>>> > > > >>>>> [1] -
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > >
>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>> > > > >>>>> [2] -
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > >
>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>>>> > > > >>>>>time_for_eScience_workflows
>>>> > > > >>>>> [3] -
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > >
>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>>>> > > > >>>>>6.pdf
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>> > > > >>>>>
>>>> > > > >>>>> > Hi Guys,
>>>> > > > >>>>> >
>>>> > > > >>>>> > I've been interested in this too - we don't per have a
>>>> specific
>>>> > > > >>>>> > OODT workflow language, but we specific workflows using
>>>> XML,
>>>> > and
>>>> > > > >>>>> > other configuration (we are also thinking of moving to
>>>> JSON for
>>>> > > > >>>>> > this).
>>>> > > > >>>>> >
>>>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
>>>> > complex
>>>> > > > >>>>> > to me.
>>>> > > > >>>>> >
>>>> > > > >>>>> > I wonder at the end of the day if we should adopt
>>>> something
>>>> > more
>>>> > > > >>>>> > modern like PIG or some other data flow type of language
>>>> (PIG
>>>> > > > >>>>> > is really neat).
>>>> > > > >>>>> >
>>>> > > > >>>>> > Cheers,
>>>> > > > >>>>> > Chris
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > > > >>>>> > Chris Mattmann, Ph.D.
>>>> > > > >>>>> > Chief Architect
>>>> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
>>>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
>>>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
>>>> > > > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>>> > > > >>>>> >
>>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
>>>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089
>>>> USA
>>>> > > > >>>>> >
>>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> > -----Original Message-----
>>>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
>>>> > > > >>>>> > <ar...@airavata.apache.org>
>>>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>> > > > >>>>> > To: "architecture@airavata.apache.org" <
>>>> > > > >>>>> architecture@airavata.apache.org>,
>>>> > > > >>>>> > dev <de...@airavata.apache.org>
>>>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language
>>>> for
>>>> > > > >>>>>Airavata.
>>>> > > > >>>>> >
>>>> > > > >>>>> >> Hi All,
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> As we all know Airavata has its own workflow language
>>>> call
>>>> > XWF.
>>>> > > > >>>>>When
>>>> > > > >>>>> XWF
>>>> > > > >>>>> >> was introduced, main focus points are interoperability
>>>> and
>>>> > > > >>>>> convertibility.
>>>> > > > >>>>> >> But with years of experience it is convinced that above
>>>> > > > >>>>>requirements
>>>> > > > >>>>> are
>>>> > > > >>>>> >> not really useful when we come to real world use cases.
>>>> And
>>>> > XWF
>>>> > > is
>>>> > > > >>>>> XML
>>>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
>>>> image
>>>> > > it
>>>> > > > >>>>> self.
>>>> > > > >>>>> >> But
>>>> > > > >>>>> >> with the recent changes WSDL part is being removed from
>>>> XWF.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
>>>> languages in
>>>> > > > >>>>> industry
>>>> > > > >>>>> >> and
>>>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
>>>> need
>>>> > to
>>>> > > > >>>>> come up
>>>> > > > >>>>> >> with idea how we should improve Airavata workflow
>>>> language,
>>>> > > either
>>>> > > > >>>>> we can
>>>> > > > >>>>> >> improve existing XWF language, totally change to a new
>>>> > language
>>>> > > > >>>>> available
>>>> > > > >>>>> >> in industry or write a new light weight language. Basic
>>>> > > > >>>>>requirements
>>>> > > > >>>>> that
>>>> > > > >>>>> >> we expect from new improvement are, high usability,
>>>> flexible,
>>>> > > > >>>>>light
>>>> > > > >>>>> weight
>>>> > > > >>>>> >> and real time monitoring support. As you can see above
>>>> > > > >>>>>requirements
>>>> > > > >>>>> are
>>>> > > > >>>>> >> not
>>>> > > > >>>>> >> direct comes with workflow languages but we need workflow
>>>> > > language
>>>> > > > >>>>> which
>>>> > > > >>>>> >> help to support above requirements.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> After reading few papers and googling, initially i have
>>>> come
>>>> > up
>>>> > > > >>>>>with
>>>> > > > >>>>> >> following three existing languages,
>>>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>> > > > >>>>> >> 2. WS-BPEL
>>>> > > > >>>>> >> 3. SIDL
>>>> > > > >>>>> >> <
>>>> > > http://computation.llnl.gov/casc/components/index.html#page=home
>>>> > > > >
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific
>>>> domain,
>>>> > > > >>>>> Radical-SAGA
>>>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
>>>> above
>>>> > > > >>>>>three
>>>> > > > >>>>> >> languages we can come up with simple workflow language
>>>> base on
>>>> > > > >>>>> json(or
>>>> > > > >>>>> >> yaml) which support all our requirements for some
>>>> extends.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> It would be grate if I can get more input regarding the
>>>> > $Subject
>>>> > > > >>>>> form the
>>>> > > > >>>>> >> airavata community. You all are more than welcome to
>>>> provide
>>>> > any
>>>> > > > >>>>> type of
>>>> > > > >>>>> >> suggestions.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> Thanks,
>>>> > > > >>>>> >> Shameera.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> 
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> --
>>>> > > > >>>>> >> Best Regards,
>>>> > > > >>>>> >> Shameera Rathnayaka.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT
>>>> gmail.com
>>>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > > >>>>
>>>> > > > >>>>
>>>> > > > >>>> --
>>>> > > > >>>> The best way to predict the future is to invent it - Alan Kay
>>>> > > > >>>>
>>>> > > > >>>
>>>> > > > >>>
>>>> > > > >>>
>>>> > > > >>> --
>>>> > > > >>> Best Regards,
>>>> > > > >>> Shameera Rathnayaka.
>>>> > > > >>>
>>>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
>>>> > > > >>>
>>>> > > > >>
>>>> > > > >>
>>>> > > > >>
>>>> > > > >> --
>>>> > > > >> Supun Kamburugamuva
>>>> > > > >> Member, Apache Software Foundation; http://www.apache.org
>>>> > > > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>>> > > > >> Blog: http://supunk.blogspot.com
>>>> > > > >>
>>>> > > > >>
>>>> > > > >
>>>> > > > >
>>>> > > > >--
>>>> > > > >Supun Kamburugamuva
>>>> > > > >Member, Apache Software Foundation; http://www.apache.org
>>>> > > > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>>> > > > >Blog: http://supunk.blogspot.com
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > > Best Regards,
>>>> > > Shameera Rathnayaka.
>>>> > >
>>>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> > > Blog : http://shameerarathnayaka.blogspot.com/
>>>> > >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Supun Kamburugamuva
>>>> > Member, Apache Software Foundation; http://www.apache.org
>>>> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>>> > Blog: http://supunk.blogspot.com
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Shameera Rathnayaka.
>>>>
>>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>>
>>>
>>>
>>>
>>> --
>>> Supun Kamburugamuva
>>> Member, Apache Software Foundation; http://www.apache.org
>>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>> Blog: http://supunk.blogspot.com
>>>
>>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>



-- 
Research Assistant
Science Gateways Group
Indiana University

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Lahiru Gunathilake <gl...@gmail.com>.

Hi Shameera,

I think the whole point of introducing thrift is to support language
bindings for gateway developers and I do not think we should limit this
freedom by introducing a python client just to build a workflow. I propose
to represent a workflow as a thrift data-structure (I think by using
thrift's list objects we can create an adjacency list and of course it will
be tricky because of thrift unsupport for inheritance and other limitations
but I think it will be worth the effort assuming thrift IDL will be
improved a lot in future). If we can represent a workflow as a thrift
object model in the client side all the workflow related CRUD operations
can be generated using thrift, so we can support all the languages which
thrift support.

To submit a workflow we can introduce another thrift operation which accept
the big workflow thrift object and workflow interpreter can parse this and
execute the workflow. Internally we can use XWF and we can convert thrift
model to XWF and XWF to a thrift object.

Regards
Lahiru

On Mon, Oct 20, 2014 at 10:35 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:

> Hi Amila,
>
> Please see my comments inline.
>
> On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara <thejaka.amila@gmail.com
> > wrote:
>
>> Hello,
>>
>> I am sorry, I am bit late on this thread. But when reading through this
>> thread I simply got lost, what this thread is discussing. I have few
>> questions.
>>
>> 1. @Shameera : Is XWF actually a language to define workflow ? To my
>> understanding it was an intermediate representation to convert workflow
>> defined in UI to java object model. Was XWF ever used by any airavata user
>> to define a workflow graph ?
>>
>
> Yes, XWF is the language defined and used by Airavata  to explain the
> workflows but it is not well documented. Both server and client sides read
> this description language to generate runtime java representation.  so
> when you used XBaya to create a workflow with multiple applications, under
> the hood XBaya generate xwf which describe that workflow to server.
>
>
>
>>
>> From initial description what I understood is we are looking for a
>> improved "intermediate representation", not a language which describes
>> workflows.
>> 
>>
>>
>
>> 2. So what is the exact purpose of this proposed language ?
>>         - Is it to hide parallelism in workflows ?
>>         - Is it to replace the XBAYA functionalities ? (i hope not)
>>
>
>
> Actually initial idea was to identify good, well-defined and scientific
> domain friendly language. Whole purpose of this effort is reduce the entry
> barrier of the end user. But later it is understood that introducing a new
> language won't fix our issue.
>
>
>>
>> 3. What are we trying to achieve by this proposed language which we
>> cannot achieved through workflow UI tool ?
>>
>> 4. Who is going to use this language ?
>>
>
> As I explained, our direction has been changed.By introducing a new
> language we are get nothing but nice description file. No functional
> improvements etc ... The current language should use all airavata
> client(Currently we have only XBaya) side applications to explain the
> workflow to the server side.
>
>
>
>>
>> 5. Why would a user prefer (assuming intended audience of proposed
>> language is end users) a language over a Work Flow UI tool such as XBAYA ?
>> (In other words what are the things we can do with language which we cannot
>> do with UI)
>>
>
> Let's say you are going to write a new web base client for Airavata which
> generate workflows and launch it. What you need to do is do some magic
> with UI and finally come up with description language and send it to server
> side. Here you need to learn how to write a valid XWF file and write your
> own JS code to generate it. But if airavata has a JavaScript API which can
> be used to generate XWF for you by getting JSON objects as input then it
> would be great help for client side developers.
>
> Thank,
> Shameera.
>
>
>
>>
>> Sorry, if above questions are in-appropriate, just wanted to understand
>> what exactly needed.
>>
>> Thanks
>> -Thejaka Amila
>>
>>
>>
>> On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
>> wrote:
>>
>>> I think I'm not suggesting to create a Workflow interpreter using Python
>>> etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
>>> and move it to a more higher level component. The more I think about it,
>>> the model I'm suggesting is similar to what Hadoop, Storm etc has done for
>>> distributed system computations. This model is proven to be successful over
>>> the years.
>>>
>>> Keeping what Airavata does at its core can help you to build a more
>>> robust system. If we look at Airavata as middleware to execute applications
>>> on computing resources we can simplify what Airavata does and focus on
>>> improving the core functionality. All the successful systems have thrived
>>> on defining what it does at its core and keeping it simple and being
>>> excellent at what it does. In that regard keeping workflow aspect out of
>>> Airavata can help you to focus on the core problem. That is to execute an
>>> application on a remote computing resource in a fault tolerant and scalable
>>> manner.
>>>
>>> What I'm suggesting is to give the Orchestrator the capability to
>>> execute a Driver program that is specified by the user. (This program can
>>> be written in Python, Java or any other language). This driver program is
>>> similar to what you define in a Hadoop or Storm configuration. The driver
>>> program specifies the flow of the computation. It specifies what are the
>>> applications needs to be executed, how to manipulate input output. The
>>> driver program is the workflow for the user. Because the user specifies the
>>> program he can program it to handle workflow steering etc. Every time the
>>> user wants to execute this program he can tell Airavata to execute the
>>> Driver program.
>>>
>>> I'm also not 100% sure about all the details. But this can be a new way
>>> to look at how systems like Airavata should behave. Your thoughts and
>>> suggestions are more than welcome.
>>>
>>> Thanks,
>>> Supun..
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
>>> shameerainfo@gmail.com> wrote:
>>>
>>>> Hi Supun,
>>>>
>>>> I think we were in two context, because I as suggesting a way to
>>>> serialize
>>>> and deserialize the workflow description while you are suggesting to
>>>> implement
>>>> some kind of workflow interpreter using Python, where Python client can
>>>> send
>>>> thrift calls to Airavata server to run the application. I can see with
>>>> your
>>>> suggested
>>>> approach we can control the workflow execution process from client side
>>>> which
>>>> make it easy to implement workflow debugger.As you mentioned this is a
>>>> major change
>>>> to Airavata. So we should neatly think as this will change our existing
>>>> architecture.
>>>>
>>>> Still if someone need to use different language java, php, JS etc ... to
>>>> run the same
>>>> workflow which generated by Python, we need a language independent
>>>> workflow
>>>> description.
>>>> My initial question was what is the best language for this?. But as I
>>>> have
>>>> explained in
>>>> one of my previous reply, It is not matter what language we used Either
>>>> we
>>>> can use
>>>> XML or JSON to write this description, what matters is how easy to
>>>> generate
>>>> workflow with the provided API. Hence it would be great to have set of
>>>> neat
>>>> APIs in
>>>> different languages.
>>>>
>>>> Thanks,
>>>> Shameera.
>>>>
>>>>
>>>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <
>>>> supun06@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi Shameera,
>>>> >
>>>> > Using python is a radical approach for workflow I think. But I
>>>> believe this
>>>> > is a very beautiful and new approach.Here is a possible scenario for
>>>> > implementation and running using a Python based library.
>>>> >
>>>> > The Python library facilitates the creation of Applications and
>>>> submitting
>>>> > them to Airavata for execution. The underlying functionality is
>>>> exactly
>>>> > similar to what Java clients provides.The only difference is that,
>>>> Python
>>>> > library should have a more fluent API than Java for easy creation of
>>>> > workflows. We can generate the Python clients that talk to Airavata
>>>> server
>>>> > using Thrift.
>>>> >
>>>> > Here is an example off the top of my head to a Python script created
>>>> by
>>>> > user for a Workflow. This is a very crude example and we need to come
>>>> up
>>>> > with a much better API if we are going to go along this path. First
>>>> we need
>>>> > to write a Python script that can execute a workflow using Airavata.
>>>> >
>>>> > import airavata
>>>> >
>>>> > host = Host("localhost", ....)
>>>> > app1 = Application(host, ....)
>>>> > app2 = Application(host, ....)
>>>> >
>>>> > # we will connect these applications as a workflow using some topology
>>>> > builder or other constructs
>>>> >
>>>> > wb = WorkFlowBuilder()
>>>> > wb.setApp("name1", app1)
>>>> > # you can do a simple output transformation here etc
>>>> >
>>>> > # connects the input of app1 to app2 etc
>>>> > wb.setApp("name2", app2).connectInput("name1")
>>>> >
>>>> > wb.submit()
>>>> >
>>>> > Now we can load this Python script from XBaya. When XBaya loads this
>>>> script
>>>> > the Python script can output an XML configuration of the topology,
>>>> XBaya
>>>> > can render. There are other ways like directly executing the Python
>>>> script
>>>> > from command line and connecting XBaya indirectly as well. Now you
>>>> can run
>>>> > the workflow from XBaya. Running the Workflow means just executing the
>>>> > Python script.
>>>> >
>>>> > XBaya gets the notifications through messaging and update the UI
>>>> > accordingly.
>>>> >
>>>> > The users need to write the Python script by hand. XBaya cannot
>>>> create the
>>>> > script. Because workflow language is an actual python program the
>>>> benefits
>>>> > are immense. For example user can do workflow steering in the workflow
>>>> > itself by subscribing to messages from Airavata.
>>>> >
>>>> > Thanks,
>>>> > Supun..
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>>>> > shameerainfo@gmail.com> wrote:
>>>> >
>>>> > > Hi Supun,
>>>> > >
>>>> > > I meant to say JS is a well-known client side scripting language i
>>>> have
>>>> > > messed scripting part. Even we use Python, ultimately we should
>>>> convert
>>>> > > this to java model in server side, somehow we need to serialized
>>>> python
>>>> > > representation to the language which java can parse and generate
>>>> that
>>>> > > model. In this case we need to parse python script in java isn't
>>>> it? I am
>>>> > > not exactly clear how you suggesting to use python here. More
>>>> details on
>>>> > > how end system works if we use Python would be great help to clearly
>>>> > > understand your points.
>>>> > >
>>>> > > Thanks,
>>>> > > Shameera.
>>>> > >
>>>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>>>> > chris.mattmann@gmail.com>
>>>> > > wrote:
>>>> > >
>>>> > > > Have you guys considered using JCC [1] as a means
>>>> > > > to expose the workflow API currently in Java as a
>>>> > > > Python API?
>>>> > > >
>>>> > > > We are exploring its use in OODT, and we have already
>>>> > > > created a Tika [2] JCC-based python API.
>>>> > > >
>>>> > > > Cheers,
>>>> > > > Chris
>>>> > > >
>>>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>>>> > > > [2] http://github.com/chrismattmann/tika-python/
>>>> > > >
>>>> > > > ------------------------
>>>> > > > Chris Mattmann
>>>> > > > chris.mattmann@gmail.com
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > > -----Original Message-----
>>>> > > > From: Supun Kamburugamuva <su...@gmail.com>
>>>> > > > Reply-To: <de...@oodt.apache.org>
>>>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>>>> > > > To: dev <de...@airavata.apache.org>
>>>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
>>>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
>>>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
>>>> > > > <de...@oodt.apache.org>
>>>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>>>> > Airavata.
>>>> > > >
>>>> > > > >Once we had an offline discussion about the Airavata Workflow
>>>> language
>>>> > > > >(with Milinda, Saliya and Shameera). In that discussion one
>>>> thing came
>>>> > > out
>>>> > > > >was why we need to invent a different language when a simple
>>>> library
>>>> > > like
>>>> > > > >Python will full fill of Airavata requirements.
>>>> > > > >
>>>> > > > >There are many benefits in using a Python library as the API for
>>>> > > > >controlling Airavata workflows.
>>>> > > > >
>>>> > > > >1. It is a library, gives the ultimate control over the
>>>> execution and
>>>> > it
>>>> > > > >can be simpler than any domain specific language that we can
>>>> come with
>>>> > > > >like
>>>> > > > >XML, JSON etc
>>>> > > > >2. Most people use python and can learn it easily than any Domain
>>>> > > specific
>>>> > > > >language
>>>> > > > >3. You can easily create a Python library for Airavata because
>>>> all the
>>>> > > > >APIs
>>>> > > > >are thrift based.
>>>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>>>> > > > >
>>>> > > > >Any thoughts?
>>>> > > > >
>>>> > > > >Thanks,
>>>> > > > >Supun..
>>>> > > > >
>>>> > > > >
>>>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
>>>> > supun06@gmail.com
>>>> > > >
>>>> > > > >wrote:
>>>> > > > >
>>>> > > > >> Hi Shameera,
>>>> > > > >>
>>>> > > > >> Why you prefer JavaScript over a language like Python?
>>>> > > > >>
>>>> > > > >> Thanks,
>>>> > > > >> Supun..
>>>> > > > >>
>>>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>>>> > > > >> shameerainfo@gmail.com> wrote:
>>>> > > > >>
>>>> > > > >>> Hi,
>>>> > > > >>>
>>>> > > > >>> First of all thanks everyone for giving valuable inputs. After
>>>> > doing
>>>> > > > >>>some
>>>> > > > >>> background search and talking to different people in the
>>>> University
>>>> > > > >>>who has
>>>> > > > >>> used different workflow languages, I myself convinced that
>>>> > > introducing
>>>> > > > >>>an
>>>> > > > >>> another workflow language is not what actually they need. By
>>>> > changing
>>>> > > > >>> exiting workflow language to another will not solve problems.
>>>> What
>>>> > > they
>>>> > > > >>> asking is a easy way to construct the workflows. Indirectly
>>>> what
>>>> > they
>>>> > > > >>> asking for a sort of API which they can use to generate the
>>>> > workflows
>>>> > > > >>>and
>>>> > > > >>> run it. Correct me if i am wrong here.
>>>> > > > >>>
>>>> > > > >>> As most of above replies depict,  if we can get a simple API,
>>>> as an
>>>> > > > >>> example, for a web based application, JavaScript API would be
>>>> a
>>>> > good
>>>> > > > >>> solution, and probably JSON would be a good candidate for
>>>> language,
>>>> > > > >>>instead
>>>> > > > >>> of XML.
>>>> > > > >>>
>>>> > > > >>> Airavata community already have started to implement web base
>>>> GUI.
>>>> > > > >>>Hence
>>>> > > > >>> introducing a JSON base JavaScript API would be great help.
>>>> WDYT?
>>>> > > > >>>
>>>> > > > >>> Thanks,
>>>> > > > >>> Shameera.
>>>> > > > >>>
>>>> > > > >>>
>>>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>>> > > > >>> alekny7@gmail.com> wrote:
>>>> > > > >>>
>>>> > > > >>>> Hi,
>>>> > > > >>>>
>>>> > > > >>>> it is not dataflow instead focused on orchestrating REST
>>>> services
>>>> > > but
>>>> > > > >>>> you may find it useful datapoint - we created worfklow
>>>> service
>>>> > that
>>>> > > > >>>>uses
>>>> > > > >>>> natively JavaScript and JSON to describe what happens during
>>>> > > workflow
>>>> > > > >>>> execution:
>>>> > > > >>>>
>>>> > > >
>>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>> > > > >>>>
>>>> > > > >>>> HTH,
>>>> > > > >>>>
>>>> > > > >>>> Alek
>>>> > > > >>>>
>>>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
>>>> smarru@apache.org>
>>>> > > > >>>>wrote:
>>>> > > > >>>>
>>>> > > > >>>>> Hi Chris,
>>>> > > > >>>>>
>>>> > > > >>>>> Great to hear OODT community will be interested in adopting
>>>> a
>>>> > JSON
>>>> > > > >>>>> based workflow language and potentially a web based
>>>> composer as
>>>> > > well.
>>>> > > > >>>>> Airavata previously had BPEL support initially through a
>>>> home
>>>> > grown
>>>> > > > >>>>> implementation [1] by Alek Slominski and later through
>>>> Apache ODE
>>>> > > > >>>>>[2]. Also
>>>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
>>>> read.
>>>> > > > >>>>>
>>>> > > > >>>>> I am of the same opinion that we should adopt something more
>>>> > modern
>>>> > > > >>>>>as
>>>> > > > >>>>> the challenges from scientific workflows seems to be
>>>> converging
>>>> > > with
>>>> > > > >>>>>the
>>>> > > > >>>>> data flow patterns in business workflows.
>>>> > > > >>>>>
>>>> > > > >>>>> It will be great if we can all compile a list of potential
>>>> > > candidates
>>>> > > > >>>>> and hack them through.
>>>> > > > >>>>>
>>>> > > > >>>>> Suresh
>>>> > > > >>>>> [1] -
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > >
>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>> > > > >>>>> [2] -
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > >
>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>>>> > > > >>>>>time_for_eScience_workflows
>>>> > > > >>>>> [3] -
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > >
>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>>>> > > > >>>>>6.pdf
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>> > > > >>>>>
>>>> > > > >>>>> > Hi Guys,
>>>> > > > >>>>> >
>>>> > > > >>>>> > I've been interested in this too - we don't per have a
>>>> specific
>>>> > > > >>>>> > OODT workflow language, but we specific workflows using
>>>> XML,
>>>> > and
>>>> > > > >>>>> > other configuration (we are also thinking of moving to
>>>> JSON for
>>>> > > > >>>>> > this).
>>>> > > > >>>>> >
>>>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
>>>> > complex
>>>> > > > >>>>> > to me.
>>>> > > > >>>>> >
>>>> > > > >>>>> > I wonder at the end of the day if we should adopt
>>>> something
>>>> > more
>>>> > > > >>>>> > modern like PIG or some other data flow type of language
>>>> (PIG
>>>> > > > >>>>> > is really neat).
>>>> > > > >>>>> >
>>>> > > > >>>>> > Cheers,
>>>> > > > >>>>> > Chris
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > > > >>>>> > Chris Mattmann, Ph.D.
>>>> > > > >>>>> > Chief Architect
>>>> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
>>>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
>>>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
>>>> > > > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>>> > > > >>>>> >
>>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
>>>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089
>>>> USA
>>>> > > > >>>>> >
>>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> > -----Original Message-----
>>>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
>>>> > > > >>>>> > <ar...@airavata.apache.org>
>>>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>> > > > >>>>> > To: "architecture@airavata.apache.org" <
>>>> > > > >>>>> architecture@airavata.apache.org>,
>>>> > > > >>>>> > dev <de...@airavata.apache.org>
>>>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language
>>>> for
>>>> > > > >>>>>Airavata.
>>>> > > > >>>>> >
>>>> > > > >>>>> >> Hi All,
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> As we all know Airavata has its own workflow language
>>>> call
>>>> > XWF.
>>>> > > > >>>>>When
>>>> > > > >>>>> XWF
>>>> > > > >>>>> >> was introduced, main focus points are interoperability
>>>> and
>>>> > > > >>>>> convertibility.
>>>> > > > >>>>> >> But with years of experience it is convinced that above
>>>> > > > >>>>>requirements
>>>> > > > >>>>> are
>>>> > > > >>>>> >> not really useful when we come to real world use cases.
>>>> And
>>>> > XWF
>>>> > > is
>>>> > > > >>>>> XML
>>>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
>>>> image
>>>> > > it
>>>> > > > >>>>> self.
>>>> > > > >>>>> >> But
>>>> > > > >>>>> >> with the recent changes WSDL part is being removed from
>>>> XWF.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
>>>> languages in
>>>> > > > >>>>> industry
>>>> > > > >>>>> >> and
>>>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
>>>> need
>>>> > to
>>>> > > > >>>>> come up
>>>> > > > >>>>> >> with idea how we should improve Airavata workflow
>>>> language,
>>>> > > either
>>>> > > > >>>>> we can
>>>> > > > >>>>> >> improve existing XWF language, totally change to a new
>>>> > language
>>>> > > > >>>>> available
>>>> > > > >>>>> >> in industry or write a new light weight language. Basic
>>>> > > > >>>>>requirements
>>>> > > > >>>>> that
>>>> > > > >>>>> >> we expect from new improvement are, high usability,
>>>> flexible,
>>>> > > > >>>>>light
>>>> > > > >>>>> weight
>>>> > > > >>>>> >> and real time monitoring support. As you can see above
>>>> > > > >>>>>requirements
>>>> > > > >>>>> are
>>>> > > > >>>>> >> not
>>>> > > > >>>>> >> direct comes with workflow languages but we need workflow
>>>> > > language
>>>> > > > >>>>> which
>>>> > > > >>>>> >> help to support above requirements.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> After reading few papers and googling, initially i have
>>>> come
>>>> > up
>>>> > > > >>>>>with
>>>> > > > >>>>> >> following three existing languages,
>>>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>> > > > >>>>> >> 2. WS-BPEL
>>>> > > > >>>>> >> 3. SIDL
>>>> > > > >>>>> >> <
>>>> > > http://computation.llnl.gov/casc/components/index.html#page=home
>>>> > > > >
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific
>>>> domain,
>>>> > > > >>>>> Radical-SAGA
>>>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
>>>> above
>>>> > > > >>>>>three
>>>> > > > >>>>> >> languages we can come up with simple workflow language
>>>> base on
>>>> > > > >>>>> json(or
>>>> > > > >>>>> >> yaml) which support all our requirements for some
>>>> extends.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> It would be grate if I can get more input regarding the
>>>> > $Subject
>>>> > > > >>>>> form the
>>>> > > > >>>>> >> airavata community. You all are more than welcome to
>>>> provide
>>>> > any
>>>> > > > >>>>> type of
>>>> > > > >>>>> >> suggestions.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> Thanks,
>>>> > > > >>>>> >> Shameera.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> 
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> --
>>>> > > > >>>>> >> Best Regards,
>>>> > > > >>>>> >> Shameera Rathnayaka.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT
>>>> gmail.com
>>>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > > >>>>
>>>> > > > >>>>
>>>> > > > >>>> --
>>>> > > > >>>> The best way to predict the future is to invent it - Alan Kay
>>>> > > > >>>>
>>>> > > > >>>
>>>> > > > >>>
>>>> > > > >>>
>>>> > > > >>> --
>>>> > > > >>> Best Regards,
>>>> > > > >>> Shameera Rathnayaka.
>>>> > > > >>>
>>>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
>>>> > > > >>>
>>>> > > > >>
>>>> > > > >>
>>>> > > > >>
>>>> > > > >> --
>>>> > > > >> Supun Kamburugamuva
>>>> > > > >> Member, Apache Software Foundation; http://www.apache.org
>>>> > > > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>>> > > > >> Blog: http://supunk.blogspot.com
>>>> > > > >>
>>>> > > > >>
>>>> > > > >
>>>> > > > >
>>>> > > > >--
>>>> > > > >Supun Kamburugamuva
>>>> > > > >Member, Apache Software Foundation; http://www.apache.org
>>>> > > > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>>> > > > >Blog: http://supunk.blogspot.com
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > > Best Regards,
>>>> > > Shameera Rathnayaka.
>>>> > >
>>>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> > > Blog : http://shameerarathnayaka.blogspot.com/
>>>> > >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Supun Kamburugamuva
>>>> > Member, Apache Software Foundation; http://www.apache.org
>>>> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>>> > Blog: http://supunk.blogspot.com
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Shameera Rathnayaka.
>>>>
>>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>>
>>>
>>>
>>>
>>> --
>>> Supun Kamburugamuva
>>> Member, Apache Software Foundation; http://www.apache.org
>>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>> Blog: http://supunk.blogspot.com
>>>
>>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>



-- 
Research Assistant
Science Gateways Group
Indiana University

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Lahiru Gunathilake <gl...@gmail.com>.

Hi Shameera,

I think the whole point of introducing thrift is to support language
bindings for gateway developers and I do not think we should limit this
freedom by introducing a python client just to build a workflow. I propose
to represent a workflow as a thrift data-structure (I think by using
thrift's list objects we can create an adjacency list and of course it will
be tricky because of thrift unsupport for inheritance and other limitations
but I think it will be worth the effort assuming thrift IDL will be
improved a lot in future). If we can represent a workflow as a thrift
object model in the client side all the workflow related CRUD operations
can be generated using thrift, so we can support all the languages which
thrift support.

To submit a workflow we can introduce another thrift operation which accept
the big workflow thrift object and workflow interpreter can parse this and
execute the workflow. Internally we can use XWF and we can convert thrift
model to XWF and XWF to a thrift object.

Regards
Lahiru

On Mon, Oct 20, 2014 at 10:35 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:

> Hi Amila,
>
> Please see my comments inline.
>
> On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara <thejaka.amila@gmail.com
> > wrote:
>
>> Hello,
>>
>> I am sorry, I am bit late on this thread. But when reading through this
>> thread I simply got lost, what this thread is discussing. I have few
>> questions.
>>
>> 1. @Shameera : Is XWF actually a language to define workflow ? To my
>> understanding it was an intermediate representation to convert workflow
>> defined in UI to java object model. Was XWF ever used by any airavata user
>> to define a workflow graph ?
>>
>
> Yes, XWF is the language defined and used by Airavata  to explain the
> workflows but it is not well documented. Both server and client sides read
> this description language to generate runtime java representation.  so
> when you used XBaya to create a workflow with multiple applications, under
> the hood XBaya generate xwf which describe that workflow to server.
>
>
>
>>
>> From initial description what I understood is we are looking for a
>> improved "intermediate representation", not a language which describes
>> workflows.
>> 
>>
>>
>
>> 2. So what is the exact purpose of this proposed language ?
>>         - Is it to hide parallelism in workflows ?
>>         - Is it to replace the XBAYA functionalities ? (i hope not)
>>
>
>
> Actually initial idea was to identify good, well-defined and scientific
> domain friendly language. Whole purpose of this effort is reduce the entry
> barrier of the end user. But later it is understood that introducing a new
> language won't fix our issue.
>
>
>>
>> 3. What are we trying to achieve by this proposed language which we
>> cannot achieved through workflow UI tool ?
>>
>> 4. Who is going to use this language ?
>>
>
> As I explained, our direction has been changed.By introducing a new
> language we are get nothing but nice description file. No functional
> improvements etc ... The current language should use all airavata
> client(Currently we have only XBaya) side applications to explain the
> workflow to the server side.
>
>
>
>>
>> 5. Why would a user prefer (assuming intended audience of proposed
>> language is end users) a language over a Work Flow UI tool such as XBAYA ?
>> (In other words what are the things we can do with language which we cannot
>> do with UI)
>>
>
> Let's say you are going to write a new web base client for Airavata which
> generate workflows and launch it. What you need to do is do some magic
> with UI and finally come up with description language and send it to server
> side. Here you need to learn how to write a valid XWF file and write your
> own JS code to generate it. But if airavata has a JavaScript API which can
> be used to generate XWF for you by getting JSON objects as input then it
> would be great help for client side developers.
>
> Thank,
> Shameera.
>
>
>
>>
>> Sorry, if above questions are in-appropriate, just wanted to understand
>> what exactly needed.
>>
>> Thanks
>> -Thejaka Amila
>>
>>
>>
>> On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
>> wrote:
>>
>>> I think I'm not suggesting to create a Workflow interpreter using Python
>>> etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
>>> and move it to a more higher level component. The more I think about it,
>>> the model I'm suggesting is similar to what Hadoop, Storm etc has done for
>>> distributed system computations. This model is proven to be successful over
>>> the years.
>>>
>>> Keeping what Airavata does at its core can help you to build a more
>>> robust system. If we look at Airavata as middleware to execute applications
>>> on computing resources we can simplify what Airavata does and focus on
>>> improving the core functionality. All the successful systems have thrived
>>> on defining what it does at its core and keeping it simple and being
>>> excellent at what it does. In that regard keeping workflow aspect out of
>>> Airavata can help you to focus on the core problem. That is to execute an
>>> application on a remote computing resource in a fault tolerant and scalable
>>> manner.
>>>
>>> What I'm suggesting is to give the Orchestrator the capability to
>>> execute a Driver program that is specified by the user. (This program can
>>> be written in Python, Java or any other language). This driver program is
>>> similar to what you define in a Hadoop or Storm configuration. The driver
>>> program specifies the flow of the computation. It specifies what are the
>>> applications needs to be executed, how to manipulate input output. The
>>> driver program is the workflow for the user. Because the user specifies the
>>> program he can program it to handle workflow steering etc. Every time the
>>> user wants to execute this program he can tell Airavata to execute the
>>> Driver program.
>>>
>>> I'm also not 100% sure about all the details. But this can be a new way
>>> to look at how systems like Airavata should behave. Your thoughts and
>>> suggestions are more than welcome.
>>>
>>> Thanks,
>>> Supun..
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
>>> shameerainfo@gmail.com> wrote:
>>>
>>>> Hi Supun,
>>>>
>>>> I think we were in two context, because I as suggesting a way to
>>>> serialize
>>>> and deserialize the workflow description while you are suggesting to
>>>> implement
>>>> some kind of workflow interpreter using Python, where Python client can
>>>> send
>>>> thrift calls to Airavata server to run the application. I can see with
>>>> your
>>>> suggested
>>>> approach we can control the workflow execution process from client side
>>>> which
>>>> make it easy to implement workflow debugger.As you mentioned this is a
>>>> major change
>>>> to Airavata. So we should neatly think as this will change our existing
>>>> architecture.
>>>>
>>>> Still if someone need to use different language java, php, JS etc ... to
>>>> run the same
>>>> workflow which generated by Python, we need a language independent
>>>> workflow
>>>> description.
>>>> My initial question was what is the best language for this?. But as I
>>>> have
>>>> explained in
>>>> one of my previous reply, It is not matter what language we used Either
>>>> we
>>>> can use
>>>> XML or JSON to write this description, what matters is how easy to
>>>> generate
>>>> workflow with the provided API. Hence it would be great to have set of
>>>> neat
>>>> APIs in
>>>> different languages.
>>>>
>>>> Thanks,
>>>> Shameera.
>>>>
>>>>
>>>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <
>>>> supun06@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi Shameera,
>>>> >
>>>> > Using python is a radical approach for workflow I think. But I
>>>> believe this
>>>> > is a very beautiful and new approach.Here is a possible scenario for
>>>> > implementation and running using a Python based library.
>>>> >
>>>> > The Python library facilitates the creation of Applications and
>>>> submitting
>>>> > them to Airavata for execution. The underlying functionality is
>>>> exactly
>>>> > similar to what Java clients provides.The only difference is that,
>>>> Python
>>>> > library should have a more fluent API than Java for easy creation of
>>>> > workflows. We can generate the Python clients that talk to Airavata
>>>> server
>>>> > using Thrift.
>>>> >
>>>> > Here is an example off the top of my head to a Python script created
>>>> by
>>>> > user for a Workflow. This is a very crude example and we need to come
>>>> up
>>>> > with a much better API if we are going to go along this path. First
>>>> we need
>>>> > to write a Python script that can execute a workflow using Airavata.
>>>> >
>>>> > import airavata
>>>> >
>>>> > host = Host("localhost", ....)
>>>> > app1 = Application(host, ....)
>>>> > app2 = Application(host, ....)
>>>> >
>>>> > # we will connect these applications as a workflow using some topology
>>>> > builder or other constructs
>>>> >
>>>> > wb = WorkFlowBuilder()
>>>> > wb.setApp("name1", app1)
>>>> > # you can do a simple output transformation here etc
>>>> >
>>>> > # connects the input of app1 to app2 etc
>>>> > wb.setApp("name2", app2).connectInput("name1")
>>>> >
>>>> > wb.submit()
>>>> >
>>>> > Now we can load this Python script from XBaya. When XBaya loads this
>>>> script
>>>> > the Python script can output an XML configuration of the topology,
>>>> XBaya
>>>> > can render. There are other ways like directly executing the Python
>>>> script
>>>> > from command line and connecting XBaya indirectly as well. Now you
>>>> can run
>>>> > the workflow from XBaya. Running the Workflow means just executing the
>>>> > Python script.
>>>> >
>>>> > XBaya gets the notifications through messaging and update the UI
>>>> > accordingly.
>>>> >
>>>> > The users need to write the Python script by hand. XBaya cannot
>>>> create the
>>>> > script. Because workflow language is an actual python program the
>>>> benefits
>>>> > are immense. For example user can do workflow steering in the workflow
>>>> > itself by subscribing to messages from Airavata.
>>>> >
>>>> > Thanks,
>>>> > Supun..
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>>>> > shameerainfo@gmail.com> wrote:
>>>> >
>>>> > > Hi Supun,
>>>> > >
>>>> > > I meant to say JS is a well-known client side scripting language i
>>>> have
>>>> > > messed scripting part. Even we use Python, ultimately we should
>>>> convert
>>>> > > this to java model in server side, somehow we need to serialized
>>>> python
>>>> > > representation to the language which java can parse and generate
>>>> that
>>>> > > model. In this case we need to parse python script in java isn't
>>>> it? I am
>>>> > > not exactly clear how you suggesting to use python here. More
>>>> details on
>>>> > > how end system works if we use Python would be great help to clearly
>>>> > > understand your points.
>>>> > >
>>>> > > Thanks,
>>>> > > Shameera.
>>>> > >
>>>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>>>> > chris.mattmann@gmail.com>
>>>> > > wrote:
>>>> > >
>>>> > > > Have you guys considered using JCC [1] as a means
>>>> > > > to expose the workflow API currently in Java as a
>>>> > > > Python API?
>>>> > > >
>>>> > > > We are exploring its use in OODT, and we have already
>>>> > > > created a Tika [2] JCC-based python API.
>>>> > > >
>>>> > > > Cheers,
>>>> > > > Chris
>>>> > > >
>>>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>>>> > > > [2] http://github.com/chrismattmann/tika-python/
>>>> > > >
>>>> > > > ------------------------
>>>> > > > Chris Mattmann
>>>> > > > chris.mattmann@gmail.com
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > > -----Original Message-----
>>>> > > > From: Supun Kamburugamuva <su...@gmail.com>
>>>> > > > Reply-To: <de...@oodt.apache.org>
>>>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>>>> > > > To: dev <de...@airavata.apache.org>
>>>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
>>>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
>>>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
>>>> > > > <de...@oodt.apache.org>
>>>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>>>> > Airavata.
>>>> > > >
>>>> > > > >Once we had an offline discussion about the Airavata Workflow
>>>> language
>>>> > > > >(with Milinda, Saliya and Shameera). In that discussion one
>>>> thing came
>>>> > > out
>>>> > > > >was why we need to invent a different language when a simple
>>>> library
>>>> > > like
>>>> > > > >Python will full fill of Airavata requirements.
>>>> > > > >
>>>> > > > >There are many benefits in using a Python library as the API for
>>>> > > > >controlling Airavata workflows.
>>>> > > > >
>>>> > > > >1. It is a library, gives the ultimate control over the
>>>> execution and
>>>> > it
>>>> > > > >can be simpler than any domain specific language that we can
>>>> come with
>>>> > > > >like
>>>> > > > >XML, JSON etc
>>>> > > > >2. Most people use python and can learn it easily than any Domain
>>>> > > specific
>>>> > > > >language
>>>> > > > >3. You can easily create a Python library for Airavata because
>>>> all the
>>>> > > > >APIs
>>>> > > > >are thrift based.
>>>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>>>> > > > >
>>>> > > > >Any thoughts?
>>>> > > > >
>>>> > > > >Thanks,
>>>> > > > >Supun..
>>>> > > > >
>>>> > > > >
>>>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
>>>> > supun06@gmail.com
>>>> > > >
>>>> > > > >wrote:
>>>> > > > >
>>>> > > > >> Hi Shameera,
>>>> > > > >>
>>>> > > > >> Why you prefer JavaScript over a language like Python?
>>>> > > > >>
>>>> > > > >> Thanks,
>>>> > > > >> Supun..
>>>> > > > >>
>>>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>>>> > > > >> shameerainfo@gmail.com> wrote:
>>>> > > > >>
>>>> > > > >>> Hi,
>>>> > > > >>>
>>>> > > > >>> First of all thanks everyone for giving valuable inputs. After
>>>> > doing
>>>> > > > >>>some
>>>> > > > >>> background search and talking to different people in the
>>>> University
>>>> > > > >>>who has
>>>> > > > >>> used different workflow languages, I myself convinced that
>>>> > > introducing
>>>> > > > >>>an
>>>> > > > >>> another workflow language is not what actually they need. By
>>>> > changing
>>>> > > > >>> exiting workflow language to another will not solve problems.
>>>> What
>>>> > > they
>>>> > > > >>> asking is a easy way to construct the workflows. Indirectly
>>>> what
>>>> > they
>>>> > > > >>> asking for a sort of API which they can use to generate the
>>>> > workflows
>>>> > > > >>>and
>>>> > > > >>> run it. Correct me if i am wrong here.
>>>> > > > >>>
>>>> > > > >>> As most of above replies depict,  if we can get a simple API,
>>>> as an
>>>> > > > >>> example, for a web based application, JavaScript API would be
>>>> a
>>>> > good
>>>> > > > >>> solution, and probably JSON would be a good candidate for
>>>> language,
>>>> > > > >>>instead
>>>> > > > >>> of XML.
>>>> > > > >>>
>>>> > > > >>> Airavata community already have started to implement web base
>>>> GUI.
>>>> > > > >>>Hence
>>>> > > > >>> introducing a JSON base JavaScript API would be great help.
>>>> WDYT?
>>>> > > > >>>
>>>> > > > >>> Thanks,
>>>> > > > >>> Shameera.
>>>> > > > >>>
>>>> > > > >>>
>>>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>>> > > > >>> alekny7@gmail.com> wrote:
>>>> > > > >>>
>>>> > > > >>>> Hi,
>>>> > > > >>>>
>>>> > > > >>>> it is not dataflow instead focused on orchestrating REST
>>>> services
>>>> > > but
>>>> > > > >>>> you may find it useful datapoint - we created worfklow
>>>> service
>>>> > that
>>>> > > > >>>>uses
>>>> > > > >>>> natively JavaScript and JSON to describe what happens during
>>>> > > workflow
>>>> > > > >>>> execution:
>>>> > > > >>>>
>>>> > > >
>>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>> > > > >>>>
>>>> > > > >>>> HTH,
>>>> > > > >>>>
>>>> > > > >>>> Alek
>>>> > > > >>>>
>>>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
>>>> smarru@apache.org>
>>>> > > > >>>>wrote:
>>>> > > > >>>>
>>>> > > > >>>>> Hi Chris,
>>>> > > > >>>>>
>>>> > > > >>>>> Great to hear OODT community will be interested in adopting
>>>> a
>>>> > JSON
>>>> > > > >>>>> based workflow language and potentially a web based
>>>> composer as
>>>> > > well.
>>>> > > > >>>>> Airavata previously had BPEL support initially through a
>>>> home
>>>> > grown
>>>> > > > >>>>> implementation [1] by Alek Slominski and later through
>>>> Apache ODE
>>>> > > > >>>>>[2]. Also
>>>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
>>>> read.
>>>> > > > >>>>>
>>>> > > > >>>>> I am of the same opinion that we should adopt something more
>>>> > modern
>>>> > > > >>>>>as
>>>> > > > >>>>> the challenges from scientific workflows seems to be
>>>> converging
>>>> > > with
>>>> > > > >>>>>the
>>>> > > > >>>>> data flow patterns in business workflows.
>>>> > > > >>>>>
>>>> > > > >>>>> It will be great if we can all compile a list of potential
>>>> > > candidates
>>>> > > > >>>>> and hack them through.
>>>> > > > >>>>>
>>>> > > > >>>>> Suresh
>>>> > > > >>>>> [1] -
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > >
>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>> > > > >>>>> [2] -
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > >
>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>>>> > > > >>>>>time_for_eScience_workflows
>>>> > > > >>>>> [3] -
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > >
>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>>>> > > > >>>>>6.pdf
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>> > > > >>>>>
>>>> > > > >>>>> > Hi Guys,
>>>> > > > >>>>> >
>>>> > > > >>>>> > I've been interested in this too - we don't per have a
>>>> specific
>>>> > > > >>>>> > OODT workflow language, but we specific workflows using
>>>> XML,
>>>> > and
>>>> > > > >>>>> > other configuration (we are also thinking of moving to
>>>> JSON for
>>>> > > > >>>>> > this).
>>>> > > > >>>>> >
>>>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
>>>> > complex
>>>> > > > >>>>> > to me.
>>>> > > > >>>>> >
>>>> > > > >>>>> > I wonder at the end of the day if we should adopt
>>>> something
>>>> > more
>>>> > > > >>>>> > modern like PIG or some other data flow type of language
>>>> (PIG
>>>> > > > >>>>> > is really neat).
>>>> > > > >>>>> >
>>>> > > > >>>>> > Cheers,
>>>> > > > >>>>> > Chris
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > > > >>>>> > Chris Mattmann, Ph.D.
>>>> > > > >>>>> > Chief Architect
>>>> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
>>>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
>>>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
>>>> > > > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>>> > > > >>>>> >
>>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
>>>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089
>>>> USA
>>>> > > > >>>>> >
>>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> >
>>>> > > > >>>>> > -----Original Message-----
>>>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
>>>> > > > >>>>> > <ar...@airavata.apache.org>
>>>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>> > > > >>>>> > To: "architecture@airavata.apache.org" <
>>>> > > > >>>>> architecture@airavata.apache.org>,
>>>> > > > >>>>> > dev <de...@airavata.apache.org>
>>>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language
>>>> for
>>>> > > > >>>>>Airavata.
>>>> > > > >>>>> >
>>>> > > > >>>>> >> Hi All,
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> As we all know Airavata has its own workflow language
>>>> call
>>>> > XWF.
>>>> > > > >>>>>When
>>>> > > > >>>>> XWF
>>>> > > > >>>>> >> was introduced, main focus points are interoperability
>>>> and
>>>> > > > >>>>> convertibility.
>>>> > > > >>>>> >> But with years of experience it is convinced that above
>>>> > > > >>>>>requirements
>>>> > > > >>>>> are
>>>> > > > >>>>> >> not really useful when we come to real world use cases.
>>>> And
>>>> > XWF
>>>> > > is
>>>> > > > >>>>> XML
>>>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
>>>> image
>>>> > > it
>>>> > > > >>>>> self.
>>>> > > > >>>>> >> But
>>>> > > > >>>>> >> with the recent changes WSDL part is being removed from
>>>> XWF.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
>>>> languages in
>>>> > > > >>>>> industry
>>>> > > > >>>>> >> and
>>>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
>>>> need
>>>> > to
>>>> > > > >>>>> come up
>>>> > > > >>>>> >> with idea how we should improve Airavata workflow
>>>> language,
>>>> > > either
>>>> > > > >>>>> we can
>>>> > > > >>>>> >> improve existing XWF language, totally change to a new
>>>> > language
>>>> > > > >>>>> available
>>>> > > > >>>>> >> in industry or write a new light weight language. Basic
>>>> > > > >>>>>requirements
>>>> > > > >>>>> that
>>>> > > > >>>>> >> we expect from new improvement are, high usability,
>>>> flexible,
>>>> > > > >>>>>light
>>>> > > > >>>>> weight
>>>> > > > >>>>> >> and real time monitoring support. As you can see above
>>>> > > > >>>>>requirements
>>>> > > > >>>>> are
>>>> > > > >>>>> >> not
>>>> > > > >>>>> >> direct comes with workflow languages but we need workflow
>>>> > > language
>>>> > > > >>>>> which
>>>> > > > >>>>> >> help to support above requirements.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> After reading few papers and googling, initially i have
>>>> come
>>>> > up
>>>> > > > >>>>>with
>>>> > > > >>>>> >> following three existing languages,
>>>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>> > > > >>>>> >> 2. WS-BPEL
>>>> > > > >>>>> >> 3. SIDL
>>>> > > > >>>>> >> <
>>>> > > http://computation.llnl.gov/casc/components/index.html#page=home
>>>> > > > >
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific
>>>> domain,
>>>> > > > >>>>> Radical-SAGA
>>>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
>>>> above
>>>> > > > >>>>>three
>>>> > > > >>>>> >> languages we can come up with simple workflow language
>>>> base on
>>>> > > > >>>>> json(or
>>>> > > > >>>>> >> yaml) which support all our requirements for some
>>>> extends.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> It would be grate if I can get more input regarding the
>>>> > $Subject
>>>> > > > >>>>> form the
>>>> > > > >>>>> >> airavata community. You all are more than welcome to
>>>> provide
>>>> > any
>>>> > > > >>>>> type of
>>>> > > > >>>>> >> suggestions.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> Thanks,
>>>> > > > >>>>> >> Shameera.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> 
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> --
>>>> > > > >>>>> >> Best Regards,
>>>> > > > >>>>> >> Shameera Rathnayaka.
>>>> > > > >>>>> >>
>>>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT
>>>> gmail.com
>>>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>> > > > >>>>>
>>>> > > > >>>>>
>>>> > > > >>>>
>>>> > > > >>>>
>>>> > > > >>>> --
>>>> > > > >>>> The best way to predict the future is to invent it - Alan Kay
>>>> > > > >>>>
>>>> > > > >>>
>>>> > > > >>>
>>>> > > > >>>
>>>> > > > >>> --
>>>> > > > >>> Best Regards,
>>>> > > > >>> Shameera Rathnayaka.
>>>> > > > >>>
>>>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
>>>> > > > >>>
>>>> > > > >>
>>>> > > > >>
>>>> > > > >>
>>>> > > > >> --
>>>> > > > >> Supun Kamburugamuva
>>>> > > > >> Member, Apache Software Foundation; http://www.apache.org
>>>> > > > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>>> > > > >> Blog: http://supunk.blogspot.com
>>>> > > > >>
>>>> > > > >>
>>>> > > > >
>>>> > > > >
>>>> > > > >--
>>>> > > > >Supun Kamburugamuva
>>>> > > > >Member, Apache Software Foundation; http://www.apache.org
>>>> > > > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>>> > > > >Blog: http://supunk.blogspot.com
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > > Best Regards,
>>>> > > Shameera Rathnayaka.
>>>> > >
>>>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> > > Blog : http://shameerarathnayaka.blogspot.com/
>>>> > >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Supun Kamburugamuva
>>>> > Member, Apache Software Foundation; http://www.apache.org
>>>> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>>> > Blog: http://supunk.blogspot.com
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards,
>>>> Shameera Rathnayaka.
>>>>
>>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>>
>>>
>>>
>>>
>>> --
>>> Supun Kamburugamuva
>>> Member, Apache Software Foundation; http://www.apache.org
>>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>> Blog: http://supunk.blogspot.com
>>>
>>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>



-- 
Research Assistant
Science Gateways Group
Indiana University

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Amila Jayasekara <th...@gmail.com>.

Hi Shameera,

Thanks for the reply.

I wouldnt call XWF a "language". It is not a language in any sense but an
intermediate representation of a workflow graph.
To my understanding what you need is not a language but an efficient
portable intermediate representation. Seems you also have realized it. JSON
might be a good candidate provided most of the frontends are web based.

Thanks
-Thejaka

On Mon, Oct 20, 2014 at 10:35 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:

> Hi Amila,
>
> Please see my comments inline.
>
> On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara <thejaka.amila@gmail.com
> >
> wrote:
>
> > Hello,
> >
> > I am sorry, I am bit late on this thread. But when reading through this
> > thread I simply got lost, what this thread is discussing. I have few
> > questions.
> >
> > 1. @Shameera : Is XWF actually a language to define workflow ? To my
> > understanding it was an intermediate representation to convert workflow
> > defined in UI to java object model. Was XWF ever used by any airavata
> user
> > to define a workflow graph ?
> >
>
> Yes, XWF is the language defined and used by Airavata  to explain the
> workflows but it is not well documented. Both server and client sides read
> this description language to generate runtime java representation.  so
> when you used XBaya to create a workflow with multiple applications, under
> the hood XBaya generate xwf which describe that workflow to server.
>
>
>
> >
> > From initial description what I understood is we are looking for a
> > improved "intermediate representation", not a language which describes
> > workflows.
> > 
> >
> >
>
> > 2. So what is the exact purpose of this proposed language ?
> >         - Is it to hide parallelism in workflows ?
> >         - Is it to replace the XBAYA functionalities ? (i hope not)
> >
>
>
> Actually initial idea was to identify good, well-defined and scientific
> domain friendly language. Whole purpose of this effort is reduce the entry
> barrier of the end user. But later it is understood that introducing a new
> language won't fix our issue.
>
>
> >
> > 3. What are we trying to achieve by this proposed language which we
> cannot
> > achieved through workflow UI tool ?
> >
> > 4. Who is going to use this language ?
> >
>
> As I explained, our direction has been changed.By introducing a new
> language we are get nothing but nice description file. No functional
> improvements etc ... The current language should use all airavata
> client(Currently we have only XBaya) side applications to explain the
> workflow to the server side.
>
>
>
> >
> > 5. Why would a user prefer (assuming intended audience of proposed
> > language is end users) a language over a Work Flow UI tool such as XBAYA
> ?
> > (In other words what are the things we can do with language which we
> cannot
> > do with UI)
> >
>
> Let's say you are going to write a new web base client for Airavata which
> generate workflows and launch it. What you need to do is do some magic
> with UI and finally come up with description language and send it to server
> side. Here you need to learn how to write a valid XWF file and write your
> own JS code to generate it. But if airavata has a JavaScript API which can
> be used to generate XWF for you by getting JSON objects as input then it
> would be great help for client side developers.
>
> Thank,
> Shameera.
>
>
>
> >
> > Sorry, if above questions are in-appropriate, just wanted to understand
> > what exactly needed.
> >
> > Thanks
> > -Thejaka Amila
> >
> >
> >
> > On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
> > wrote:
> >
> >> I think I'm not suggesting to create a Workflow interpreter using Python
> >> etc. What I'm suggesting is to remove the Worflow aspect from core
> Airavata
> >> and move it to a more higher level component. The more I think about it,
> >> the model I'm suggesting is similar to what Hadoop, Storm etc has done
> for
> >> distributed system computations. This model is proven to be successful
> over
> >> the years.
> >>
> >> Keeping what Airavata does at its core can help you to build a more
> >> robust system. If we look at Airavata as middleware to execute
> applications
> >> on computing resources we can simplify what Airavata does and focus on
> >> improving the core functionality. All the successful systems have
> thrived
> >> on defining what it does at its core and keeping it simple and being
> >> excellent at what it does. In that regard keeping workflow aspect out of
> >> Airavata can help you to focus on the core problem. That is to execute
> an
> >> application on a remote computing resource in a fault tolerant and
> scalable
> >> manner.
> >>
> >> What I'm suggesting is to give the Orchestrator the capability to
> execute
> >> a Driver program that is specified by the user. (This program can be
> >> written in Python, Java or any other language). This driver program is
> >> similar to what you define in a Hadoop or Storm configuration. The
> driver
> >> program specifies the flow of the computation. It specifies what are the
> >> applications needs to be executed, how to manipulate input output. The
> >> driver program is the workflow for the user. Because the user specifies
> the
> >> program he can program it to handle workflow steering etc. Every time
> the
> >> user wants to execute this program he can tell Airavata to execute the
> >> Driver program.
> >>
> >> I'm also not 100% sure about all the details. But this can be a new way
> >> to look at how systems like Airavata should behave. Your thoughts and
> >> suggestions are more than welcome.
> >>
> >> Thanks,
> >> Supun..
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
> >> shameerainfo@gmail.com> wrote:
> >>
> >>> Hi Supun,
> >>>
> >>> I think we were in two context, because I as suggesting a way to
> >>> serialize
> >>> and deserialize the workflow description while you are suggesting to
> >>> implement
> >>> some kind of workflow interpreter using Python, where Python client can
> >>> send
> >>> thrift calls to Airavata server to run the application. I can see with
> >>> your
> >>> suggested
> >>> approach we can control the workflow execution process from client side
> >>> which
> >>> make it easy to implement workflow debugger.As you mentioned this is a
> >>> major change
> >>> to Airavata. So we should neatly think as this will change our existing
> >>> architecture.
> >>>
> >>> Still if someone need to use different language java, php, JS etc ...
> to
> >>> run the same
> >>> workflow which generated by Python, we need a language independent
> >>> workflow
> >>> description.
> >>> My initial question was what is the best language for this?. But as I
> >>> have
> >>> explained in
> >>> one of my previous reply, It is not matter what language we used Either
> >>> we
> >>> can use
> >>> XML or JSON to write this description, what matters is how easy to
> >>> generate
> >>> workflow with the provided API. Hence it would be great to have set of
> >>> neat
> >>> APIs in
> >>> different languages.
> >>>
> >>> Thanks,
> >>> Shameera.
> >>>
> >>>
> >>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <
> supun06@gmail.com
> >>> >
> >>> wrote:
> >>>
> >>> > Hi Shameera,
> >>> >
> >>> > Using python is a radical approach for workflow I think. But I
> believe
> >>> this
> >>> > is a very beautiful and new approach.Here is a possible scenario for
> >>> > implementation and running using a Python based library.
> >>> >
> >>> > The Python library facilitates the creation of Applications and
> >>> submitting
> >>> > them to Airavata for execution. The underlying functionality is
> exactly
> >>> > similar to what Java clients provides.The only difference is that,
> >>> Python
> >>> > library should have a more fluent API than Java for easy creation of
> >>> > workflows. We can generate the Python clients that talk to Airavata
> >>> server
> >>> > using Thrift.
> >>> >
> >>> > Here is an example off the top of my head to a Python script created
> by
> >>> > user for a Workflow. This is a very crude example and we need to come
> >>> up
> >>> > with a much better API if we are going to go along this path. First
> we
> >>> need
> >>> > to write a Python script that can execute a workflow using Airavata.
> >>> >
> >>> > import airavata
> >>> >
> >>> > host = Host("localhost", ....)
> >>> > app1 = Application(host, ....)
> >>> > app2 = Application(host, ....)
> >>> >
> >>> > # we will connect these applications as a workflow using some
> topology
> >>> > builder or other constructs
> >>> >
> >>> > wb = WorkFlowBuilder()
> >>> > wb.setApp("name1", app1)
> >>> > # you can do a simple output transformation here etc
> >>> >
> >>> > # connects the input of app1 to app2 etc
> >>> > wb.setApp("name2", app2).connectInput("name1")
> >>> >
> >>> > wb.submit()
> >>> >
> >>> > Now we can load this Python script from XBaya. When XBaya loads this
> >>> script
> >>> > the Python script can output an XML configuration of the topology,
> >>> XBaya
> >>> > can render. There are other ways like directly executing the Python
> >>> script
> >>> > from command line and connecting XBaya indirectly as well. Now you
> can
> >>> run
> >>> > the workflow from XBaya. Running the Workflow means just executing
> the
> >>> > Python script.
> >>> >
> >>> > XBaya gets the notifications through messaging and update the UI
> >>> > accordingly.
> >>> >
> >>> > The users need to write the Python script by hand. XBaya cannot
> create
> >>> the
> >>> > script. Because workflow language is an actual python program the
> >>> benefits
> >>> > are immense. For example user can do workflow steering in the
> workflow
> >>> > itself by subscribing to messages from Airavata.
> >>> >
> >>> > Thanks,
> >>> > Supun..
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> >>> > shameerainfo@gmail.com> wrote:
> >>> >
> >>> > > Hi Supun,
> >>> > >
> >>> > > I meant to say JS is a well-known client side scripting language i
> >>> have
> >>> > > messed scripting part. Even we use Python, ultimately we should
> >>> convert
> >>> > > this to java model in server side, somehow we need to serialized
> >>> python
> >>> > > representation to the language which java can parse and generate
> that
> >>> > > model. In this case we need to parse python script in java isn't
> it?
> >>> I am
> >>> > > not exactly clear how you suggesting to use python here. More
> >>> details on
> >>> > > how end system works if we use Python would be great help to
> clearly
> >>> > > understand your points.
> >>> > >
> >>> > > Thanks,
> >>> > > Shameera.
> >>> > >
> >>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> >>> > chris.mattmann@gmail.com>
> >>> > > wrote:
> >>> > >
> >>> > > > Have you guys considered using JCC [1] as a means
> >>> > > > to expose the workflow API currently in Java as a
> >>> > > > Python API?
> >>> > > >
> >>> > > > We are exploring its use in OODT, and we have already
> >>> > > > created a Tika [2] JCC-based python API.
> >>> > > >
> >>> > > > Cheers,
> >>> > > > Chris
> >>> > > >
> >>> > > > [1] http://lucene.apache.org/pylucene/jcc/
> >>> > > > [2] http://github.com/chrismattmann/tika-python/
> >>> > > >
> >>> > > > ------------------------
> >>> > > > Chris Mattmann
> >>> > > > chris.mattmann@gmail.com
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > -----Original Message-----
> >>> > > > From: Supun Kamburugamuva <su...@gmail.com>
> >>> > > > Reply-To: <de...@oodt.apache.org>
> >>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
> >>> > > > To: dev <de...@airavata.apache.org>
> >>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> >>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
> >>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> >>> > > > <de...@oodt.apache.org>
> >>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> >>> > Airavata.
> >>> > > >
> >>> > > > >Once we had an offline discussion about the Airavata Workflow
> >>> language
> >>> > > > >(with Milinda, Saliya and Shameera). In that discussion one
> thing
> >>> came
> >>> > > out
> >>> > > > >was why we need to invent a different language when a simple
> >>> library
> >>> > > like
> >>> > > > >Python will full fill of Airavata requirements.
> >>> > > > >
> >>> > > > >There are many benefits in using a Python library as the API for
> >>> > > > >controlling Airavata workflows.
> >>> > > > >
> >>> > > > >1. It is a library, gives the ultimate control over the
> execution
> >>> and
> >>> > it
> >>> > > > >can be simpler than any domain specific language that we can
> come
> >>> with
> >>> > > > >like
> >>> > > > >XML, JSON etc
> >>> > > > >2. Most people use python and can learn it easily than any
> Domain
> >>> > > specific
> >>> > > > >language
> >>> > > > >3. You can easily create a Python library for Airavata because
> >>> all the
> >>> > > > >APIs
> >>> > > > >are thrift based.
> >>> > > > >4. If you design the constructs correctly you can plug an XBaya.
> >>> > > > >
> >>> > > > >Any thoughts?
> >>> > > > >
> >>> > > > >Thanks,
> >>> > > > >Supun..
> >>> > > > >
> >>> > > > >
> >>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> >>> > supun06@gmail.com
> >>> > > >
> >>> > > > >wrote:
> >>> > > > >
> >>> > > > >> Hi Shameera,
> >>> > > > >>
> >>> > > > >> Why you prefer JavaScript over a language like Python?
> >>> > > > >>
> >>> > > > >> Thanks,
> >>> > > > >> Supun..
> >>> > > > >>
> >>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> >>> > > > >> shameerainfo@gmail.com> wrote:
> >>> > > > >>
> >>> > > > >>> Hi,
> >>> > > > >>>
> >>> > > > >>> First of all thanks everyone for giving valuable inputs.
> After
> >>> > doing
> >>> > > > >>>some
> >>> > > > >>> background search and talking to different people in the
> >>> University
> >>> > > > >>>who has
> >>> > > > >>> used different workflow languages, I myself convinced that
> >>> > > introducing
> >>> > > > >>>an
> >>> > > > >>> another workflow language is not what actually they need. By
> >>> > changing
> >>> > > > >>> exiting workflow language to another will not solve problems.
> >>> What
> >>> > > they
> >>> > > > >>> asking is a easy way to construct the workflows. Indirectly
> >>> what
> >>> > they
> >>> > > > >>> asking for a sort of API which they can use to generate the
> >>> > workflows
> >>> > > > >>>and
> >>> > > > >>> run it. Correct me if i am wrong here.
> >>> > > > >>>
> >>> > > > >>> As most of above replies depict,  if we can get a simple API,
> >>> as an
> >>> > > > >>> example, for a web based application, JavaScript API would
> be a
> >>> > good
> >>> > > > >>> solution, and probably JSON would be a good candidate for
> >>> language,
> >>> > > > >>>instead
> >>> > > > >>> of XML.
> >>> > > > >>>
> >>> > > > >>> Airavata community already have started to implement web base
> >>> GUI.
> >>> > > > >>>Hence
> >>> > > > >>> introducing a JSON base JavaScript API would be great help.
> >>> WDYT?
> >>> > > > >>>
> >>> > > > >>> Thanks,
> >>> > > > >>> Shameera.
> >>> > > > >>>
> >>> > > > >>>
> >>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >>> > > > >>> alekny7@gmail.com> wrote:
> >>> > > > >>>
> >>> > > > >>>> Hi,
> >>> > > > >>>>
> >>> > > > >>>> it is not dataflow instead focused on orchestrating REST
> >>> services
> >>> > > but
> >>> > > > >>>> you may find it useful datapoint - we created worfklow
> service
> >>> > that
> >>> > > > >>>>uses
> >>> > > > >>>> natively JavaScript and JSON to describe what happens during
> >>> > > workflow
> >>> > > > >>>> execution:
> >>> > > > >>>>
> >>> > > >
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>> > > > >>>>
> >>> > > > >>>> HTH,
> >>> > > > >>>>
> >>> > > > >>>> Alek
> >>> > > > >>>>
> >>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
> >>> smarru@apache.org>
> >>> > > > >>>>wrote:
> >>> > > > >>>>
> >>> > > > >>>>> Hi Chris,
> >>> > > > >>>>>
> >>> > > > >>>>> Great to hear OODT community will be interested in
> adopting a
> >>> > JSON
> >>> > > > >>>>> based workflow language and potentially a web based
> composer
> >>> as
> >>> > > well.
> >>> > > > >>>>> Airavata previously had BPEL support initially through a
> home
> >>> > grown
> >>> > > > >>>>> implementation [1] by Alek Slominski and later through
> >>> Apache ODE
> >>> > > > >>>>>[2]. Also
> >>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
> >>> read.
> >>> > > > >>>>>
> >>> > > > >>>>> I am of the same opinion that we should adopt something
> more
> >>> > modern
> >>> > > > >>>>>as
> >>> > > > >>>>> the challenges from scientific workflows seems to be
> >>> converging
> >>> > > with
> >>> > > > >>>>>the
> >>> > > > >>>>> data flow patterns in business workflows.
> >>> > > > >>>>>
> >>> > > > >>>>> It will be great if we can all compile a list of potential
> >>> > > candidates
> >>> > > > >>>>> and hack them through.
> >>> > > > >>>>>
> >>> > > > >>>>> Suresh
> >>> > > > >>>>> [1] -
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > >
> >>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>> > > > >>>>> [2] -
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > >
> >>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> >>> > > > >>>>>time_for_eScience_workflows
> >>> > > > >>>>> [3] -
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > >
> >>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> >>> > > > >>>>>6.pdf
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>> > > > >>>>>
> >>> > > > >>>>> > Hi Guys,
> >>> > > > >>>>> >
> >>> > > > >>>>> > I've been interested in this too - we don't per have a
> >>> specific
> >>> > > > >>>>> > OODT workflow language, but we specific workflows using
> >>> XML,
> >>> > and
> >>> > > > >>>>> > other configuration (we are also thinking of moving to
> >>> JSON for
> >>> > > > >>>>> > this).
> >>> > > > >>>>> >
> >>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
> >>> > complex
> >>> > > > >>>>> > to me.
> >>> > > > >>>>> >
> >>> > > > >>>>> > I wonder at the end of the day if we should adopt
> something
> >>> > more
> >>> > > > >>>>> > modern like PIG or some other data flow type of language
> >>> (PIG
> >>> > > > >>>>> > is really neat).
> >>> > > > >>>>> >
> >>> > > > >>>>> > Cheers,
> >>> > > > >>>>> > Chris
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> > > > >>>>> > Chris Mattmann, Ph.D.
> >>> > > > >>>>> > Chief Architect
> >>> > > > >>>>> > Instrument Software and Science Data Systems Section
> (398)
> >>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
> >>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
> >>> > > > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
> >>> > > > >>>>> >
> >>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
> >>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089
> >>> USA
> >>> > > > >>>>> >
> >>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> >
> >>> > > > >>>>> > -----Original Message-----
> >>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
> >>> > > > >>>>> > <ar...@airavata.apache.org>
> >>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>> > > > >>>>> > To: "architecture@airavata.apache.org" <
> >>> > > > >>>>> architecture@airavata.apache.org>,
> >>> > > > >>>>> > dev <de...@airavata.apache.org>
> >>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language
> for
> >>> > > > >>>>>Airavata.
> >>> > > > >>>>> >
> >>> > > > >>>>> >> Hi All,
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> As we all know Airavata has its own workflow language
> call
> >>> > XWF.
> >>> > > > >>>>>When
> >>> > > > >>>>> XWF
> >>> > > > >>>>> >> was introduced, main focus points are interoperability
> and
> >>> > > > >>>>> convertibility.
> >>> > > > >>>>> >> But with years of experience it is convinced that above
> >>> > > > >>>>>requirements
> >>> > > > >>>>> are
> >>> > > > >>>>> >> not really useful when we come to real world use cases.
> >>> And
> >>> > XWF
> >>> > > is
> >>> > > > >>>>> XML
> >>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
> >>> image
> >>> > > it
> >>> > > > >>>>> self.
> >>> > > > >>>>> >> But
> >>> > > > >>>>> >> with the recent changes WSDL part is being removed from
> >>> XWF.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
> >>> languages in
> >>> > > > >>>>> industry
> >>> > > > >>>>> >> and
> >>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
> >>> need
> >>> > to
> >>> > > > >>>>> come up
> >>> > > > >>>>> >> with idea how we should improve Airavata workflow
> >>> language,
> >>> > > either
> >>> > > > >>>>> we can
> >>> > > > >>>>> >> improve existing XWF language, totally change to a new
> >>> > language
> >>> > > > >>>>> available
> >>> > > > >>>>> >> in industry or write a new light weight language. Basic
> >>> > > > >>>>>requirements
> >>> > > > >>>>> that
> >>> > > > >>>>> >> we expect from new improvement are, high usability,
> >>> flexible,
> >>> > > > >>>>>light
> >>> > > > >>>>> weight
> >>> > > > >>>>> >> and real time monitoring support. As you can see above
> >>> > > > >>>>>requirements
> >>> > > > >>>>> are
> >>> > > > >>>>> >> not
> >>> > > > >>>>> >> direct comes with workflow languages but we need
> workflow
> >>> > > language
> >>> > > > >>>>> which
> >>> > > > >>>>> >> help to support above requirements.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> After reading few papers and googling, initially i have
> >>> come
> >>> > up
> >>> > > > >>>>>with
> >>> > > > >>>>> >> following three existing languages,
> >>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>> > > > >>>>> >> 2. WS-BPEL
> >>> > > > >>>>> >> 3. SIDL
> >>> > > > >>>>> >> <
> >>> > > http://computation.llnl.gov/casc/components/index.html#page=home
> >>> > > > >
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific
> >>> domain,
> >>> > > > >>>>> Radical-SAGA
> >>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
> >>> above
> >>> > > > >>>>>three
> >>> > > > >>>>> >> languages we can come up with simple workflow language
> >>> base on
> >>> > > > >>>>> json(or
> >>> > > > >>>>> >> yaml) which support all our requirements for some
> extends.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> It would be grate if I can get more input regarding the
> >>> > $Subject
> >>> > > > >>>>> form the
> >>> > > > >>>>> >> airavata community. You all are more than welcome to
> >>> provide
> >>> > any
> >>> > > > >>>>> type of
> >>> > > > >>>>> >> suggestions.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> Thanks,
> >>> > > > >>>>> >> Shameera.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> 
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> --
> >>> > > > >>>>> >> Best Regards,
> >>> > > > >>>>> >> Shameera Rathnayaka.
> >>> > > > >>>>> >>
> >>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT
> gmail.com
> >>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>> > > > >>>>>
> >>> > > > >>>>>
> >>> > > > >>>>
> >>> > > > >>>>
> >>> > > > >>>> --
> >>> > > > >>>> The best way to predict the future is to invent it - Alan
> Kay
> >>> > > > >>>>
> >>> > > > >>>
> >>> > > > >>>
> >>> > > > >>>
> >>> > > > >>> --
> >>> > > > >>> Best Regards,
> >>> > > > >>> Shameera Rathnayaka.
> >>> > > > >>>
> >>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
> >>> > > > >>>
> >>> > > > >>
> >>> > > > >>
> >>> > > > >>
> >>> > > > >> --
> >>> > > > >> Supun Kamburugamuva
> >>> > > > >> Member, Apache Software Foundation; http://www.apache.org
> >>> > > > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >>> > > > >> Blog: http://supunk.blogspot.com
> >>> > > > >>
> >>> > > > >>
> >>> > > > >
> >>> > > > >
> >>> > > > >--
> >>> > > > >Supun Kamburugamuva
> >>> > > > >Member, Apache Software Foundation; http://www.apache.org
> >>> > > > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >>> > > > >Blog: http://supunk.blogspot.com
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > >
> >>> > >
> >>> > > --
> >>> > > Best Regards,
> >>> > > Shameera Rathnayaka.
> >>> > >
> >>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> > > Blog : http://shameerarathnayaka.blogspot.com/
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Supun Kamburugamuva
> >>> > Member, Apache Software Foundation; http://www.apache.org
> >>> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >>> > Blog: http://supunk.blogspot.com
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards,
> >>> Shameera Rathnayaka.
> >>>
> >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> Blog : http://shameerarathnayaka.blogspot.com/
> >>>
> >>
> >>
> >>
> >> --
> >> Supun Kamburugamuva
> >> Member, Apache Software Foundation; http://www.apache.org
> >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >> Blog: http://supunk.blogspot.com
> >>
> >>
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Shameera Rathnayaka <sh...@gmail.com>.

Hi Amila,

Please see my comments inline.

On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara <th...@gmail.com>
wrote:

> Hello,
>
> I am sorry, I am bit late on this thread. But when reading through this
> thread I simply got lost, what this thread is discussing. I have few
> questions.
>
> 1. @Shameera : Is XWF actually a language to define workflow ? To my
> understanding it was an intermediate representation to convert workflow
> defined in UI to java object model. Was XWF ever used by any airavata user
> to define a workflow graph ?
>

Yes, XWF is the language defined and used by Airavata  to explain the
workflows but it is not well documented. Both server and client sides read
this description language to generate runtime java representation.  so
when you used XBaya to create a workflow with multiple applications, under
the hood XBaya generate xwf which describe that workflow to server.



>
> From initial description what I understood is we are looking for a
> improved "intermediate representation", not a language which describes
> workflows.
> 
>
>

> 2. So what is the exact purpose of this proposed language ?
>         - Is it to hide parallelism in workflows ?
>         - Is it to replace the XBAYA functionalities ? (i hope not)
>


Actually initial idea was to identify good, well-defined and scientific
domain friendly language. Whole purpose of this effort is reduce the entry
barrier of the end user. But later it is understood that introducing a new
language won't fix our issue.


>
> 3. What are we trying to achieve by this proposed language which we cannot
> achieved through workflow UI tool ?
>
> 4. Who is going to use this language ?
>

As I explained, our direction has been changed.By introducing a new
language we are get nothing but nice description file. No functional
improvements etc ... The current language should use all airavata
client(Currently we have only XBaya) side applications to explain the
workflow to the server side.



>
> 5. Why would a user prefer (assuming intended audience of proposed
> language is end users) a language over a Work Flow UI tool such as XBAYA ?
> (In other words what are the things we can do with language which we cannot
> do with UI)
>

Let's say you are going to write a new web base client for Airavata which
generate workflows and launch it. What you need to do is do some magic
with UI and finally come up with description language and send it to server
side. Here you need to learn how to write a valid XWF file and write your
own JS code to generate it. But if airavata has a JavaScript API which can
be used to generate XWF for you by getting JSON objects as input then it
would be great help for client side developers.

Thank,
Shameera.



>
> Sorry, if above questions are in-appropriate, just wanted to understand
> what exactly needed.
>
> Thanks
> -Thejaka Amila
>
>
>
> On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
>> I think I'm not suggesting to create a Workflow interpreter using Python
>> etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
>> and move it to a more higher level component. The more I think about it,
>> the model I'm suggesting is similar to what Hadoop, Storm etc has done for
>> distributed system computations. This model is proven to be successful over
>> the years.
>>
>> Keeping what Airavata does at its core can help you to build a more
>> robust system. If we look at Airavata as middleware to execute applications
>> on computing resources we can simplify what Airavata does and focus on
>> improving the core functionality. All the successful systems have thrived
>> on defining what it does at its core and keeping it simple and being
>> excellent at what it does. In that regard keeping workflow aspect out of
>> Airavata can help you to focus on the core problem. That is to execute an
>> application on a remote computing resource in a fault tolerant and scalable
>> manner.
>>
>> What I'm suggesting is to give the Orchestrator the capability to execute
>> a Driver program that is specified by the user. (This program can be
>> written in Python, Java or any other language). This driver program is
>> similar to what you define in a Hadoop or Storm configuration. The driver
>> program specifies the flow of the computation. It specifies what are the
>> applications needs to be executed, how to manipulate input output. The
>> driver program is the workflow for the user. Because the user specifies the
>> program he can program it to handle workflow steering etc. Every time the
>> user wants to execute this program he can tell Airavata to execute the
>> Driver program.
>>
>> I'm also not 100% sure about all the details. But this can be a new way
>> to look at how systems like Airavata should behave. Your thoughts and
>> suggestions are more than welcome.
>>
>> Thanks,
>> Supun..
>>
>>
>>
>>
>>
>>
>> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>>
>>> Hi Supun,
>>>
>>> I think we were in two context, because I as suggesting a way to
>>> serialize
>>> and deserialize the workflow description while you are suggesting to
>>> implement
>>> some kind of workflow interpreter using Python, where Python client can
>>> send
>>> thrift calls to Airavata server to run the application. I can see with
>>> your
>>> suggested
>>> approach we can control the workflow execution process from client side
>>> which
>>> make it easy to implement workflow debugger.As you mentioned this is a
>>> major change
>>> to Airavata. So we should neatly think as this will change our existing
>>> architecture.
>>>
>>> Still if someone need to use different language java, php, JS etc ... to
>>> run the same
>>> workflow which generated by Python, we need a language independent
>>> workflow
>>> description.
>>> My initial question was what is the best language for this?. But as I
>>> have
>>> explained in
>>> one of my previous reply, It is not matter what language we used Either
>>> we
>>> can use
>>> XML or JSON to write this description, what matters is how easy to
>>> generate
>>> workflow with the provided API. Hence it would be great to have set of
>>> neat
>>> APIs in
>>> different languages.
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <supun06@gmail.com
>>> >
>>> wrote:
>>>
>>> > Hi Shameera,
>>> >
>>> > Using python is a radical approach for workflow I think. But I believe
>>> this
>>> > is a very beautiful and new approach.Here is a possible scenario for
>>> > implementation and running using a Python based library.
>>> >
>>> > The Python library facilitates the creation of Applications and
>>> submitting
>>> > them to Airavata for execution. The underlying functionality is exactly
>>> > similar to what Java clients provides.The only difference is that,
>>> Python
>>> > library should have a more fluent API than Java for easy creation of
>>> > workflows. We can generate the Python clients that talk to Airavata
>>> server
>>> > using Thrift.
>>> >
>>> > Here is an example off the top of my head to a Python script created by
>>> > user for a Workflow. This is a very crude example and we need to come
>>> up
>>> > with a much better API if we are going to go along this path. First we
>>> need
>>> > to write a Python script that can execute a workflow using Airavata.
>>> >
>>> > import airavata
>>> >
>>> > host = Host("localhost", ....)
>>> > app1 = Application(host, ....)
>>> > app2 = Application(host, ....)
>>> >
>>> > # we will connect these applications as a workflow using some topology
>>> > builder or other constructs
>>> >
>>> > wb = WorkFlowBuilder()
>>> > wb.setApp("name1", app1)
>>> > # you can do a simple output transformation here etc
>>> >
>>> > # connects the input of app1 to app2 etc
>>> > wb.setApp("name2", app2).connectInput("name1")
>>> >
>>> > wb.submit()
>>> >
>>> > Now we can load this Python script from XBaya. When XBaya loads this
>>> script
>>> > the Python script can output an XML configuration of the topology,
>>> XBaya
>>> > can render. There are other ways like directly executing the Python
>>> script
>>> > from command line and connecting XBaya indirectly as well. Now you can
>>> run
>>> > the workflow from XBaya. Running the Workflow means just executing the
>>> > Python script.
>>> >
>>> > XBaya gets the notifications through messaging and update the UI
>>> > accordingly.
>>> >
>>> > The users need to write the Python script by hand. XBaya cannot create
>>> the
>>> > script. Because workflow language is an actual python program the
>>> benefits
>>> > are immense. For example user can do workflow steering in the workflow
>>> > itself by subscribing to messages from Airavata.
>>> >
>>> > Thanks,
>>> > Supun..
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>>> > shameerainfo@gmail.com> wrote:
>>> >
>>> > > Hi Supun,
>>> > >
>>> > > I meant to say JS is a well-known client side scripting language i
>>> have
>>> > > messed scripting part. Even we use Python, ultimately we should
>>> convert
>>> > > this to java model in server side, somehow we need to serialized
>>> python
>>> > > representation to the language which java can parse and generate that
>>> > > model. In this case we need to parse python script in java isn't it?
>>> I am
>>> > > not exactly clear how you suggesting to use python here. More
>>> details on
>>> > > how end system works if we use Python would be great help to clearly
>>> > > understand your points.
>>> > >
>>> > > Thanks,
>>> > > Shameera.
>>> > >
>>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>>> > chris.mattmann@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > Have you guys considered using JCC [1] as a means
>>> > > > to expose the workflow API currently in Java as a
>>> > > > Python API?
>>> > > >
>>> > > > We are exploring its use in OODT, and we have already
>>> > > > created a Tika [2] JCC-based python API.
>>> > > >
>>> > > > Cheers,
>>> > > > Chris
>>> > > >
>>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>>> > > > [2] http://github.com/chrismattmann/tika-python/
>>> > > >
>>> > > > ------------------------
>>> > > > Chris Mattmann
>>> > > > chris.mattmann@gmail.com
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > -----Original Message-----
>>> > > > From: Supun Kamburugamuva <su...@gmail.com>
>>> > > > Reply-To: <de...@oodt.apache.org>
>>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>>> > > > To: dev <de...@airavata.apache.org>
>>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
>>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
>>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
>>> > > > <de...@oodt.apache.org>
>>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>>> > Airavata.
>>> > > >
>>> > > > >Once we had an offline discussion about the Airavata Workflow
>>> language
>>> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
>>> came
>>> > > out
>>> > > > >was why we need to invent a different language when a simple
>>> library
>>> > > like
>>> > > > >Python will full fill of Airavata requirements.
>>> > > > >
>>> > > > >There are many benefits in using a Python library as the API for
>>> > > > >controlling Airavata workflows.
>>> > > > >
>>> > > > >1. It is a library, gives the ultimate control over the execution
>>> and
>>> > it
>>> > > > >can be simpler than any domain specific language that we can come
>>> with
>>> > > > >like
>>> > > > >XML, JSON etc
>>> > > > >2. Most people use python and can learn it easily than any Domain
>>> > > specific
>>> > > > >language
>>> > > > >3. You can easily create a Python library for Airavata because
>>> all the
>>> > > > >APIs
>>> > > > >are thrift based.
>>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>>> > > > >
>>> > > > >Any thoughts?
>>> > > > >
>>> > > > >Thanks,
>>> > > > >Supun..
>>> > > > >
>>> > > > >
>>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
>>> > supun06@gmail.com
>>> > > >
>>> > > > >wrote:
>>> > > > >
>>> > > > >> Hi Shameera,
>>> > > > >>
>>> > > > >> Why you prefer JavaScript over a language like Python?
>>> > > > >>
>>> > > > >> Thanks,
>>> > > > >> Supun..
>>> > > > >>
>>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>>> > > > >> shameerainfo@gmail.com> wrote:
>>> > > > >>
>>> > > > >>> Hi,
>>> > > > >>>
>>> > > > >>> First of all thanks everyone for giving valuable inputs. After
>>> > doing
>>> > > > >>>some
>>> > > > >>> background search and talking to different people in the
>>> University
>>> > > > >>>who has
>>> > > > >>> used different workflow languages, I myself convinced that
>>> > > introducing
>>> > > > >>>an
>>> > > > >>> another workflow language is not what actually they need. By
>>> > changing
>>> > > > >>> exiting workflow language to another will not solve problems.
>>> What
>>> > > they
>>> > > > >>> asking is a easy way to construct the workflows. Indirectly
>>> what
>>> > they
>>> > > > >>> asking for a sort of API which they can use to generate the
>>> > workflows
>>> > > > >>>and
>>> > > > >>> run it. Correct me if i am wrong here.
>>> > > > >>>
>>> > > > >>> As most of above replies depict,  if we can get a simple API,
>>> as an
>>> > > > >>> example, for a web based application, JavaScript API would be a
>>> > good
>>> > > > >>> solution, and probably JSON would be a good candidate for
>>> language,
>>> > > > >>>instead
>>> > > > >>> of XML.
>>> > > > >>>
>>> > > > >>> Airavata community already have started to implement web base
>>> GUI.
>>> > > > >>>Hence
>>> > > > >>> introducing a JSON base JavaScript API would be great help.
>>> WDYT?
>>> > > > >>>
>>> > > > >>> Thanks,
>>> > > > >>> Shameera.
>>> > > > >>>
>>> > > > >>>
>>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> > > > >>> alekny7@gmail.com> wrote:
>>> > > > >>>
>>> > > > >>>> Hi,
>>> > > > >>>>
>>> > > > >>>> it is not dataflow instead focused on orchestrating REST
>>> services
>>> > > but
>>> > > > >>>> you may find it useful datapoint - we created worfklow service
>>> > that
>>> > > > >>>>uses
>>> > > > >>>> natively JavaScript and JSON to describe what happens during
>>> > > workflow
>>> > > > >>>> execution:
>>> > > > >>>>
>>> > > >
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>> > > > >>>>
>>> > > > >>>> HTH,
>>> > > > >>>>
>>> > > > >>>> Alek
>>> > > > >>>>
>>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
>>> smarru@apache.org>
>>> > > > >>>>wrote:
>>> > > > >>>>
>>> > > > >>>>> Hi Chris,
>>> > > > >>>>>
>>> > > > >>>>> Great to hear OODT community will be interested in adopting a
>>> > JSON
>>> > > > >>>>> based workflow language and potentially a web based composer
>>> as
>>> > > well.
>>> > > > >>>>> Airavata previously had BPEL support initially through a home
>>> > grown
>>> > > > >>>>> implementation [1] by Alek Slominski and later through
>>> Apache ODE
>>> > > > >>>>>[2]. Also
>>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
>>> read.
>>> > > > >>>>>
>>> > > > >>>>> I am of the same opinion that we should adopt something more
>>> > modern
>>> > > > >>>>>as
>>> > > > >>>>> the challenges from scientific workflows seems to be
>>> converging
>>> > > with
>>> > > > >>>>>the
>>> > > > >>>>> data flow patterns in business workflows.
>>> > > > >>>>>
>>> > > > >>>>> It will be great if we can all compile a list of potential
>>> > > candidates
>>> > > > >>>>> and hack them through.
>>> > > > >>>>>
>>> > > > >>>>> Suresh
>>> > > > >>>>> [1] -
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > >
>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>> > > > >>>>> [2] -
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > >
>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>>> > > > >>>>>time_for_eScience_workflows
>>> > > > >>>>> [3] -
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > >
>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>>> > > > >>>>>6.pdf
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>> > > > >>>>>
>>> > > > >>>>> > Hi Guys,
>>> > > > >>>>> >
>>> > > > >>>>> > I've been interested in this too - we don't per have a
>>> specific
>>> > > > >>>>> > OODT workflow language, but we specific workflows using
>>> XML,
>>> > and
>>> > > > >>>>> > other configuration (we are also thinking of moving to
>>> JSON for
>>> > > > >>>>> > this).
>>> > > > >>>>> >
>>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
>>> > complex
>>> > > > >>>>> > to me.
>>> > > > >>>>> >
>>> > > > >>>>> > I wonder at the end of the day if we should adopt something
>>> > more
>>> > > > >>>>> > modern like PIG or some other data flow type of language
>>> (PIG
>>> > > > >>>>> > is really neat).
>>> > > > >>>>> >
>>> > > > >>>>> > Cheers,
>>> > > > >>>>> > Chris
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > > > >>>>> > Chris Mattmann, Ph.D.
>>> > > > >>>>> > Chief Architect
>>> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
>>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
>>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
>>> > > > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>> > > > >>>>> >
>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
>>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089
>>> USA
>>> > > > >>>>> >
>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> > -----Original Message-----
>>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
>>> > > > >>>>> > <ar...@airavata.apache.org>
>>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>> > > > >>>>> > To: "architecture@airavata.apache.org" <
>>> > > > >>>>> architecture@airavata.apache.org>,
>>> > > > >>>>> > dev <de...@airavata.apache.org>
>>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
>>> > > > >>>>>Airavata.
>>> > > > >>>>> >
>>> > > > >>>>> >> Hi All,
>>> > > > >>>>> >>
>>> > > > >>>>> >> As we all know Airavata has its own workflow language call
>>> > XWF.
>>> > > > >>>>>When
>>> > > > >>>>> XWF
>>> > > > >>>>> >> was introduced, main focus points are interoperability and
>>> > > > >>>>> convertibility.
>>> > > > >>>>> >> But with years of experience it is convinced that above
>>> > > > >>>>>requirements
>>> > > > >>>>> are
>>> > > > >>>>> >> not really useful when we come to real world use cases.
>>> And
>>> > XWF
>>> > > is
>>> > > > >>>>> XML
>>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
>>> image
>>> > > it
>>> > > > >>>>> self.
>>> > > > >>>>> >> But
>>> > > > >>>>> >> with the recent changes WSDL part is being removed from
>>> XWF.
>>> > > > >>>>> >>
>>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
>>> languages in
>>> > > > >>>>> industry
>>> > > > >>>>> >> and
>>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
>>> need
>>> > to
>>> > > > >>>>> come up
>>> > > > >>>>> >> with idea how we should improve Airavata workflow
>>> language,
>>> > > either
>>> > > > >>>>> we can
>>> > > > >>>>> >> improve existing XWF language, totally change to a new
>>> > language
>>> > > > >>>>> available
>>> > > > >>>>> >> in industry or write a new light weight language. Basic
>>> > > > >>>>>requirements
>>> > > > >>>>> that
>>> > > > >>>>> >> we expect from new improvement are, high usability,
>>> flexible,
>>> > > > >>>>>light
>>> > > > >>>>> weight
>>> > > > >>>>> >> and real time monitoring support. As you can see above
>>> > > > >>>>>requirements
>>> > > > >>>>> are
>>> > > > >>>>> >> not
>>> > > > >>>>> >> direct comes with workflow languages but we need workflow
>>> > > language
>>> > > > >>>>> which
>>> > > > >>>>> >> help to support above requirements.
>>> > > > >>>>> >>
>>> > > > >>>>> >> After reading few papers and googling, initially i have
>>> come
>>> > up
>>> > > > >>>>>with
>>> > > > >>>>> >> following three existing languages,
>>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>> > > > >>>>> >> 2. WS-BPEL
>>> > > > >>>>> >> 3. SIDL
>>> > > > >>>>> >> <
>>> > > http://computation.llnl.gov/casc/components/index.html#page=home
>>> > > > >
>>> > > > >>>>> >>
>>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific
>>> domain,
>>> > > > >>>>> Radical-SAGA
>>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
>>> above
>>> > > > >>>>>three
>>> > > > >>>>> >> languages we can come up with simple workflow language
>>> base on
>>> > > > >>>>> json(or
>>> > > > >>>>> >> yaml) which support all our requirements for some extends.
>>> > > > >>>>> >>
>>> > > > >>>>> >> It would be grate if I can get more input regarding the
>>> > $Subject
>>> > > > >>>>> form the
>>> > > > >>>>> >> airavata community. You all are more than welcome to
>>> provide
>>> > any
>>> > > > >>>>> type of
>>> > > > >>>>> >> suggestions.
>>> > > > >>>>> >>
>>> > > > >>>>> >> Thanks,
>>> > > > >>>>> >> Shameera.
>>> > > > >>>>> >>
>>> > > > >>>>> >> 
>>> > > > >>>>> >>
>>> > > > >>>>> >> --
>>> > > > >>>>> >> Best Regards,
>>> > > > >>>>> >> Shameera Rathnayaka.
>>> > > > >>>>> >>
>>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > > >>>>
>>> > > > >>>>
>>> > > > >>>> --
>>> > > > >>>> The best way to predict the future is to invent it - Alan Kay
>>> > > > >>>>
>>> > > > >>>
>>> > > > >>>
>>> > > > >>>
>>> > > > >>> --
>>> > > > >>> Best Regards,
>>> > > > >>> Shameera Rathnayaka.
>>> > > > >>>
>>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
>>> > > > >>>
>>> > > > >>
>>> > > > >>
>>> > > > >>
>>> > > > >> --
>>> > > > >> Supun Kamburugamuva
>>> > > > >> Member, Apache Software Foundation; http://www.apache.org
>>> > > > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>> > > > >> Blog: http://supunk.blogspot.com
>>> > > > >>
>>> > > > >>
>>> > > > >
>>> > > > >
>>> > > > >--
>>> > > > >Supun Kamburugamuva
>>> > > > >Member, Apache Software Foundation; http://www.apache.org
>>> > > > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>> > > > >Blog: http://supunk.blogspot.com
>>> > > >
>>> > > >
>>> > > >
>>> > >
>>> > >
>>> > > --
>>> > > Best Regards,
>>> > > Shameera Rathnayaka.
>>> > >
>>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
>>> > > Blog : http://shameerarathnayaka.blogspot.com/
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Supun Kamburugamuva
>>> > Member, Apache Software Foundation; http://www.apache.org
>>> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>> > Blog: http://supunk.blogspot.com
>>> >
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>>
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>


-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Shameera Rathnayaka <sh...@gmail.com>.

Hi Amila,

Please see my comments inline.

On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara <th...@gmail.com>
wrote:

> Hello,
>
> I am sorry, I am bit late on this thread. But when reading through this
> thread I simply got lost, what this thread is discussing. I have few
> questions.
>
> 1. @Shameera : Is XWF actually a language to define workflow ? To my
> understanding it was an intermediate representation to convert workflow
> defined in UI to java object model. Was XWF ever used by any airavata user
> to define a workflow graph ?
>

Yes, XWF is the language defined and used by Airavata  to explain the
workflows but it is not well documented. Both server and client sides read
this description language to generate runtime java representation.  so
when you used XBaya to create a workflow with multiple applications, under
the hood XBaya generate xwf which describe that workflow to server.



>
> From initial description what I understood is we are looking for a
> improved "intermediate representation", not a language which describes
> workflows.
> 
>
>

> 2. So what is the exact purpose of this proposed language ?
>         - Is it to hide parallelism in workflows ?
>         - Is it to replace the XBAYA functionalities ? (i hope not)
>


Actually initial idea was to identify good, well-defined and scientific
domain friendly language. Whole purpose of this effort is reduce the entry
barrier of the end user. But later it is understood that introducing a new
language won't fix our issue.


>
> 3. What are we trying to achieve by this proposed language which we cannot
> achieved through workflow UI tool ?
>
> 4. Who is going to use this language ?
>

As I explained, our direction has been changed.By introducing a new
language we are get nothing but nice description file. No functional
improvements etc ... The current language should use all airavata
client(Currently we have only XBaya) side applications to explain the
workflow to the server side.



>
> 5. Why would a user prefer (assuming intended audience of proposed
> language is end users) a language over a Work Flow UI tool such as XBAYA ?
> (In other words what are the things we can do with language which we cannot
> do with UI)
>

Let's say you are going to write a new web base client for Airavata which
generate workflows and launch it. What you need to do is do some magic
with UI and finally come up with description language and send it to server
side. Here you need to learn how to write a valid XWF file and write your
own JS code to generate it. But if airavata has a JavaScript API which can
be used to generate XWF for you by getting JSON objects as input then it
would be great help for client side developers.

Thank,
Shameera.



>
> Sorry, if above questions are in-appropriate, just wanted to understand
> what exactly needed.
>
> Thanks
> -Thejaka Amila
>
>
>
> On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
>> I think I'm not suggesting to create a Workflow interpreter using Python
>> etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
>> and move it to a more higher level component. The more I think about it,
>> the model I'm suggesting is similar to what Hadoop, Storm etc has done for
>> distributed system computations. This model is proven to be successful over
>> the years.
>>
>> Keeping what Airavata does at its core can help you to build a more
>> robust system. If we look at Airavata as middleware to execute applications
>> on computing resources we can simplify what Airavata does and focus on
>> improving the core functionality. All the successful systems have thrived
>> on defining what it does at its core and keeping it simple and being
>> excellent at what it does. In that regard keeping workflow aspect out of
>> Airavata can help you to focus on the core problem. That is to execute an
>> application on a remote computing resource in a fault tolerant and scalable
>> manner.
>>
>> What I'm suggesting is to give the Orchestrator the capability to execute
>> a Driver program that is specified by the user. (This program can be
>> written in Python, Java or any other language). This driver program is
>> similar to what you define in a Hadoop or Storm configuration. The driver
>> program specifies the flow of the computation. It specifies what are the
>> applications needs to be executed, how to manipulate input output. The
>> driver program is the workflow for the user. Because the user specifies the
>> program he can program it to handle workflow steering etc. Every time the
>> user wants to execute this program he can tell Airavata to execute the
>> Driver program.
>>
>> I'm also not 100% sure about all the details. But this can be a new way
>> to look at how systems like Airavata should behave. Your thoughts and
>> suggestions are more than welcome.
>>
>> Thanks,
>> Supun..
>>
>>
>>
>>
>>
>>
>> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>>
>>> Hi Supun,
>>>
>>> I think we were in two context, because I as suggesting a way to
>>> serialize
>>> and deserialize the workflow description while you are suggesting to
>>> implement
>>> some kind of workflow interpreter using Python, where Python client can
>>> send
>>> thrift calls to Airavata server to run the application. I can see with
>>> your
>>> suggested
>>> approach we can control the workflow execution process from client side
>>> which
>>> make it easy to implement workflow debugger.As you mentioned this is a
>>> major change
>>> to Airavata. So we should neatly think as this will change our existing
>>> architecture.
>>>
>>> Still if someone need to use different language java, php, JS etc ... to
>>> run the same
>>> workflow which generated by Python, we need a language independent
>>> workflow
>>> description.
>>> My initial question was what is the best language for this?. But as I
>>> have
>>> explained in
>>> one of my previous reply, It is not matter what language we used Either
>>> we
>>> can use
>>> XML or JSON to write this description, what matters is how easy to
>>> generate
>>> workflow with the provided API. Hence it would be great to have set of
>>> neat
>>> APIs in
>>> different languages.
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <supun06@gmail.com
>>> >
>>> wrote:
>>>
>>> > Hi Shameera,
>>> >
>>> > Using python is a radical approach for workflow I think. But I believe
>>> this
>>> > is a very beautiful and new approach.Here is a possible scenario for
>>> > implementation and running using a Python based library.
>>> >
>>> > The Python library facilitates the creation of Applications and
>>> submitting
>>> > them to Airavata for execution. The underlying functionality is exactly
>>> > similar to what Java clients provides.The only difference is that,
>>> Python
>>> > library should have a more fluent API than Java for easy creation of
>>> > workflows. We can generate the Python clients that talk to Airavata
>>> server
>>> > using Thrift.
>>> >
>>> > Here is an example off the top of my head to a Python script created by
>>> > user for a Workflow. This is a very crude example and we need to come
>>> up
>>> > with a much better API if we are going to go along this path. First we
>>> need
>>> > to write a Python script that can execute a workflow using Airavata.
>>> >
>>> > import airavata
>>> >
>>> > host = Host("localhost", ....)
>>> > app1 = Application(host, ....)
>>> > app2 = Application(host, ....)
>>> >
>>> > # we will connect these applications as a workflow using some topology
>>> > builder or other constructs
>>> >
>>> > wb = WorkFlowBuilder()
>>> > wb.setApp("name1", app1)
>>> > # you can do a simple output transformation here etc
>>> >
>>> > # connects the input of app1 to app2 etc
>>> > wb.setApp("name2", app2).connectInput("name1")
>>> >
>>> > wb.submit()
>>> >
>>> > Now we can load this Python script from XBaya. When XBaya loads this
>>> script
>>> > the Python script can output an XML configuration of the topology,
>>> XBaya
>>> > can render. There are other ways like directly executing the Python
>>> script
>>> > from command line and connecting XBaya indirectly as well. Now you can
>>> run
>>> > the workflow from XBaya. Running the Workflow means just executing the
>>> > Python script.
>>> >
>>> > XBaya gets the notifications through messaging and update the UI
>>> > accordingly.
>>> >
>>> > The users need to write the Python script by hand. XBaya cannot create
>>> the
>>> > script. Because workflow language is an actual python program the
>>> benefits
>>> > are immense. For example user can do workflow steering in the workflow
>>> > itself by subscribing to messages from Airavata.
>>> >
>>> > Thanks,
>>> > Supun..
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>>> > shameerainfo@gmail.com> wrote:
>>> >
>>> > > Hi Supun,
>>> > >
>>> > > I meant to say JS is a well-known client side scripting language i
>>> have
>>> > > messed scripting part. Even we use Python, ultimately we should
>>> convert
>>> > > this to java model in server side, somehow we need to serialized
>>> python
>>> > > representation to the language which java can parse and generate that
>>> > > model. In this case we need to parse python script in java isn't it?
>>> I am
>>> > > not exactly clear how you suggesting to use python here. More
>>> details on
>>> > > how end system works if we use Python would be great help to clearly
>>> > > understand your points.
>>> > >
>>> > > Thanks,
>>> > > Shameera.
>>> > >
>>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>>> > chris.mattmann@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > Have you guys considered using JCC [1] as a means
>>> > > > to expose the workflow API currently in Java as a
>>> > > > Python API?
>>> > > >
>>> > > > We are exploring its use in OODT, and we have already
>>> > > > created a Tika [2] JCC-based python API.
>>> > > >
>>> > > > Cheers,
>>> > > > Chris
>>> > > >
>>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>>> > > > [2] http://github.com/chrismattmann/tika-python/
>>> > > >
>>> > > > ------------------------
>>> > > > Chris Mattmann
>>> > > > chris.mattmann@gmail.com
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > -----Original Message-----
>>> > > > From: Supun Kamburugamuva <su...@gmail.com>
>>> > > > Reply-To: <de...@oodt.apache.org>
>>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>>> > > > To: dev <de...@airavata.apache.org>
>>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
>>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
>>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
>>> > > > <de...@oodt.apache.org>
>>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>>> > Airavata.
>>> > > >
>>> > > > >Once we had an offline discussion about the Airavata Workflow
>>> language
>>> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
>>> came
>>> > > out
>>> > > > >was why we need to invent a different language when a simple
>>> library
>>> > > like
>>> > > > >Python will full fill of Airavata requirements.
>>> > > > >
>>> > > > >There are many benefits in using a Python library as the API for
>>> > > > >controlling Airavata workflows.
>>> > > > >
>>> > > > >1. It is a library, gives the ultimate control over the execution
>>> and
>>> > it
>>> > > > >can be simpler than any domain specific language that we can come
>>> with
>>> > > > >like
>>> > > > >XML, JSON etc
>>> > > > >2. Most people use python and can learn it easily than any Domain
>>> > > specific
>>> > > > >language
>>> > > > >3. You can easily create a Python library for Airavata because
>>> all the
>>> > > > >APIs
>>> > > > >are thrift based.
>>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>>> > > > >
>>> > > > >Any thoughts?
>>> > > > >
>>> > > > >Thanks,
>>> > > > >Supun..
>>> > > > >
>>> > > > >
>>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
>>> > supun06@gmail.com
>>> > > >
>>> > > > >wrote:
>>> > > > >
>>> > > > >> Hi Shameera,
>>> > > > >>
>>> > > > >> Why you prefer JavaScript over a language like Python?
>>> > > > >>
>>> > > > >> Thanks,
>>> > > > >> Supun..
>>> > > > >>
>>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>>> > > > >> shameerainfo@gmail.com> wrote:
>>> > > > >>
>>> > > > >>> Hi,
>>> > > > >>>
>>> > > > >>> First of all thanks everyone for giving valuable inputs. After
>>> > doing
>>> > > > >>>some
>>> > > > >>> background search and talking to different people in the
>>> University
>>> > > > >>>who has
>>> > > > >>> used different workflow languages, I myself convinced that
>>> > > introducing
>>> > > > >>>an
>>> > > > >>> another workflow language is not what actually they need. By
>>> > changing
>>> > > > >>> exiting workflow language to another will not solve problems.
>>> What
>>> > > they
>>> > > > >>> asking is a easy way to construct the workflows. Indirectly
>>> what
>>> > they
>>> > > > >>> asking for a sort of API which they can use to generate the
>>> > workflows
>>> > > > >>>and
>>> > > > >>> run it. Correct me if i am wrong here.
>>> > > > >>>
>>> > > > >>> As most of above replies depict,  if we can get a simple API,
>>> as an
>>> > > > >>> example, for a web based application, JavaScript API would be a
>>> > good
>>> > > > >>> solution, and probably JSON would be a good candidate for
>>> language,
>>> > > > >>>instead
>>> > > > >>> of XML.
>>> > > > >>>
>>> > > > >>> Airavata community already have started to implement web base
>>> GUI.
>>> > > > >>>Hence
>>> > > > >>> introducing a JSON base JavaScript API would be great help.
>>> WDYT?
>>> > > > >>>
>>> > > > >>> Thanks,
>>> > > > >>> Shameera.
>>> > > > >>>
>>> > > > >>>
>>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> > > > >>> alekny7@gmail.com> wrote:
>>> > > > >>>
>>> > > > >>>> Hi,
>>> > > > >>>>
>>> > > > >>>> it is not dataflow instead focused on orchestrating REST
>>> services
>>> > > but
>>> > > > >>>> you may find it useful datapoint - we created worfklow service
>>> > that
>>> > > > >>>>uses
>>> > > > >>>> natively JavaScript and JSON to describe what happens during
>>> > > workflow
>>> > > > >>>> execution:
>>> > > > >>>>
>>> > > >
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>> > > > >>>>
>>> > > > >>>> HTH,
>>> > > > >>>>
>>> > > > >>>> Alek
>>> > > > >>>>
>>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
>>> smarru@apache.org>
>>> > > > >>>>wrote:
>>> > > > >>>>
>>> > > > >>>>> Hi Chris,
>>> > > > >>>>>
>>> > > > >>>>> Great to hear OODT community will be interested in adopting a
>>> > JSON
>>> > > > >>>>> based workflow language and potentially a web based composer
>>> as
>>> > > well.
>>> > > > >>>>> Airavata previously had BPEL support initially through a home
>>> > grown
>>> > > > >>>>> implementation [1] by Alek Slominski and later through
>>> Apache ODE
>>> > > > >>>>>[2]. Also
>>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
>>> read.
>>> > > > >>>>>
>>> > > > >>>>> I am of the same opinion that we should adopt something more
>>> > modern
>>> > > > >>>>>as
>>> > > > >>>>> the challenges from scientific workflows seems to be
>>> converging
>>> > > with
>>> > > > >>>>>the
>>> > > > >>>>> data flow patterns in business workflows.
>>> > > > >>>>>
>>> > > > >>>>> It will be great if we can all compile a list of potential
>>> > > candidates
>>> > > > >>>>> and hack them through.
>>> > > > >>>>>
>>> > > > >>>>> Suresh
>>> > > > >>>>> [1] -
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > >
>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>> > > > >>>>> [2] -
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > >
>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>>> > > > >>>>>time_for_eScience_workflows
>>> > > > >>>>> [3] -
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > >
>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>>> > > > >>>>>6.pdf
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>> > > > >>>>>
>>> > > > >>>>> > Hi Guys,
>>> > > > >>>>> >
>>> > > > >>>>> > I've been interested in this too - we don't per have a
>>> specific
>>> > > > >>>>> > OODT workflow language, but we specific workflows using
>>> XML,
>>> > and
>>> > > > >>>>> > other configuration (we are also thinking of moving to
>>> JSON for
>>> > > > >>>>> > this).
>>> > > > >>>>> >
>>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
>>> > complex
>>> > > > >>>>> > to me.
>>> > > > >>>>> >
>>> > > > >>>>> > I wonder at the end of the day if we should adopt something
>>> > more
>>> > > > >>>>> > modern like PIG or some other data flow type of language
>>> (PIG
>>> > > > >>>>> > is really neat).
>>> > > > >>>>> >
>>> > > > >>>>> > Cheers,
>>> > > > >>>>> > Chris
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > > > >>>>> > Chris Mattmann, Ph.D.
>>> > > > >>>>> > Chief Architect
>>> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
>>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
>>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
>>> > > > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>> > > > >>>>> >
>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
>>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089
>>> USA
>>> > > > >>>>> >
>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> > -----Original Message-----
>>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
>>> > > > >>>>> > <ar...@airavata.apache.org>
>>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>> > > > >>>>> > To: "architecture@airavata.apache.org" <
>>> > > > >>>>> architecture@airavata.apache.org>,
>>> > > > >>>>> > dev <de...@airavata.apache.org>
>>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
>>> > > > >>>>>Airavata.
>>> > > > >>>>> >
>>> > > > >>>>> >> Hi All,
>>> > > > >>>>> >>
>>> > > > >>>>> >> As we all know Airavata has its own workflow language call
>>> > XWF.
>>> > > > >>>>>When
>>> > > > >>>>> XWF
>>> > > > >>>>> >> was introduced, main focus points are interoperability and
>>> > > > >>>>> convertibility.
>>> > > > >>>>> >> But with years of experience it is convinced that above
>>> > > > >>>>>requirements
>>> > > > >>>>> are
>>> > > > >>>>> >> not really useful when we come to real world use cases.
>>> And
>>> > XWF
>>> > > is
>>> > > > >>>>> XML
>>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
>>> image
>>> > > it
>>> > > > >>>>> self.
>>> > > > >>>>> >> But
>>> > > > >>>>> >> with the recent changes WSDL part is being removed from
>>> XWF.
>>> > > > >>>>> >>
>>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
>>> languages in
>>> > > > >>>>> industry
>>> > > > >>>>> >> and
>>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
>>> need
>>> > to
>>> > > > >>>>> come up
>>> > > > >>>>> >> with idea how we should improve Airavata workflow
>>> language,
>>> > > either
>>> > > > >>>>> we can
>>> > > > >>>>> >> improve existing XWF language, totally change to a new
>>> > language
>>> > > > >>>>> available
>>> > > > >>>>> >> in industry or write a new light weight language. Basic
>>> > > > >>>>>requirements
>>> > > > >>>>> that
>>> > > > >>>>> >> we expect from new improvement are, high usability,
>>> flexible,
>>> > > > >>>>>light
>>> > > > >>>>> weight
>>> > > > >>>>> >> and real time monitoring support. As you can see above
>>> > > > >>>>>requirements
>>> > > > >>>>> are
>>> > > > >>>>> >> not
>>> > > > >>>>> >> direct comes with workflow languages but we need workflow
>>> > > language
>>> > > > >>>>> which
>>> > > > >>>>> >> help to support above requirements.
>>> > > > >>>>> >>
>>> > > > >>>>> >> After reading few papers and googling, initially i have
>>> come
>>> > up
>>> > > > >>>>>with
>>> > > > >>>>> >> following three existing languages,
>>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>> > > > >>>>> >> 2. WS-BPEL
>>> > > > >>>>> >> 3. SIDL
>>> > > > >>>>> >> <
>>> > > http://computation.llnl.gov/casc/components/index.html#page=home
>>> > > > >
>>> > > > >>>>> >>
>>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific
>>> domain,
>>> > > > >>>>> Radical-SAGA
>>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
>>> above
>>> > > > >>>>>three
>>> > > > >>>>> >> languages we can come up with simple workflow language
>>> base on
>>> > > > >>>>> json(or
>>> > > > >>>>> >> yaml) which support all our requirements for some extends.
>>> > > > >>>>> >>
>>> > > > >>>>> >> It would be grate if I can get more input regarding the
>>> > $Subject
>>> > > > >>>>> form the
>>> > > > >>>>> >> airavata community. You all are more than welcome to
>>> provide
>>> > any
>>> > > > >>>>> type of
>>> > > > >>>>> >> suggestions.
>>> > > > >>>>> >>
>>> > > > >>>>> >> Thanks,
>>> > > > >>>>> >> Shameera.
>>> > > > >>>>> >>
>>> > > > >>>>> >> 
>>> > > > >>>>> >>
>>> > > > >>>>> >> --
>>> > > > >>>>> >> Best Regards,
>>> > > > >>>>> >> Shameera Rathnayaka.
>>> > > > >>>>> >>
>>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > > >>>>
>>> > > > >>>>
>>> > > > >>>> --
>>> > > > >>>> The best way to predict the future is to invent it - Alan Kay
>>> > > > >>>>
>>> > > > >>>
>>> > > > >>>
>>> > > > >>>
>>> > > > >>> --
>>> > > > >>> Best Regards,
>>> > > > >>> Shameera Rathnayaka.
>>> > > > >>>
>>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
>>> > > > >>>
>>> > > > >>
>>> > > > >>
>>> > > > >>
>>> > > > >> --
>>> > > > >> Supun Kamburugamuva
>>> > > > >> Member, Apache Software Foundation; http://www.apache.org
>>> > > > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>> > > > >> Blog: http://supunk.blogspot.com
>>> > > > >>
>>> > > > >>
>>> > > > >
>>> > > > >
>>> > > > >--
>>> > > > >Supun Kamburugamuva
>>> > > > >Member, Apache Software Foundation; http://www.apache.org
>>> > > > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>> > > > >Blog: http://supunk.blogspot.com
>>> > > >
>>> > > >
>>> > > >
>>> > >
>>> > >
>>> > > --
>>> > > Best Regards,
>>> > > Shameera Rathnayaka.
>>> > >
>>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
>>> > > Blog : http://shameerarathnayaka.blogspot.com/
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Supun Kamburugamuva
>>> > Member, Apache Software Foundation; http://www.apache.org
>>> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>> > Blog: http://supunk.blogspot.com
>>> >
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>>
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>


-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Shameera Rathnayaka <sh...@gmail.com>.

Hi Amila,

Please see my comments inline.

On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara <th...@gmail.com>
wrote:

> Hello,
>
> I am sorry, I am bit late on this thread. But when reading through this
> thread I simply got lost, what this thread is discussing. I have few
> questions.
>
> 1. @Shameera : Is XWF actually a language to define workflow ? To my
> understanding it was an intermediate representation to convert workflow
> defined in UI to java object model. Was XWF ever used by any airavata user
> to define a workflow graph ?
>

Yes, XWF is the language defined and used by Airavata  to explain the
workflows but it is not well documented. Both server and client sides read
this description language to generate runtime java representation.  so
when you used XBaya to create a workflow with multiple applications, under
the hood XBaya generate xwf which describe that workflow to server.



>
> From initial description what I understood is we are looking for a
> improved "intermediate representation", not a language which describes
> workflows.
> 
>
>

> 2. So what is the exact purpose of this proposed language ?
>         - Is it to hide parallelism in workflows ?
>         - Is it to replace the XBAYA functionalities ? (i hope not)
>


Actually initial idea was to identify good, well-defined and scientific
domain friendly language. Whole purpose of this effort is reduce the entry
barrier of the end user. But later it is understood that introducing a new
language won't fix our issue.


>
> 3. What are we trying to achieve by this proposed language which we cannot
> achieved through workflow UI tool ?
>
> 4. Who is going to use this language ?
>

As I explained, our direction has been changed.By introducing a new
language we are get nothing but nice description file. No functional
improvements etc ... The current language should use all airavata
client(Currently we have only XBaya) side applications to explain the
workflow to the server side.



>
> 5. Why would a user prefer (assuming intended audience of proposed
> language is end users) a language over a Work Flow UI tool such as XBAYA ?
> (In other words what are the things we can do with language which we cannot
> do with UI)
>

Let's say you are going to write a new web base client for Airavata which
generate workflows and launch it. What you need to do is do some magic
with UI and finally come up with description language and send it to server
side. Here you need to learn how to write a valid XWF file and write your
own JS code to generate it. But if airavata has a JavaScript API which can
be used to generate XWF for you by getting JSON objects as input then it
would be great help for client side developers.

Thank,
Shameera.



>
> Sorry, if above questions are in-appropriate, just wanted to understand
> what exactly needed.
>
> Thanks
> -Thejaka Amila
>
>
>
> On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
>> I think I'm not suggesting to create a Workflow interpreter using Python
>> etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
>> and move it to a more higher level component. The more I think about it,
>> the model I'm suggesting is similar to what Hadoop, Storm etc has done for
>> distributed system computations. This model is proven to be successful over
>> the years.
>>
>> Keeping what Airavata does at its core can help you to build a more
>> robust system. If we look at Airavata as middleware to execute applications
>> on computing resources we can simplify what Airavata does and focus on
>> improving the core functionality. All the successful systems have thrived
>> on defining what it does at its core and keeping it simple and being
>> excellent at what it does. In that regard keeping workflow aspect out of
>> Airavata can help you to focus on the core problem. That is to execute an
>> application on a remote computing resource in a fault tolerant and scalable
>> manner.
>>
>> What I'm suggesting is to give the Orchestrator the capability to execute
>> a Driver program that is specified by the user. (This program can be
>> written in Python, Java or any other language). This driver program is
>> similar to what you define in a Hadoop or Storm configuration. The driver
>> program specifies the flow of the computation. It specifies what are the
>> applications needs to be executed, how to manipulate input output. The
>> driver program is the workflow for the user. Because the user specifies the
>> program he can program it to handle workflow steering etc. Every time the
>> user wants to execute this program he can tell Airavata to execute the
>> Driver program.
>>
>> I'm also not 100% sure about all the details. But this can be a new way
>> to look at how systems like Airavata should behave. Your thoughts and
>> suggestions are more than welcome.
>>
>> Thanks,
>> Supun..
>>
>>
>>
>>
>>
>>
>> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>>
>>> Hi Supun,
>>>
>>> I think we were in two context, because I as suggesting a way to
>>> serialize
>>> and deserialize the workflow description while you are suggesting to
>>> implement
>>> some kind of workflow interpreter using Python, where Python client can
>>> send
>>> thrift calls to Airavata server to run the application. I can see with
>>> your
>>> suggested
>>> approach we can control the workflow execution process from client side
>>> which
>>> make it easy to implement workflow debugger.As you mentioned this is a
>>> major change
>>> to Airavata. So we should neatly think as this will change our existing
>>> architecture.
>>>
>>> Still if someone need to use different language java, php, JS etc ... to
>>> run the same
>>> workflow which generated by Python, we need a language independent
>>> workflow
>>> description.
>>> My initial question was what is the best language for this?. But as I
>>> have
>>> explained in
>>> one of my previous reply, It is not matter what language we used Either
>>> we
>>> can use
>>> XML or JSON to write this description, what matters is how easy to
>>> generate
>>> workflow with the provided API. Hence it would be great to have set of
>>> neat
>>> APIs in
>>> different languages.
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <supun06@gmail.com
>>> >
>>> wrote:
>>>
>>> > Hi Shameera,
>>> >
>>> > Using python is a radical approach for workflow I think. But I believe
>>> this
>>> > is a very beautiful and new approach.Here is a possible scenario for
>>> > implementation and running using a Python based library.
>>> >
>>> > The Python library facilitates the creation of Applications and
>>> submitting
>>> > them to Airavata for execution. The underlying functionality is exactly
>>> > similar to what Java clients provides.The only difference is that,
>>> Python
>>> > library should have a more fluent API than Java for easy creation of
>>> > workflows. We can generate the Python clients that talk to Airavata
>>> server
>>> > using Thrift.
>>> >
>>> > Here is an example off the top of my head to a Python script created by
>>> > user for a Workflow. This is a very crude example and we need to come
>>> up
>>> > with a much better API if we are going to go along this path. First we
>>> need
>>> > to write a Python script that can execute a workflow using Airavata.
>>> >
>>> > import airavata
>>> >
>>> > host = Host("localhost", ....)
>>> > app1 = Application(host, ....)
>>> > app2 = Application(host, ....)
>>> >
>>> > # we will connect these applications as a workflow using some topology
>>> > builder or other constructs
>>> >
>>> > wb = WorkFlowBuilder()
>>> > wb.setApp("name1", app1)
>>> > # you can do a simple output transformation here etc
>>> >
>>> > # connects the input of app1 to app2 etc
>>> > wb.setApp("name2", app2).connectInput("name1")
>>> >
>>> > wb.submit()
>>> >
>>> > Now we can load this Python script from XBaya. When XBaya loads this
>>> script
>>> > the Python script can output an XML configuration of the topology,
>>> XBaya
>>> > can render. There are other ways like directly executing the Python
>>> script
>>> > from command line and connecting XBaya indirectly as well. Now you can
>>> run
>>> > the workflow from XBaya. Running the Workflow means just executing the
>>> > Python script.
>>> >
>>> > XBaya gets the notifications through messaging and update the UI
>>> > accordingly.
>>> >
>>> > The users need to write the Python script by hand. XBaya cannot create
>>> the
>>> > script. Because workflow language is an actual python program the
>>> benefits
>>> > are immense. For example user can do workflow steering in the workflow
>>> > itself by subscribing to messages from Airavata.
>>> >
>>> > Thanks,
>>> > Supun..
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>>> > shameerainfo@gmail.com> wrote:
>>> >
>>> > > Hi Supun,
>>> > >
>>> > > I meant to say JS is a well-known client side scripting language i
>>> have
>>> > > messed scripting part. Even we use Python, ultimately we should
>>> convert
>>> > > this to java model in server side, somehow we need to serialized
>>> python
>>> > > representation to the language which java can parse and generate that
>>> > > model. In this case we need to parse python script in java isn't it?
>>> I am
>>> > > not exactly clear how you suggesting to use python here. More
>>> details on
>>> > > how end system works if we use Python would be great help to clearly
>>> > > understand your points.
>>> > >
>>> > > Thanks,
>>> > > Shameera.
>>> > >
>>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>>> > chris.mattmann@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > Have you guys considered using JCC [1] as a means
>>> > > > to expose the workflow API currently in Java as a
>>> > > > Python API?
>>> > > >
>>> > > > We are exploring its use in OODT, and we have already
>>> > > > created a Tika [2] JCC-based python API.
>>> > > >
>>> > > > Cheers,
>>> > > > Chris
>>> > > >
>>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>>> > > > [2] http://github.com/chrismattmann/tika-python/
>>> > > >
>>> > > > ------------------------
>>> > > > Chris Mattmann
>>> > > > chris.mattmann@gmail.com
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > -----Original Message-----
>>> > > > From: Supun Kamburugamuva <su...@gmail.com>
>>> > > > Reply-To: <de...@oodt.apache.org>
>>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>>> > > > To: dev <de...@airavata.apache.org>
>>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
>>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
>>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
>>> > > > <de...@oodt.apache.org>
>>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>>> > Airavata.
>>> > > >
>>> > > > >Once we had an offline discussion about the Airavata Workflow
>>> language
>>> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
>>> came
>>> > > out
>>> > > > >was why we need to invent a different language when a simple
>>> library
>>> > > like
>>> > > > >Python will full fill of Airavata requirements.
>>> > > > >
>>> > > > >There are many benefits in using a Python library as the API for
>>> > > > >controlling Airavata workflows.
>>> > > > >
>>> > > > >1. It is a library, gives the ultimate control over the execution
>>> and
>>> > it
>>> > > > >can be simpler than any domain specific language that we can come
>>> with
>>> > > > >like
>>> > > > >XML, JSON etc
>>> > > > >2. Most people use python and can learn it easily than any Domain
>>> > > specific
>>> > > > >language
>>> > > > >3. You can easily create a Python library for Airavata because
>>> all the
>>> > > > >APIs
>>> > > > >are thrift based.
>>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>>> > > > >
>>> > > > >Any thoughts?
>>> > > > >
>>> > > > >Thanks,
>>> > > > >Supun..
>>> > > > >
>>> > > > >
>>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
>>> > supun06@gmail.com
>>> > > >
>>> > > > >wrote:
>>> > > > >
>>> > > > >> Hi Shameera,
>>> > > > >>
>>> > > > >> Why you prefer JavaScript over a language like Python?
>>> > > > >>
>>> > > > >> Thanks,
>>> > > > >> Supun..
>>> > > > >>
>>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>>> > > > >> shameerainfo@gmail.com> wrote:
>>> > > > >>
>>> > > > >>> Hi,
>>> > > > >>>
>>> > > > >>> First of all thanks everyone for giving valuable inputs. After
>>> > doing
>>> > > > >>>some
>>> > > > >>> background search and talking to different people in the
>>> University
>>> > > > >>>who has
>>> > > > >>> used different workflow languages, I myself convinced that
>>> > > introducing
>>> > > > >>>an
>>> > > > >>> another workflow language is not what actually they need. By
>>> > changing
>>> > > > >>> exiting workflow language to another will not solve problems.
>>> What
>>> > > they
>>> > > > >>> asking is a easy way to construct the workflows. Indirectly
>>> what
>>> > they
>>> > > > >>> asking for a sort of API which they can use to generate the
>>> > workflows
>>> > > > >>>and
>>> > > > >>> run it. Correct me if i am wrong here.
>>> > > > >>>
>>> > > > >>> As most of above replies depict,  if we can get a simple API,
>>> as an
>>> > > > >>> example, for a web based application, JavaScript API would be a
>>> > good
>>> > > > >>> solution, and probably JSON would be a good candidate for
>>> language,
>>> > > > >>>instead
>>> > > > >>> of XML.
>>> > > > >>>
>>> > > > >>> Airavata community already have started to implement web base
>>> GUI.
>>> > > > >>>Hence
>>> > > > >>> introducing a JSON base JavaScript API would be great help.
>>> WDYT?
>>> > > > >>>
>>> > > > >>> Thanks,
>>> > > > >>> Shameera.
>>> > > > >>>
>>> > > > >>>
>>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> > > > >>> alekny7@gmail.com> wrote:
>>> > > > >>>
>>> > > > >>>> Hi,
>>> > > > >>>>
>>> > > > >>>> it is not dataflow instead focused on orchestrating REST
>>> services
>>> > > but
>>> > > > >>>> you may find it useful datapoint - we created worfklow service
>>> > that
>>> > > > >>>>uses
>>> > > > >>>> natively JavaScript and JSON to describe what happens during
>>> > > workflow
>>> > > > >>>> execution:
>>> > > > >>>>
>>> > > >
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>> > > > >>>>
>>> > > > >>>> HTH,
>>> > > > >>>>
>>> > > > >>>> Alek
>>> > > > >>>>
>>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
>>> smarru@apache.org>
>>> > > > >>>>wrote:
>>> > > > >>>>
>>> > > > >>>>> Hi Chris,
>>> > > > >>>>>
>>> > > > >>>>> Great to hear OODT community will be interested in adopting a
>>> > JSON
>>> > > > >>>>> based workflow language and potentially a web based composer
>>> as
>>> > > well.
>>> > > > >>>>> Airavata previously had BPEL support initially through a home
>>> > grown
>>> > > > >>>>> implementation [1] by Alek Slominski and later through
>>> Apache ODE
>>> > > > >>>>>[2]. Also
>>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
>>> read.
>>> > > > >>>>>
>>> > > > >>>>> I am of the same opinion that we should adopt something more
>>> > modern
>>> > > > >>>>>as
>>> > > > >>>>> the challenges from scientific workflows seems to be
>>> converging
>>> > > with
>>> > > > >>>>>the
>>> > > > >>>>> data flow patterns in business workflows.
>>> > > > >>>>>
>>> > > > >>>>> It will be great if we can all compile a list of potential
>>> > > candidates
>>> > > > >>>>> and hack them through.
>>> > > > >>>>>
>>> > > > >>>>> Suresh
>>> > > > >>>>> [1] -
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > >
>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>> > > > >>>>> [2] -
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > >
>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>>> > > > >>>>>time_for_eScience_workflows
>>> > > > >>>>> [3] -
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > >
>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>>> > > > >>>>>6.pdf
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>> > > > >>>>>
>>> > > > >>>>> > Hi Guys,
>>> > > > >>>>> >
>>> > > > >>>>> > I've been interested in this too - we don't per have a
>>> specific
>>> > > > >>>>> > OODT workflow language, but we specific workflows using
>>> XML,
>>> > and
>>> > > > >>>>> > other configuration (we are also thinking of moving to
>>> JSON for
>>> > > > >>>>> > this).
>>> > > > >>>>> >
>>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
>>> > complex
>>> > > > >>>>> > to me.
>>> > > > >>>>> >
>>> > > > >>>>> > I wonder at the end of the day if we should adopt something
>>> > more
>>> > > > >>>>> > modern like PIG or some other data flow type of language
>>> (PIG
>>> > > > >>>>> > is really neat).
>>> > > > >>>>> >
>>> > > > >>>>> > Cheers,
>>> > > > >>>>> > Chris
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > > > >>>>> > Chris Mattmann, Ph.D.
>>> > > > >>>>> > Chief Architect
>>> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
>>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
>>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
>>> > > > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>> > > > >>>>> >
>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
>>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089
>>> USA
>>> > > > >>>>> >
>>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> >
>>> > > > >>>>> > -----Original Message-----
>>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
>>> > > > >>>>> > <ar...@airavata.apache.org>
>>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>> > > > >>>>> > To: "architecture@airavata.apache.org" <
>>> > > > >>>>> architecture@airavata.apache.org>,
>>> > > > >>>>> > dev <de...@airavata.apache.org>
>>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
>>> > > > >>>>>Airavata.
>>> > > > >>>>> >
>>> > > > >>>>> >> Hi All,
>>> > > > >>>>> >>
>>> > > > >>>>> >> As we all know Airavata has its own workflow language call
>>> > XWF.
>>> > > > >>>>>When
>>> > > > >>>>> XWF
>>> > > > >>>>> >> was introduced, main focus points are interoperability and
>>> > > > >>>>> convertibility.
>>> > > > >>>>> >> But with years of experience it is convinced that above
>>> > > > >>>>>requirements
>>> > > > >>>>> are
>>> > > > >>>>> >> not really useful when we come to real world use cases.
>>> And
>>> > XWF
>>> > > is
>>> > > > >>>>> XML
>>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
>>> image
>>> > > it
>>> > > > >>>>> self.
>>> > > > >>>>> >> But
>>> > > > >>>>> >> with the recent changes WSDL part is being removed from
>>> XWF.
>>> > > > >>>>> >>
>>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
>>> languages in
>>> > > > >>>>> industry
>>> > > > >>>>> >> and
>>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
>>> need
>>> > to
>>> > > > >>>>> come up
>>> > > > >>>>> >> with idea how we should improve Airavata workflow
>>> language,
>>> > > either
>>> > > > >>>>> we can
>>> > > > >>>>> >> improve existing XWF language, totally change to a new
>>> > language
>>> > > > >>>>> available
>>> > > > >>>>> >> in industry or write a new light weight language. Basic
>>> > > > >>>>>requirements
>>> > > > >>>>> that
>>> > > > >>>>> >> we expect from new improvement are, high usability,
>>> flexible,
>>> > > > >>>>>light
>>> > > > >>>>> weight
>>> > > > >>>>> >> and real time monitoring support. As you can see above
>>> > > > >>>>>requirements
>>> > > > >>>>> are
>>> > > > >>>>> >> not
>>> > > > >>>>> >> direct comes with workflow languages but we need workflow
>>> > > language
>>> > > > >>>>> which
>>> > > > >>>>> >> help to support above requirements.
>>> > > > >>>>> >>
>>> > > > >>>>> >> After reading few papers and googling, initially i have
>>> come
>>> > up
>>> > > > >>>>>with
>>> > > > >>>>> >> following three existing languages,
>>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>> > > > >>>>> >> 2. WS-BPEL
>>> > > > >>>>> >> 3. SIDL
>>> > > > >>>>> >> <
>>> > > http://computation.llnl.gov/casc/components/index.html#page=home
>>> > > > >
>>> > > > >>>>> >>
>>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific
>>> domain,
>>> > > > >>>>> Radical-SAGA
>>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
>>> above
>>> > > > >>>>>three
>>> > > > >>>>> >> languages we can come up with simple workflow language
>>> base on
>>> > > > >>>>> json(or
>>> > > > >>>>> >> yaml) which support all our requirements for some extends.
>>> > > > >>>>> >>
>>> > > > >>>>> >> It would be grate if I can get more input regarding the
>>> > $Subject
>>> > > > >>>>> form the
>>> > > > >>>>> >> airavata community. You all are more than welcome to
>>> provide
>>> > any
>>> > > > >>>>> type of
>>> > > > >>>>> >> suggestions.
>>> > > > >>>>> >>
>>> > > > >>>>> >> Thanks,
>>> > > > >>>>> >> Shameera.
>>> > > > >>>>> >>
>>> > > > >>>>> >> 
>>> > > > >>>>> >>
>>> > > > >>>>> >> --
>>> > > > >>>>> >> Best Regards,
>>> > > > >>>>> >> Shameera Rathnayaka.
>>> > > > >>>>> >>
>>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>> > > > >>>>>
>>> > > > >>>>>
>>> > > > >>>>
>>> > > > >>>>
>>> > > > >>>> --
>>> > > > >>>> The best way to predict the future is to invent it - Alan Kay
>>> > > > >>>>
>>> > > > >>>
>>> > > > >>>
>>> > > > >>>
>>> > > > >>> --
>>> > > > >>> Best Regards,
>>> > > > >>> Shameera Rathnayaka.
>>> > > > >>>
>>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
>>> > > > >>>
>>> > > > >>
>>> > > > >>
>>> > > > >>
>>> > > > >> --
>>> > > > >> Supun Kamburugamuva
>>> > > > >> Member, Apache Software Foundation; http://www.apache.org
>>> > > > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>> > > > >> Blog: http://supunk.blogspot.com
>>> > > > >>
>>> > > > >>
>>> > > > >
>>> > > > >
>>> > > > >--
>>> > > > >Supun Kamburugamuva
>>> > > > >Member, Apache Software Foundation; http://www.apache.org
>>> > > > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>> > > > >Blog: http://supunk.blogspot.com
>>> > > >
>>> > > >
>>> > > >
>>> > >
>>> > >
>>> > > --
>>> > > Best Regards,
>>> > > Shameera Rathnayaka.
>>> > >
>>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
>>> > > Blog : http://shameerarathnayaka.blogspot.com/
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Supun Kamburugamuva
>>> > Member, Apache Software Foundation; http://www.apache.org
>>> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>>> > Blog: http://supunk.blogspot.com
>>> >
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>>
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>


-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Amila Jayasekara <th...@gmail.com>.

Hello,

I am sorry, I am bit late on this thread. But when reading through this
thread I simply got lost, what this thread is discussing. I have few
questions.

1. @Shameera : Is XWF actually a language to define workflow ? To my
understanding it was an intermediate representation to convert workflow
defined in UI to java object model. Was XWF ever used by any airavata user
to define a workflow graph ?

>From initial description what I understood is we are looking for a improved
"intermediate representation", not a language which describes workflows.

2. So what is the exact purpose of this proposed language ?
        - Is it to hide parallelism in workflows ?
        - Is it to replace the XBAYA functionalities ? (i hope not)

3. What are we trying to achieve by this proposed language which we cannot
achieved through workflow UI tool ?

4. Who is going to use this language ?

5. Why would a user prefer (assuming intended audience of proposed language
is end users) a language over a Work Flow UI tool such as XBAYA ? (In other
words what are the things we can do with language which we cannot do with
UI)

Sorry, if above questions are in-appropriate, just wanted to understand
what exactly needed.

Thanks
-Thejaka Amila



On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:

> I think I'm not suggesting to create a Workflow interpreter using Python
> etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
> and move it to a more higher level component. The more I think about it,
> the model I'm suggesting is similar to what Hadoop, Storm etc has done for
> distributed system computations. This model is proven to be successful over
> the years.
>
> Keeping what Airavata does at its core can help you to build a more robust
> system. If we look at Airavata as middleware to execute applications on
> computing resources we can simplify what Airavata does and focus on
> improving the core functionality. All the successful systems have thrived
> on defining what it does at its core and keeping it simple and being
> excellent at what it does. In that regard keeping workflow aspect out of
> Airavata can help you to focus on the core problem. That is to execute an
> application on a remote computing resource in a fault tolerant and scalable
> manner.
>
> What I'm suggesting is to give the Orchestrator the capability to execute
> a Driver program that is specified by the user. (This program can be
> written in Python, Java or any other language). This driver program is
> similar to what you define in a Hadoop or Storm configuration. The driver
> program specifies the flow of the computation. It specifies what are the
> applications needs to be executed, how to manipulate input output. The
> driver program is the workflow for the user. Because the user specifies the
> program he can program it to handle workflow steering etc. Every time the
> user wants to execute this program he can tell Airavata to execute the
> Driver program.
>
> I'm also not 100% sure about all the details. But this can be a new way to
> look at how systems like Airavata should behave. Your thoughts and
> suggestions are more than welcome.
>
> Thanks,
> Supun..
>
>
>
>
>
>
> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi Supun,
>>
>> I think we were in two context, because I as suggesting a way to serialize
>> and deserialize the workflow description while you are suggesting to
>> implement
>> some kind of workflow interpreter using Python, where Python client can
>> send
>> thrift calls to Airavata server to run the application. I can see with
>> your
>> suggested
>> approach we can control the workflow execution process from client side
>> which
>> make it easy to implement workflow debugger.As you mentioned this is a
>> major change
>> to Airavata. So we should neatly think as this will change our existing
>> architecture.
>>
>> Still if someone need to use different language java, php, JS etc ... to
>> run the same
>> workflow which generated by Python, we need a language independent
>> workflow
>> description.
>> My initial question was what is the best language for this?. But as I have
>> explained in
>> one of my previous reply, It is not matter what language we used Either we
>> can use
>> XML or JSON to write this description, what matters is how easy to
>> generate
>> workflow with the provided API. Hence it would be great to have set of
>> neat
>> APIs in
>> different languages.
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <su...@gmail.com>
>> wrote:
>>
>> > Hi Shameera,
>> >
>> > Using python is a radical approach for workflow I think. But I believe
>> this
>> > is a very beautiful and new approach.Here is a possible scenario for
>> > implementation and running using a Python based library.
>> >
>> > The Python library facilitates the creation of Applications and
>> submitting
>> > them to Airavata for execution. The underlying functionality is exactly
>> > similar to what Java clients provides.The only difference is that,
>> Python
>> > library should have a more fluent API than Java for easy creation of
>> > workflows. We can generate the Python clients that talk to Airavata
>> server
>> > using Thrift.
>> >
>> > Here is an example off the top of my head to a Python script created by
>> > user for a Workflow. This is a very crude example and we need to come up
>> > with a much better API if we are going to go along this path. First we
>> need
>> > to write a Python script that can execute a workflow using Airavata.
>> >
>> > import airavata
>> >
>> > host = Host("localhost", ....)
>> > app1 = Application(host, ....)
>> > app2 = Application(host, ....)
>> >
>> > # we will connect these applications as a workflow using some topology
>> > builder or other constructs
>> >
>> > wb = WorkFlowBuilder()
>> > wb.setApp("name1", app1)
>> > # you can do a simple output transformation here etc
>> >
>> > # connects the input of app1 to app2 etc
>> > wb.setApp("name2", app2).connectInput("name1")
>> >
>> > wb.submit()
>> >
>> > Now we can load this Python script from XBaya. When XBaya loads this
>> script
>> > the Python script can output an XML configuration of the topology, XBaya
>> > can render. There are other ways like directly executing the Python
>> script
>> > from command line and connecting XBaya indirectly as well. Now you can
>> run
>> > the workflow from XBaya. Running the Workflow means just executing the
>> > Python script.
>> >
>> > XBaya gets the notifications through messaging and update the UI
>> > accordingly.
>> >
>> > The users need to write the Python script by hand. XBaya cannot create
>> the
>> > script. Because workflow language is an actual python program the
>> benefits
>> > are immense. For example user can do workflow steering in the workflow
>> > itself by subscribing to messages from Airavata.
>> >
>> > Thanks,
>> > Supun..
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>> > shameerainfo@gmail.com> wrote:
>> >
>> > > Hi Supun,
>> > >
>> > > I meant to say JS is a well-known client side scripting language i
>> have
>> > > messed scripting part. Even we use Python, ultimately we should
>> convert
>> > > this to java model in server side, somehow we need to serialized
>> python
>> > > representation to the language which java can parse and generate that
>> > > model. In this case we need to parse python script in java isn't it?
>> I am
>> > > not exactly clear how you suggesting to use python here. More details
>> on
>> > > how end system works if we use Python would be great help to clearly
>> > > understand your points.
>> > >
>> > > Thanks,
>> > > Shameera.
>> > >
>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>> > chris.mattmann@gmail.com>
>> > > wrote:
>> > >
>> > > > Have you guys considered using JCC [1] as a means
>> > > > to expose the workflow API currently in Java as a
>> > > > Python API?
>> > > >
>> > > > We are exploring its use in OODT, and we have already
>> > > > created a Tika [2] JCC-based python API.
>> > > >
>> > > > Cheers,
>> > > > Chris
>> > > >
>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>> > > > [2] http://github.com/chrismattmann/tika-python/
>> > > >
>> > > > ------------------------
>> > > > Chris Mattmann
>> > > > chris.mattmann@gmail.com
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > -----Original Message-----
>> > > > From: Supun Kamburugamuva <su...@gmail.com>
>> > > > Reply-To: <de...@oodt.apache.org>
>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>> > > > To: dev <de...@airavata.apache.org>
>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
>> > > > <de...@oodt.apache.org>
>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>> > Airavata.
>> > > >
>> > > > >Once we had an offline discussion about the Airavata Workflow
>> language
>> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
>> came
>> > > out
>> > > > >was why we need to invent a different language when a simple
>> library
>> > > like
>> > > > >Python will full fill of Airavata requirements.
>> > > > >
>> > > > >There are many benefits in using a Python library as the API for
>> > > > >controlling Airavata workflows.
>> > > > >
>> > > > >1. It is a library, gives the ultimate control over the execution
>> and
>> > it
>> > > > >can be simpler than any domain specific language that we can come
>> with
>> > > > >like
>> > > > >XML, JSON etc
>> > > > >2. Most people use python and can learn it easily than any Domain
>> > > specific
>> > > > >language
>> > > > >3. You can easily create a Python library for Airavata because all
>> the
>> > > > >APIs
>> > > > >are thrift based.
>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>> > > > >
>> > > > >Any thoughts?
>> > > > >
>> > > > >Thanks,
>> > > > >Supun..
>> > > > >
>> > > > >
>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
>> > supun06@gmail.com
>> > > >
>> > > > >wrote:
>> > > > >
>> > > > >> Hi Shameera,
>> > > > >>
>> > > > >> Why you prefer JavaScript over a language like Python?
>> > > > >>
>> > > > >> Thanks,
>> > > > >> Supun..
>> > > > >>
>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> > > > >> shameerainfo@gmail.com> wrote:
>> > > > >>
>> > > > >>> Hi,
>> > > > >>>
>> > > > >>> First of all thanks everyone for giving valuable inputs. After
>> > doing
>> > > > >>>some
>> > > > >>> background search and talking to different people in the
>> University
>> > > > >>>who has
>> > > > >>> used different workflow languages, I myself convinced that
>> > > introducing
>> > > > >>>an
>> > > > >>> another workflow language is not what actually they need. By
>> > changing
>> > > > >>> exiting workflow language to another will not solve problems.
>> What
>> > > they
>> > > > >>> asking is a easy way to construct the workflows. Indirectly what
>> > they
>> > > > >>> asking for a sort of API which they can use to generate the
>> > workflows
>> > > > >>>and
>> > > > >>> run it. Correct me if i am wrong here.
>> > > > >>>
>> > > > >>> As most of above replies depict,  if we can get a simple API,
>> as an
>> > > > >>> example, for a web based application, JavaScript API would be a
>> > good
>> > > > >>> solution, and probably JSON would be a good candidate for
>> language,
>> > > > >>>instead
>> > > > >>> of XML.
>> > > > >>>
>> > > > >>> Airavata community already have started to implement web base
>> GUI.
>> > > > >>>Hence
>> > > > >>> introducing a JSON base JavaScript API would be great help.
>> WDYT?
>> > > > >>>
>> > > > >>> Thanks,
>> > > > >>> Shameera.
>> > > > >>>
>> > > > >>>
>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> > > > >>> alekny7@gmail.com> wrote:
>> > > > >>>
>> > > > >>>> Hi,
>> > > > >>>>
>> > > > >>>> it is not dataflow instead focused on orchestrating REST
>> services
>> > > but
>> > > > >>>> you may find it useful datapoint - we created worfklow service
>> > that
>> > > > >>>>uses
>> > > > >>>> natively JavaScript and JSON to describe what happens during
>> > > workflow
>> > > > >>>> execution:
>> > > > >>>>
>> > > >
>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>> > > > >>>>
>> > > > >>>> HTH,
>> > > > >>>>
>> > > > >>>> Alek
>> > > > >>>>
>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
>> smarru@apache.org>
>> > > > >>>>wrote:
>> > > > >>>>
>> > > > >>>>> Hi Chris,
>> > > > >>>>>
>> > > > >>>>> Great to hear OODT community will be interested in adopting a
>> > JSON
>> > > > >>>>> based workflow language and potentially a web based composer
>> as
>> > > well.
>> > > > >>>>> Airavata previously had BPEL support initially through a home
>> > grown
>> > > > >>>>> implementation [1] by Alek Slominski and later through Apache
>> ODE
>> > > > >>>>>[2]. Also
>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
>> read.
>> > > > >>>>>
>> > > > >>>>> I am of the same opinion that we should adopt something more
>> > modern
>> > > > >>>>>as
>> > > > >>>>> the challenges from scientific workflows seems to be
>> converging
>> > > with
>> > > > >>>>>the
>> > > > >>>>> data flow patterns in business workflows.
>> > > > >>>>>
>> > > > >>>>> It will be great if we can all compile a list of potential
>> > > candidates
>> > > > >>>>> and hack them through.
>> > > > >>>>>
>> > > > >>>>> Suresh
>> > > > >>>>> [1] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>> > > > >>>>> [2] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>> > > > >>>>>time_for_eScience_workflows
>> > > > >>>>> [3] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>> > > > >>>>>6.pdf
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>> > > > >>>>>
>> > > > >>>>> > Hi Guys,
>> > > > >>>>> >
>> > > > >>>>> > I've been interested in this too - we don't per have a
>> specific
>> > > > >>>>> > OODT workflow language, but we specific workflows using XML,
>> > and
>> > > > >>>>> > other configuration (we are also thinking of moving to JSON
>> for
>> > > > >>>>> > this).
>> > > > >>>>> >
>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
>> > complex
>> > > > >>>>> > to me.
>> > > > >>>>> >
>> > > > >>>>> > I wonder at the end of the day if we should adopt something
>> > more
>> > > > >>>>> > modern like PIG or some other data flow type of language
>> (PIG
>> > > > >>>>> > is really neat).
>> > > > >>>>> >
>> > > > >>>>> > Cheers,
>> > > > >>>>> > Chris
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> > Chris Mattmann, Ph.D.
>> > > > >>>>> > Chief Architect
>> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
>> > > > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> > -----Original Message-----
>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
>> > > > >>>>> > <ar...@airavata.apache.org>
>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
>> > > > >>>>> > To: "architecture@airavata.apache.org" <
>> > > > >>>>> architecture@airavata.apache.org>,
>> > > > >>>>> > dev <de...@airavata.apache.org>
>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
>> > > > >>>>>Airavata.
>> > > > >>>>> >
>> > > > >>>>> >> Hi All,
>> > > > >>>>> >>
>> > > > >>>>> >> As we all know Airavata has its own workflow language call
>> > XWF.
>> > > > >>>>>When
>> > > > >>>>> XWF
>> > > > >>>>> >> was introduced, main focus points are interoperability and
>> > > > >>>>> convertibility.
>> > > > >>>>> >> But with years of experience it is convinced that above
>> > > > >>>>>requirements
>> > > > >>>>> are
>> > > > >>>>> >> not really useful when we come to real world use cases. And
>> > XWF
>> > > is
>> > > > >>>>> XML
>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
>> image
>> > > it
>> > > > >>>>> self.
>> > > > >>>>> >> But
>> > > > >>>>> >> with the recent changes WSDL part is being removed from
>> XWF.
>> > > > >>>>> >>
>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
>> languages in
>> > > > >>>>> industry
>> > > > >>>>> >> and
>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
>> need
>> > to
>> > > > >>>>> come up
>> > > > >>>>> >> with idea how we should improve Airavata workflow language,
>> > > either
>> > > > >>>>> we can
>> > > > >>>>> >> improve existing XWF language, totally change to a new
>> > language
>> > > > >>>>> available
>> > > > >>>>> >> in industry or write a new light weight language. Basic
>> > > > >>>>>requirements
>> > > > >>>>> that
>> > > > >>>>> >> we expect from new improvement are, high usability,
>> flexible,
>> > > > >>>>>light
>> > > > >>>>> weight
>> > > > >>>>> >> and real time monitoring support. As you can see above
>> > > > >>>>>requirements
>> > > > >>>>> are
>> > > > >>>>> >> not
>> > > > >>>>> >> direct comes with workflow languages but we need workflow
>> > > language
>> > > > >>>>> which
>> > > > >>>>> >> help to support above requirements.
>> > > > >>>>> >>
>> > > > >>>>> >> After reading few papers and googling, initially i have
>> come
>> > up
>> > > > >>>>>with
>> > > > >>>>> >> following three existing languages,
>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>> > > > >>>>> >> 2. WS-BPEL
>> > > > >>>>> >> 3. SIDL
>> > > > >>>>> >> <
>> > > http://computation.llnl.gov/casc/components/index.html#page=home
>> > > > >
>> > > > >>>>> >>
>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
>> > > > >>>>> Radical-SAGA
>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
>> above
>> > > > >>>>>three
>> > > > >>>>> >> languages we can come up with simple workflow language
>> base on
>> > > > >>>>> json(or
>> > > > >>>>> >> yaml) which support all our requirements for some extends.
>> > > > >>>>> >>
>> > > > >>>>> >> It would be grate if I can get more input regarding the
>> > $Subject
>> > > > >>>>> form the
>> > > > >>>>> >> airavata community. You all are more than welcome to
>> provide
>> > any
>> > > > >>>>> type of
>> > > > >>>>> >> suggestions.
>> > > > >>>>> >>
>> > > > >>>>> >> Thanks,
>> > > > >>>>> >> Shameera.
>> > > > >>>>> >>
>> > > > >>>>> >> 
>> > > > >>>>> >>
>> > > > >>>>> >> --
>> > > > >>>>> >> Best Regards,
>> > > > >>>>> >> Shameera Rathnayaka.
>> > > > >>>>> >>
>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> --
>> > > > >>>> The best way to predict the future is to invent it - Alan Kay
>> > > > >>>>
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> --
>> > > > >>> Best Regards,
>> > > > >>> Shameera Rathnayaka.
>> > > > >>>
>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
>> > > > >>>
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >> --
>> > > > >> Supun Kamburugamuva
>> > > > >> Member, Apache Software Foundation; http://www.apache.org
>> > > > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> > > > >> Blog: http://supunk.blogspot.com
>> > > > >>
>> > > > >>
>> > > > >
>> > > > >
>> > > > >--
>> > > > >Supun Kamburugamuva
>> > > > >Member, Apache Software Foundation; http://www.apache.org
>> > > > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> > > > >Blog: http://supunk.blogspot.com
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Best Regards,
>> > > Shameera Rathnayaka.
>> > >
>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > Blog : http://shameerarathnayaka.blogspot.com/
>> > >
>> >
>> >
>> >
>> > --
>> > Supun Kamburugamuva
>> > Member, Apache Software Foundation; http://www.apache.org
>> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> > Blog: http://supunk.blogspot.com
>> >
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Amila Jayasekara <th...@gmail.com>.

Hello,

I am sorry, I am bit late on this thread. But when reading through this
thread I simply got lost, what this thread is discussing. I have few
questions.

1. @Shameera : Is XWF actually a language to define workflow ? To my
understanding it was an intermediate representation to convert workflow
defined in UI to java object model. Was XWF ever used by any airavata user
to define a workflow graph ?

>From initial description what I understood is we are looking for a improved
"intermediate representation", not a language which describes workflows.

2. So what is the exact purpose of this proposed language ?
        - Is it to hide parallelism in workflows ?
        - Is it to replace the XBAYA functionalities ? (i hope not)

3. What are we trying to achieve by this proposed language which we cannot
achieved through workflow UI tool ?

4. Who is going to use this language ?

5. Why would a user prefer (assuming intended audience of proposed language
is end users) a language over a Work Flow UI tool such as XBAYA ? (In other
words what are the things we can do with language which we cannot do with
UI)

Sorry, if above questions are in-appropriate, just wanted to understand
what exactly needed.

Thanks
-Thejaka Amila



On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:

> I think I'm not suggesting to create a Workflow interpreter using Python
> etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
> and move it to a more higher level component. The more I think about it,
> the model I'm suggesting is similar to what Hadoop, Storm etc has done for
> distributed system computations. This model is proven to be successful over
> the years.
>
> Keeping what Airavata does at its core can help you to build a more robust
> system. If we look at Airavata as middleware to execute applications on
> computing resources we can simplify what Airavata does and focus on
> improving the core functionality. All the successful systems have thrived
> on defining what it does at its core and keeping it simple and being
> excellent at what it does. In that regard keeping workflow aspect out of
> Airavata can help you to focus on the core problem. That is to execute an
> application on a remote computing resource in a fault tolerant and scalable
> manner.
>
> What I'm suggesting is to give the Orchestrator the capability to execute
> a Driver program that is specified by the user. (This program can be
> written in Python, Java or any other language). This driver program is
> similar to what you define in a Hadoop or Storm configuration. The driver
> program specifies the flow of the computation. It specifies what are the
> applications needs to be executed, how to manipulate input output. The
> driver program is the workflow for the user. Because the user specifies the
> program he can program it to handle workflow steering etc. Every time the
> user wants to execute this program he can tell Airavata to execute the
> Driver program.
>
> I'm also not 100% sure about all the details. But this can be a new way to
> look at how systems like Airavata should behave. Your thoughts and
> suggestions are more than welcome.
>
> Thanks,
> Supun..
>
>
>
>
>
>
> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi Supun,
>>
>> I think we were in two context, because I as suggesting a way to serialize
>> and deserialize the workflow description while you are suggesting to
>> implement
>> some kind of workflow interpreter using Python, where Python client can
>> send
>> thrift calls to Airavata server to run the application. I can see with
>> your
>> suggested
>> approach we can control the workflow execution process from client side
>> which
>> make it easy to implement workflow debugger.As you mentioned this is a
>> major change
>> to Airavata. So we should neatly think as this will change our existing
>> architecture.
>>
>> Still if someone need to use different language java, php, JS etc ... to
>> run the same
>> workflow which generated by Python, we need a language independent
>> workflow
>> description.
>> My initial question was what is the best language for this?. But as I have
>> explained in
>> one of my previous reply, It is not matter what language we used Either we
>> can use
>> XML or JSON to write this description, what matters is how easy to
>> generate
>> workflow with the provided API. Hence it would be great to have set of
>> neat
>> APIs in
>> different languages.
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <su...@gmail.com>
>> wrote:
>>
>> > Hi Shameera,
>> >
>> > Using python is a radical approach for workflow I think. But I believe
>> this
>> > is a very beautiful and new approach.Here is a possible scenario for
>> > implementation and running using a Python based library.
>> >
>> > The Python library facilitates the creation of Applications and
>> submitting
>> > them to Airavata for execution. The underlying functionality is exactly
>> > similar to what Java clients provides.The only difference is that,
>> Python
>> > library should have a more fluent API than Java for easy creation of
>> > workflows. We can generate the Python clients that talk to Airavata
>> server
>> > using Thrift.
>> >
>> > Here is an example off the top of my head to a Python script created by
>> > user for a Workflow. This is a very crude example and we need to come up
>> > with a much better API if we are going to go along this path. First we
>> need
>> > to write a Python script that can execute a workflow using Airavata.
>> >
>> > import airavata
>> >
>> > host = Host("localhost", ....)
>> > app1 = Application(host, ....)
>> > app2 = Application(host, ....)
>> >
>> > # we will connect these applications as a workflow using some topology
>> > builder or other constructs
>> >
>> > wb = WorkFlowBuilder()
>> > wb.setApp("name1", app1)
>> > # you can do a simple output transformation here etc
>> >
>> > # connects the input of app1 to app2 etc
>> > wb.setApp("name2", app2).connectInput("name1")
>> >
>> > wb.submit()
>> >
>> > Now we can load this Python script from XBaya. When XBaya loads this
>> script
>> > the Python script can output an XML configuration of the topology, XBaya
>> > can render. There are other ways like directly executing the Python
>> script
>> > from command line and connecting XBaya indirectly as well. Now you can
>> run
>> > the workflow from XBaya. Running the Workflow means just executing the
>> > Python script.
>> >
>> > XBaya gets the notifications through messaging and update the UI
>> > accordingly.
>> >
>> > The users need to write the Python script by hand. XBaya cannot create
>> the
>> > script. Because workflow language is an actual python program the
>> benefits
>> > are immense. For example user can do workflow steering in the workflow
>> > itself by subscribing to messages from Airavata.
>> >
>> > Thanks,
>> > Supun..
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>> > shameerainfo@gmail.com> wrote:
>> >
>> > > Hi Supun,
>> > >
>> > > I meant to say JS is a well-known client side scripting language i
>> have
>> > > messed scripting part. Even we use Python, ultimately we should
>> convert
>> > > this to java model in server side, somehow we need to serialized
>> python
>> > > representation to the language which java can parse and generate that
>> > > model. In this case we need to parse python script in java isn't it?
>> I am
>> > > not exactly clear how you suggesting to use python here. More details
>> on
>> > > how end system works if we use Python would be great help to clearly
>> > > understand your points.
>> > >
>> > > Thanks,
>> > > Shameera.
>> > >
>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>> > chris.mattmann@gmail.com>
>> > > wrote:
>> > >
>> > > > Have you guys considered using JCC [1] as a means
>> > > > to expose the workflow API currently in Java as a
>> > > > Python API?
>> > > >
>> > > > We are exploring its use in OODT, and we have already
>> > > > created a Tika [2] JCC-based python API.
>> > > >
>> > > > Cheers,
>> > > > Chris
>> > > >
>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>> > > > [2] http://github.com/chrismattmann/tika-python/
>> > > >
>> > > > ------------------------
>> > > > Chris Mattmann
>> > > > chris.mattmann@gmail.com
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > -----Original Message-----
>> > > > From: Supun Kamburugamuva <su...@gmail.com>
>> > > > Reply-To: <de...@oodt.apache.org>
>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>> > > > To: dev <de...@airavata.apache.org>
>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
>> > > > <de...@oodt.apache.org>
>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>> > Airavata.
>> > > >
>> > > > >Once we had an offline discussion about the Airavata Workflow
>> language
>> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
>> came
>> > > out
>> > > > >was why we need to invent a different language when a simple
>> library
>> > > like
>> > > > >Python will full fill of Airavata requirements.
>> > > > >
>> > > > >There are many benefits in using a Python library as the API for
>> > > > >controlling Airavata workflows.
>> > > > >
>> > > > >1. It is a library, gives the ultimate control over the execution
>> and
>> > it
>> > > > >can be simpler than any domain specific language that we can come
>> with
>> > > > >like
>> > > > >XML, JSON etc
>> > > > >2. Most people use python and can learn it easily than any Domain
>> > > specific
>> > > > >language
>> > > > >3. You can easily create a Python library for Airavata because all
>> the
>> > > > >APIs
>> > > > >are thrift based.
>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>> > > > >
>> > > > >Any thoughts?
>> > > > >
>> > > > >Thanks,
>> > > > >Supun..
>> > > > >
>> > > > >
>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
>> > supun06@gmail.com
>> > > >
>> > > > >wrote:
>> > > > >
>> > > > >> Hi Shameera,
>> > > > >>
>> > > > >> Why you prefer JavaScript over a language like Python?
>> > > > >>
>> > > > >> Thanks,
>> > > > >> Supun..
>> > > > >>
>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> > > > >> shameerainfo@gmail.com> wrote:
>> > > > >>
>> > > > >>> Hi,
>> > > > >>>
>> > > > >>> First of all thanks everyone for giving valuable inputs. After
>> > doing
>> > > > >>>some
>> > > > >>> background search and talking to different people in the
>> University
>> > > > >>>who has
>> > > > >>> used different workflow languages, I myself convinced that
>> > > introducing
>> > > > >>>an
>> > > > >>> another workflow language is not what actually they need. By
>> > changing
>> > > > >>> exiting workflow language to another will not solve problems.
>> What
>> > > they
>> > > > >>> asking is a easy way to construct the workflows. Indirectly what
>> > they
>> > > > >>> asking for a sort of API which they can use to generate the
>> > workflows
>> > > > >>>and
>> > > > >>> run it. Correct me if i am wrong here.
>> > > > >>>
>> > > > >>> As most of above replies depict,  if we can get a simple API,
>> as an
>> > > > >>> example, for a web based application, JavaScript API would be a
>> > good
>> > > > >>> solution, and probably JSON would be a good candidate for
>> language,
>> > > > >>>instead
>> > > > >>> of XML.
>> > > > >>>
>> > > > >>> Airavata community already have started to implement web base
>> GUI.
>> > > > >>>Hence
>> > > > >>> introducing a JSON base JavaScript API would be great help.
>> WDYT?
>> > > > >>>
>> > > > >>> Thanks,
>> > > > >>> Shameera.
>> > > > >>>
>> > > > >>>
>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> > > > >>> alekny7@gmail.com> wrote:
>> > > > >>>
>> > > > >>>> Hi,
>> > > > >>>>
>> > > > >>>> it is not dataflow instead focused on orchestrating REST
>> services
>> > > but
>> > > > >>>> you may find it useful datapoint - we created worfklow service
>> > that
>> > > > >>>>uses
>> > > > >>>> natively JavaScript and JSON to describe what happens during
>> > > workflow
>> > > > >>>> execution:
>> > > > >>>>
>> > > >
>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>> > > > >>>>
>> > > > >>>> HTH,
>> > > > >>>>
>> > > > >>>> Alek
>> > > > >>>>
>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
>> smarru@apache.org>
>> > > > >>>>wrote:
>> > > > >>>>
>> > > > >>>>> Hi Chris,
>> > > > >>>>>
>> > > > >>>>> Great to hear OODT community will be interested in adopting a
>> > JSON
>> > > > >>>>> based workflow language and potentially a web based composer
>> as
>> > > well.
>> > > > >>>>> Airavata previously had BPEL support initially through a home
>> > grown
>> > > > >>>>> implementation [1] by Alek Slominski and later through Apache
>> ODE
>> > > > >>>>>[2]. Also
>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
>> read.
>> > > > >>>>>
>> > > > >>>>> I am of the same opinion that we should adopt something more
>> > modern
>> > > > >>>>>as
>> > > > >>>>> the challenges from scientific workflows seems to be
>> converging
>> > > with
>> > > > >>>>>the
>> > > > >>>>> data flow patterns in business workflows.
>> > > > >>>>>
>> > > > >>>>> It will be great if we can all compile a list of potential
>> > > candidates
>> > > > >>>>> and hack them through.
>> > > > >>>>>
>> > > > >>>>> Suresh
>> > > > >>>>> [1] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>> > > > >>>>> [2] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>> > > > >>>>>time_for_eScience_workflows
>> > > > >>>>> [3] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>> > > > >>>>>6.pdf
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>> > > > >>>>>
>> > > > >>>>> > Hi Guys,
>> > > > >>>>> >
>> > > > >>>>> > I've been interested in this too - we don't per have a
>> specific
>> > > > >>>>> > OODT workflow language, but we specific workflows using XML,
>> > and
>> > > > >>>>> > other configuration (we are also thinking of moving to JSON
>> for
>> > > > >>>>> > this).
>> > > > >>>>> >
>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
>> > complex
>> > > > >>>>> > to me.
>> > > > >>>>> >
>> > > > >>>>> > I wonder at the end of the day if we should adopt something
>> > more
>> > > > >>>>> > modern like PIG or some other data flow type of language
>> (PIG
>> > > > >>>>> > is really neat).
>> > > > >>>>> >
>> > > > >>>>> > Cheers,
>> > > > >>>>> > Chris
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> > Chris Mattmann, Ph.D.
>> > > > >>>>> > Chief Architect
>> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
>> > > > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> > -----Original Message-----
>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
>> > > > >>>>> > <ar...@airavata.apache.org>
>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
>> > > > >>>>> > To: "architecture@airavata.apache.org" <
>> > > > >>>>> architecture@airavata.apache.org>,
>> > > > >>>>> > dev <de...@airavata.apache.org>
>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
>> > > > >>>>>Airavata.
>> > > > >>>>> >
>> > > > >>>>> >> Hi All,
>> > > > >>>>> >>
>> > > > >>>>> >> As we all know Airavata has its own workflow language call
>> > XWF.
>> > > > >>>>>When
>> > > > >>>>> XWF
>> > > > >>>>> >> was introduced, main focus points are interoperability and
>> > > > >>>>> convertibility.
>> > > > >>>>> >> But with years of experience it is convinced that above
>> > > > >>>>>requirements
>> > > > >>>>> are
>> > > > >>>>> >> not really useful when we come to real world use cases. And
>> > XWF
>> > > is
>> > > > >>>>> XML
>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
>> image
>> > > it
>> > > > >>>>> self.
>> > > > >>>>> >> But
>> > > > >>>>> >> with the recent changes WSDL part is being removed from
>> XWF.
>> > > > >>>>> >>
>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
>> languages in
>> > > > >>>>> industry
>> > > > >>>>> >> and
>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
>> need
>> > to
>> > > > >>>>> come up
>> > > > >>>>> >> with idea how we should improve Airavata workflow language,
>> > > either
>> > > > >>>>> we can
>> > > > >>>>> >> improve existing XWF language, totally change to a new
>> > language
>> > > > >>>>> available
>> > > > >>>>> >> in industry or write a new light weight language. Basic
>> > > > >>>>>requirements
>> > > > >>>>> that
>> > > > >>>>> >> we expect from new improvement are, high usability,
>> flexible,
>> > > > >>>>>light
>> > > > >>>>> weight
>> > > > >>>>> >> and real time monitoring support. As you can see above
>> > > > >>>>>requirements
>> > > > >>>>> are
>> > > > >>>>> >> not
>> > > > >>>>> >> direct comes with workflow languages but we need workflow
>> > > language
>> > > > >>>>> which
>> > > > >>>>> >> help to support above requirements.
>> > > > >>>>> >>
>> > > > >>>>> >> After reading few papers and googling, initially i have
>> come
>> > up
>> > > > >>>>>with
>> > > > >>>>> >> following three existing languages,
>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>> > > > >>>>> >> 2. WS-BPEL
>> > > > >>>>> >> 3. SIDL
>> > > > >>>>> >> <
>> > > http://computation.llnl.gov/casc/components/index.html#page=home
>> > > > >
>> > > > >>>>> >>
>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
>> > > > >>>>> Radical-SAGA
>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
>> above
>> > > > >>>>>three
>> > > > >>>>> >> languages we can come up with simple workflow language
>> base on
>> > > > >>>>> json(or
>> > > > >>>>> >> yaml) which support all our requirements for some extends.
>> > > > >>>>> >>
>> > > > >>>>> >> It would be grate if I can get more input regarding the
>> > $Subject
>> > > > >>>>> form the
>> > > > >>>>> >> airavata community. You all are more than welcome to
>> provide
>> > any
>> > > > >>>>> type of
>> > > > >>>>> >> suggestions.
>> > > > >>>>> >>
>> > > > >>>>> >> Thanks,
>> > > > >>>>> >> Shameera.
>> > > > >>>>> >>
>> > > > >>>>> >> 
>> > > > >>>>> >>
>> > > > >>>>> >> --
>> > > > >>>>> >> Best Regards,
>> > > > >>>>> >> Shameera Rathnayaka.
>> > > > >>>>> >>
>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> --
>> > > > >>>> The best way to predict the future is to invent it - Alan Kay
>> > > > >>>>
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> --
>> > > > >>> Best Regards,
>> > > > >>> Shameera Rathnayaka.
>> > > > >>>
>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
>> > > > >>>
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >> --
>> > > > >> Supun Kamburugamuva
>> > > > >> Member, Apache Software Foundation; http://www.apache.org
>> > > > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> > > > >> Blog: http://supunk.blogspot.com
>> > > > >>
>> > > > >>
>> > > > >
>> > > > >
>> > > > >--
>> > > > >Supun Kamburugamuva
>> > > > >Member, Apache Software Foundation; http://www.apache.org
>> > > > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> > > > >Blog: http://supunk.blogspot.com
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Best Regards,
>> > > Shameera Rathnayaka.
>> > >
>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > Blog : http://shameerarathnayaka.blogspot.com/
>> > >
>> >
>> >
>> >
>> > --
>> > Supun Kamburugamuva
>> > Member, Apache Software Foundation; http://www.apache.org
>> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> > Blog: http://supunk.blogspot.com
>> >
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Amila Jayasekara <th...@gmail.com>.

Hello,

I am sorry, I am bit late on this thread. But when reading through this
thread I simply got lost, what this thread is discussing. I have few
questions.

1. @Shameera : Is XWF actually a language to define workflow ? To my
understanding it was an intermediate representation to convert workflow
defined in UI to java object model. Was XWF ever used by any airavata user
to define a workflow graph ?

>From initial description what I understood is we are looking for a improved
"intermediate representation", not a language which describes workflows.

2. So what is the exact purpose of this proposed language ?
        - Is it to hide parallelism in workflows ?
        - Is it to replace the XBAYA functionalities ? (i hope not)

3. What are we trying to achieve by this proposed language which we cannot
achieved through workflow UI tool ?

4. Who is going to use this language ?

5. Why would a user prefer (assuming intended audience of proposed language
is end users) a language over a Work Flow UI tool such as XBAYA ? (In other
words what are the things we can do with language which we cannot do with
UI)

Sorry, if above questions are in-appropriate, just wanted to understand
what exactly needed.

Thanks
-Thejaka Amila



On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:

> I think I'm not suggesting to create a Workflow interpreter using Python
> etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
> and move it to a more higher level component. The more I think about it,
> the model I'm suggesting is similar to what Hadoop, Storm etc has done for
> distributed system computations. This model is proven to be successful over
> the years.
>
> Keeping what Airavata does at its core can help you to build a more robust
> system. If we look at Airavata as middleware to execute applications on
> computing resources we can simplify what Airavata does and focus on
> improving the core functionality. All the successful systems have thrived
> on defining what it does at its core and keeping it simple and being
> excellent at what it does. In that regard keeping workflow aspect out of
> Airavata can help you to focus on the core problem. That is to execute an
> application on a remote computing resource in a fault tolerant and scalable
> manner.
>
> What I'm suggesting is to give the Orchestrator the capability to execute
> a Driver program that is specified by the user. (This program can be
> written in Python, Java or any other language). This driver program is
> similar to what you define in a Hadoop or Storm configuration. The driver
> program specifies the flow of the computation. It specifies what are the
> applications needs to be executed, how to manipulate input output. The
> driver program is the workflow for the user. Because the user specifies the
> program he can program it to handle workflow steering etc. Every time the
> user wants to execute this program he can tell Airavata to execute the
> Driver program.
>
> I'm also not 100% sure about all the details. But this can be a new way to
> look at how systems like Airavata should behave. Your thoughts and
> suggestions are more than welcome.
>
> Thanks,
> Supun..
>
>
>
>
>
>
> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi Supun,
>>
>> I think we were in two context, because I as suggesting a way to serialize
>> and deserialize the workflow description while you are suggesting to
>> implement
>> some kind of workflow interpreter using Python, where Python client can
>> send
>> thrift calls to Airavata server to run the application. I can see with
>> your
>> suggested
>> approach we can control the workflow execution process from client side
>> which
>> make it easy to implement workflow debugger.As you mentioned this is a
>> major change
>> to Airavata. So we should neatly think as this will change our existing
>> architecture.
>>
>> Still if someone need to use different language java, php, JS etc ... to
>> run the same
>> workflow which generated by Python, we need a language independent
>> workflow
>> description.
>> My initial question was what is the best language for this?. But as I have
>> explained in
>> one of my previous reply, It is not matter what language we used Either we
>> can use
>> XML or JSON to write this description, what matters is how easy to
>> generate
>> workflow with the provided API. Hence it would be great to have set of
>> neat
>> APIs in
>> different languages.
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <su...@gmail.com>
>> wrote:
>>
>> > Hi Shameera,
>> >
>> > Using python is a radical approach for workflow I think. But I believe
>> this
>> > is a very beautiful and new approach.Here is a possible scenario for
>> > implementation and running using a Python based library.
>> >
>> > The Python library facilitates the creation of Applications and
>> submitting
>> > them to Airavata for execution. The underlying functionality is exactly
>> > similar to what Java clients provides.The only difference is that,
>> Python
>> > library should have a more fluent API than Java for easy creation of
>> > workflows. We can generate the Python clients that talk to Airavata
>> server
>> > using Thrift.
>> >
>> > Here is an example off the top of my head to a Python script created by
>> > user for a Workflow. This is a very crude example and we need to come up
>> > with a much better API if we are going to go along this path. First we
>> need
>> > to write a Python script that can execute a workflow using Airavata.
>> >
>> > import airavata
>> >
>> > host = Host("localhost", ....)
>> > app1 = Application(host, ....)
>> > app2 = Application(host, ....)
>> >
>> > # we will connect these applications as a workflow using some topology
>> > builder or other constructs
>> >
>> > wb = WorkFlowBuilder()
>> > wb.setApp("name1", app1)
>> > # you can do a simple output transformation here etc
>> >
>> > # connects the input of app1 to app2 etc
>> > wb.setApp("name2", app2).connectInput("name1")
>> >
>> > wb.submit()
>> >
>> > Now we can load this Python script from XBaya. When XBaya loads this
>> script
>> > the Python script can output an XML configuration of the topology, XBaya
>> > can render. There are other ways like directly executing the Python
>> script
>> > from command line and connecting XBaya indirectly as well. Now you can
>> run
>> > the workflow from XBaya. Running the Workflow means just executing the
>> > Python script.
>> >
>> > XBaya gets the notifications through messaging and update the UI
>> > accordingly.
>> >
>> > The users need to write the Python script by hand. XBaya cannot create
>> the
>> > script. Because workflow language is an actual python program the
>> benefits
>> > are immense. For example user can do workflow steering in the workflow
>> > itself by subscribing to messages from Airavata.
>> >
>> > Thanks,
>> > Supun..
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>> > shameerainfo@gmail.com> wrote:
>> >
>> > > Hi Supun,
>> > >
>> > > I meant to say JS is a well-known client side scripting language i
>> have
>> > > messed scripting part. Even we use Python, ultimately we should
>> convert
>> > > this to java model in server side, somehow we need to serialized
>> python
>> > > representation to the language which java can parse and generate that
>> > > model. In this case we need to parse python script in java isn't it?
>> I am
>> > > not exactly clear how you suggesting to use python here. More details
>> on
>> > > how end system works if we use Python would be great help to clearly
>> > > understand your points.
>> > >
>> > > Thanks,
>> > > Shameera.
>> > >
>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>> > chris.mattmann@gmail.com>
>> > > wrote:
>> > >
>> > > > Have you guys considered using JCC [1] as a means
>> > > > to expose the workflow API currently in Java as a
>> > > > Python API?
>> > > >
>> > > > We are exploring its use in OODT, and we have already
>> > > > created a Tika [2] JCC-based python API.
>> > > >
>> > > > Cheers,
>> > > > Chris
>> > > >
>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>> > > > [2] http://github.com/chrismattmann/tika-python/
>> > > >
>> > > > ------------------------
>> > > > Chris Mattmann
>> > > > chris.mattmann@gmail.com
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > -----Original Message-----
>> > > > From: Supun Kamburugamuva <su...@gmail.com>
>> > > > Reply-To: <de...@oodt.apache.org>
>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>> > > > To: dev <de...@airavata.apache.org>
>> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
>> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
>> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
>> > > > <de...@oodt.apache.org>
>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>> > Airavata.
>> > > >
>> > > > >Once we had an offline discussion about the Airavata Workflow
>> language
>> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
>> came
>> > > out
>> > > > >was why we need to invent a different language when a simple
>> library
>> > > like
>> > > > >Python will full fill of Airavata requirements.
>> > > > >
>> > > > >There are many benefits in using a Python library as the API for
>> > > > >controlling Airavata workflows.
>> > > > >
>> > > > >1. It is a library, gives the ultimate control over the execution
>> and
>> > it
>> > > > >can be simpler than any domain specific language that we can come
>> with
>> > > > >like
>> > > > >XML, JSON etc
>> > > > >2. Most people use python and can learn it easily than any Domain
>> > > specific
>> > > > >language
>> > > > >3. You can easily create a Python library for Airavata because all
>> the
>> > > > >APIs
>> > > > >are thrift based.
>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>> > > > >
>> > > > >Any thoughts?
>> > > > >
>> > > > >Thanks,
>> > > > >Supun..
>> > > > >
>> > > > >
>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
>> > supun06@gmail.com
>> > > >
>> > > > >wrote:
>> > > > >
>> > > > >> Hi Shameera,
>> > > > >>
>> > > > >> Why you prefer JavaScript over a language like Python?
>> > > > >>
>> > > > >> Thanks,
>> > > > >> Supun..
>> > > > >>
>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> > > > >> shameerainfo@gmail.com> wrote:
>> > > > >>
>> > > > >>> Hi,
>> > > > >>>
>> > > > >>> First of all thanks everyone for giving valuable inputs. After
>> > doing
>> > > > >>>some
>> > > > >>> background search and talking to different people in the
>> University
>> > > > >>>who has
>> > > > >>> used different workflow languages, I myself convinced that
>> > > introducing
>> > > > >>>an
>> > > > >>> another workflow language is not what actually they need. By
>> > changing
>> > > > >>> exiting workflow language to another will not solve problems.
>> What
>> > > they
>> > > > >>> asking is a easy way to construct the workflows. Indirectly what
>> > they
>> > > > >>> asking for a sort of API which they can use to generate the
>> > workflows
>> > > > >>>and
>> > > > >>> run it. Correct me if i am wrong here.
>> > > > >>>
>> > > > >>> As most of above replies depict,  if we can get a simple API,
>> as an
>> > > > >>> example, for a web based application, JavaScript API would be a
>> > good
>> > > > >>> solution, and probably JSON would be a good candidate for
>> language,
>> > > > >>>instead
>> > > > >>> of XML.
>> > > > >>>
>> > > > >>> Airavata community already have started to implement web base
>> GUI.
>> > > > >>>Hence
>> > > > >>> introducing a JSON base JavaScript API would be great help.
>> WDYT?
>> > > > >>>
>> > > > >>> Thanks,
>> > > > >>> Shameera.
>> > > > >>>
>> > > > >>>
>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> > > > >>> alekny7@gmail.com> wrote:
>> > > > >>>
>> > > > >>>> Hi,
>> > > > >>>>
>> > > > >>>> it is not dataflow instead focused on orchestrating REST
>> services
>> > > but
>> > > > >>>> you may find it useful datapoint - we created worfklow service
>> > that
>> > > > >>>>uses
>> > > > >>>> natively JavaScript and JSON to describe what happens during
>> > > workflow
>> > > > >>>> execution:
>> > > > >>>>
>> > > >
>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>> > > > >>>>
>> > > > >>>> HTH,
>> > > > >>>>
>> > > > >>>> Alek
>> > > > >>>>
>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
>> smarru@apache.org>
>> > > > >>>>wrote:
>> > > > >>>>
>> > > > >>>>> Hi Chris,
>> > > > >>>>>
>> > > > >>>>> Great to hear OODT community will be interested in adopting a
>> > JSON
>> > > > >>>>> based workflow language and potentially a web based composer
>> as
>> > > well.
>> > > > >>>>> Airavata previously had BPEL support initially through a home
>> > grown
>> > > > >>>>> implementation [1] by Alek Slominski and later through Apache
>> ODE
>> > > > >>>>>[2]. Also
>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
>> read.
>> > > > >>>>>
>> > > > >>>>> I am of the same opinion that we should adopt something more
>> > modern
>> > > > >>>>>as
>> > > > >>>>> the challenges from scientific workflows seems to be
>> converging
>> > > with
>> > > > >>>>>the
>> > > > >>>>> data flow patterns in business workflows.
>> > > > >>>>>
>> > > > >>>>> It will be great if we can all compile a list of potential
>> > > candidates
>> > > > >>>>> and hack them through.
>> > > > >>>>>
>> > > > >>>>> Suresh
>> > > > >>>>> [1] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>> > > > >>>>> [2] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>> > > > >>>>>time_for_eScience_workflows
>> > > > >>>>> [3] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>> > > > >>>>>6.pdf
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>> > > > >>>>>
>> > > > >>>>> > Hi Guys,
>> > > > >>>>> >
>> > > > >>>>> > I've been interested in this too - we don't per have a
>> specific
>> > > > >>>>> > OODT workflow language, but we specific workflows using XML,
>> > and
>> > > > >>>>> > other configuration (we are also thinking of moving to JSON
>> for
>> > > > >>>>> > this).
>> > > > >>>>> >
>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
>> > complex
>> > > > >>>>> > to me.
>> > > > >>>>> >
>> > > > >>>>> > I wonder at the end of the day if we should adopt something
>> > more
>> > > > >>>>> > modern like PIG or some other data flow type of language
>> (PIG
>> > > > >>>>> > is really neat).
>> > > > >>>>> >
>> > > > >>>>> > Cheers,
>> > > > >>>>> > Chris
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> > Chris Mattmann, Ph.D.
>> > > > >>>>> > Chief Architect
>> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
>> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
>> > > > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> > -----Original Message-----
>> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
>> > > > >>>>> > <ar...@airavata.apache.org>
>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
>> > > > >>>>> > To: "architecture@airavata.apache.org" <
>> > > > >>>>> architecture@airavata.apache.org>,
>> > > > >>>>> > dev <de...@airavata.apache.org>
>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
>> > > > >>>>>Airavata.
>> > > > >>>>> >
>> > > > >>>>> >> Hi All,
>> > > > >>>>> >>
>> > > > >>>>> >> As we all know Airavata has its own workflow language call
>> > XWF.
>> > > > >>>>>When
>> > > > >>>>> XWF
>> > > > >>>>> >> was introduced, main focus points are interoperability and
>> > > > >>>>> convertibility.
>> > > > >>>>> >> But with years of experience it is convinced that above
>> > > > >>>>>requirements
>> > > > >>>>> are
>> > > > >>>>> >> not really useful when we come to real world use cases. And
>> > XWF
>> > > is
>> > > > >>>>> XML
>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
>> image
>> > > it
>> > > > >>>>> self.
>> > > > >>>>> >> But
>> > > > >>>>> >> with the recent changes WSDL part is being removed from
>> XWF.
>> > > > >>>>> >>
>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
>> languages in
>> > > > >>>>> industry
>> > > > >>>>> >> and
>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
>> need
>> > to
>> > > > >>>>> come up
>> > > > >>>>> >> with idea how we should improve Airavata workflow language,
>> > > either
>> > > > >>>>> we can
>> > > > >>>>> >> improve existing XWF language, totally change to a new
>> > language
>> > > > >>>>> available
>> > > > >>>>> >> in industry or write a new light weight language. Basic
>> > > > >>>>>requirements
>> > > > >>>>> that
>> > > > >>>>> >> we expect from new improvement are, high usability,
>> flexible,
>> > > > >>>>>light
>> > > > >>>>> weight
>> > > > >>>>> >> and real time monitoring support. As you can see above
>> > > > >>>>>requirements
>> > > > >>>>> are
>> > > > >>>>> >> not
>> > > > >>>>> >> direct comes with workflow languages but we need workflow
>> > > language
>> > > > >>>>> which
>> > > > >>>>> >> help to support above requirements.
>> > > > >>>>> >>
>> > > > >>>>> >> After reading few papers and googling, initially i have
>> come
>> > up
>> > > > >>>>>with
>> > > > >>>>> >> following three existing languages,
>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>> > > > >>>>> >> 2. WS-BPEL
>> > > > >>>>> >> 3. SIDL
>> > > > >>>>> >> <
>> > > http://computation.llnl.gov/casc/components/index.html#page=home
>> > > > >
>> > > > >>>>> >>
>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
>> > > > >>>>> Radical-SAGA
>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
>> above
>> > > > >>>>>three
>> > > > >>>>> >> languages we can come up with simple workflow language
>> base on
>> > > > >>>>> json(or
>> > > > >>>>> >> yaml) which support all our requirements for some extends.
>> > > > >>>>> >>
>> > > > >>>>> >> It would be grate if I can get more input regarding the
>> > $Subject
>> > > > >>>>> form the
>> > > > >>>>> >> airavata community. You all are more than welcome to
>> provide
>> > any
>> > > > >>>>> type of
>> > > > >>>>> >> suggestions.
>> > > > >>>>> >>
>> > > > >>>>> >> Thanks,
>> > > > >>>>> >> Shameera.
>> > > > >>>>> >>
>> > > > >>>>> >> 
>> > > > >>>>> >>
>> > > > >>>>> >> --
>> > > > >>>>> >> Best Regards,
>> > > > >>>>> >> Shameera Rathnayaka.
>> > > > >>>>> >>
>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> --
>> > > > >>>> The best way to predict the future is to invent it - Alan Kay
>> > > > >>>>
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> --
>> > > > >>> Best Regards,
>> > > > >>> Shameera Rathnayaka.
>> > > > >>>
>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
>> > > > >>>
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >> --
>> > > > >> Supun Kamburugamuva
>> > > > >> Member, Apache Software Foundation; http://www.apache.org
>> > > > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> > > > >> Blog: http://supunk.blogspot.com
>> > > > >>
>> > > > >>
>> > > > >
>> > > > >
>> > > > >--
>> > > > >Supun Kamburugamuva
>> > > > >Member, Apache Software Foundation; http://www.apache.org
>> > > > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> > > > >Blog: http://supunk.blogspot.com
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Best Regards,
>> > > Shameera Rathnayaka.
>> > >
>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > Blog : http://shameerarathnayaka.blogspot.com/
>> > >
>> >
>> >
>> >
>> > --
>> > Supun Kamburugamuva
>> > Member, Apache Software Foundation; http://www.apache.org
>> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> > Blog: http://supunk.blogspot.com
>> >
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Supun Kamburugamuva <su...@gmail.com>.

I think I'm not suggesting to create a Workflow interpreter using Python
etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
and move it to a more higher level component. The more I think about it,
the model I'm suggesting is similar to what Hadoop, Storm etc has done for
distributed system computations. This model is proven to be successful over
the years.

Keeping what Airavata does at its core can help you to build a more robust
system. If we look at Airavata as middleware to execute applications on
computing resources we can simplify what Airavata does and focus on
improving the core functionality. All the successful systems have thrived
on defining what it does at its core and keeping it simple and being
excellent at what it does. In that regard keeping workflow aspect out of
Airavata can help you to focus on the core problem. That is to execute an
application on a remote computing resource in a fault tolerant and scalable
manner.

What I'm suggesting is to give the Orchestrator the capability to execute a
Driver program that is specified by the user. (This program can be written
in Python, Java or any other language). This driver program is similar to
what you define in a Hadoop or Storm configuration. The driver program
specifies the flow of the computation. It specifies what are the
applications needs to be executed, how to manipulate input output. The
driver program is the workflow for the user. Because the user specifies the
program he can program it to handle workflow steering etc. Every time the
user wants to execute this program he can tell Airavata to execute the
Driver program.

I'm also not 100% sure about all the details. But this can be a new way to
look at how systems like Airavata should behave. Your thoughts and
suggestions are more than welcome.

Thanks,
Supun..






On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:

> Hi Supun,
>
> I think we were in two context, because I as suggesting a way to serialize
> and deserialize the workflow description while you are suggesting to
> implement
> some kind of workflow interpreter using Python, where Python client can
> send
> thrift calls to Airavata server to run the application. I can see with your
> suggested
> approach we can control the workflow execution process from client side
> which
> make it easy to implement workflow debugger.As you mentioned this is a
> major change
> to Airavata. So we should neatly think as this will change our existing
> architecture.
>
> Still if someone need to use different language java, php, JS etc ... to
> run the same
> workflow which generated by Python, we need a language independent workflow
> description.
> My initial question was what is the best language for this?. But as I have
> explained in
> one of my previous reply, It is not matter what language we used Either we
> can use
> XML or JSON to write this description, what matters is how easy to generate
> workflow with the provided API. Hence it would be great to have set of neat
> APIs in
> different languages.
>
> Thanks,
> Shameera.
>
>
> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
> > Hi Shameera,
> >
> > Using python is a radical approach for workflow I think. But I believe
> this
> > is a very beautiful and new approach.Here is a possible scenario for
> > implementation and running using a Python based library.
> >
> > The Python library facilitates the creation of Applications and
> submitting
> > them to Airavata for execution. The underlying functionality is exactly
> > similar to what Java clients provides.The only difference is that, Python
> > library should have a more fluent API than Java for easy creation of
> > workflows. We can generate the Python clients that talk to Airavata
> server
> > using Thrift.
> >
> > Here is an example off the top of my head to a Python script created by
> > user for a Workflow. This is a very crude example and we need to come up
> > with a much better API if we are going to go along this path. First we
> need
> > to write a Python script that can execute a workflow using Airavata.
> >
> > import airavata
> >
> > host = Host("localhost", ....)
> > app1 = Application(host, ....)
> > app2 = Application(host, ....)
> >
> > # we will connect these applications as a workflow using some topology
> > builder or other constructs
> >
> > wb = WorkFlowBuilder()
> > wb.setApp("name1", app1)
> > # you can do a simple output transformation here etc
> >
> > # connects the input of app1 to app2 etc
> > wb.setApp("name2", app2).connectInput("name1")
> >
> > wb.submit()
> >
> > Now we can load this Python script from XBaya. When XBaya loads this
> script
> > the Python script can output an XML configuration of the topology, XBaya
> > can render. There are other ways like directly executing the Python
> script
> > from command line and connecting XBaya indirectly as well. Now you can
> run
> > the workflow from XBaya. Running the Workflow means just executing the
> > Python script.
> >
> > XBaya gets the notifications through messaging and update the UI
> > accordingly.
> >
> > The users need to write the Python script by hand. XBaya cannot create
> the
> > script. Because workflow language is an actual python program the
> benefits
> > are immense. For example user can do workflow steering in the workflow
> > itself by subscribing to messages from Airavata.
> >
> > Thanks,
> > Supun..
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> > shameerainfo@gmail.com> wrote:
> >
> > > Hi Supun,
> > >
> > > I meant to say JS is a well-known client side scripting language i have
> > > messed scripting part. Even we use Python, ultimately we should convert
> > > this to java model in server side, somehow we need to serialized python
> > > representation to the language which java can parse and generate that
> > > model. In this case we need to parse python script in java isn't it? I
> am
> > > not exactly clear how you suggesting to use python here. More details
> on
> > > how end system works if we use Python would be great help to clearly
> > > understand your points.
> > >
> > > Thanks,
> > > Shameera.
> > >
> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> > chris.mattmann@gmail.com>
> > > wrote:
> > >
> > > > Have you guys considered using JCC [1] as a means
> > > > to expose the workflow API currently in Java as a
> > > > Python API?
> > > >
> > > > We are exploring its use in OODT, and we have already
> > > > created a Tika [2] JCC-based python API.
> > > >
> > > > Cheers,
> > > > Chris
> > > >
> > > > [1] http://lucene.apache.org/pylucene/jcc/
> > > > [2] http://github.com/chrismattmann/tika-python/
> > > >
> > > > ------------------------
> > > > Chris Mattmann
> > > > chris.mattmann@gmail.com
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Supun Kamburugamuva <su...@gmail.com>
> > > > Reply-To: <de...@oodt.apache.org>
> > > > Date: Thursday, October 16, 2014 at 3:43 PM
> > > > To: dev <de...@airavata.apache.org>
> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> > > > <de...@oodt.apache.org>
> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> > Airavata.
> > > >
> > > > >Once we had an offline discussion about the Airavata Workflow
> language
> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
> came
> > > out
> > > > >was why we need to invent a different language when a simple library
> > > like
> > > > >Python will full fill of Airavata requirements.
> > > > >
> > > > >There are many benefits in using a Python library as the API for
> > > > >controlling Airavata workflows.
> > > > >
> > > > >1. It is a library, gives the ultimate control over the execution
> and
> > it
> > > > >can be simpler than any domain specific language that we can come
> with
> > > > >like
> > > > >XML, JSON etc
> > > > >2. Most people use python and can learn it easily than any Domain
> > > specific
> > > > >language
> > > > >3. You can easily create a Python library for Airavata because all
> the
> > > > >APIs
> > > > >are thrift based.
> > > > >4. If you design the constructs correctly you can plug an XBaya.
> > > > >
> > > > >Any thoughts?
> > > > >
> > > > >Thanks,
> > > > >Supun..
> > > > >
> > > > >
> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> > supun06@gmail.com
> > > >
> > > > >wrote:
> > > > >
> > > > >> Hi Shameera,
> > > > >>
> > > > >> Why you prefer JavaScript over a language like Python?
> > > > >>
> > > > >> Thanks,
> > > > >> Supun..
> > > > >>
> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > > > >> shameerainfo@gmail.com> wrote:
> > > > >>
> > > > >>> Hi,
> > > > >>>
> > > > >>> First of all thanks everyone for giving valuable inputs. After
> > doing
> > > > >>>some
> > > > >>> background search and talking to different people in the
> University
> > > > >>>who has
> > > > >>> used different workflow languages, I myself convinced that
> > > introducing
> > > > >>>an
> > > > >>> another workflow language is not what actually they need. By
> > changing
> > > > >>> exiting workflow language to another will not solve problems.
> What
> > > they
> > > > >>> asking is a easy way to construct the workflows. Indirectly what
> > they
> > > > >>> asking for a sort of API which they can use to generate the
> > workflows
> > > > >>>and
> > > > >>> run it. Correct me if i am wrong here.
> > > > >>>
> > > > >>> As most of above replies depict,  if we can get a simple API, as
> an
> > > > >>> example, for a web based application, JavaScript API would be a
> > good
> > > > >>> solution, and probably JSON would be a good candidate for
> language,
> > > > >>>instead
> > > > >>> of XML.
> > > > >>>
> > > > >>> Airavata community already have started to implement web base
> GUI.
> > > > >>>Hence
> > > > >>> introducing a JSON base JavaScript API would be great help. WDYT?
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Shameera.
> > > > >>>
> > > > >>>
> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> > > > >>> alekny7@gmail.com> wrote:
> > > > >>>
> > > > >>>> Hi,
> > > > >>>>
> > > > >>>> it is not dataflow instead focused on orchestrating REST
> services
> > > but
> > > > >>>> you may find it useful datapoint - we created worfklow service
> > that
> > > > >>>>uses
> > > > >>>> natively JavaScript and JSON to describe what happens during
> > > workflow
> > > > >>>> execution:
> > > > >>>>
> > > >
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> > > > >>>>
> > > > >>>> HTH,
> > > > >>>>
> > > > >>>> Alek
> > > > >>>>
> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
> smarru@apache.org>
> > > > >>>>wrote:
> > > > >>>>
> > > > >>>>> Hi Chris,
> > > > >>>>>
> > > > >>>>> Great to hear OODT community will be interested in adopting a
> > JSON
> > > > >>>>> based workflow language and potentially a web based composer as
> > > well.
> > > > >>>>> Airavata previously had BPEL support initially through a home
> > grown
> > > > >>>>> implementation [1] by Alek Slominski and later through Apache
> ODE
> > > > >>>>>[2]. Also
> > > > >>>>> a white paper [3] by Alek on this topic is an interesting read.
> > > > >>>>>
> > > > >>>>> I am of the same opinion that we should adopt something more
> > modern
> > > > >>>>>as
> > > > >>>>> the challenges from scientific workflows seems to be converging
> > > with
> > > > >>>>>the
> > > > >>>>> data flow patterns in business workflows.
> > > > >>>>>
> > > > >>>>> It will be great if we can all compile a list of potential
> > > candidates
> > > > >>>>> and hack them through.
> > > > >>>>>
> > > > >>>>> Suresh
> > > > >>>>> [1] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > > > >>>>> [2] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> > > > >>>>>time_for_eScience_workflows
> > > > >>>>> [3] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> > > > >>>>>6.pdf
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> > > > >>>>>
> > > > >>>>> > Hi Guys,
> > > > >>>>> >
> > > > >>>>> > I've been interested in this too - we don't per have a
> specific
> > > > >>>>> > OODT workflow language, but we specific workflows using XML,
> > and
> > > > >>>>> > other configuration (we are also thinking of moving to JSON
> for
> > > > >>>>> > this).
> > > > >>>>> >
> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
> > complex
> > > > >>>>> > to me.
> > > > >>>>> >
> > > > >>>>> > I wonder at the end of the day if we should adopt something
> > more
> > > > >>>>> > modern like PIG or some other data flow type of language (PIG
> > > > >>>>> > is really neat).
> > > > >>>>> >
> > > > >>>>> > Cheers,
> > > > >>>>> > Chris
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> > Chris Mattmann, Ph.D.
> > > > >>>>> > Chief Architect
> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > > >>>>> > Office: 168-519, Mailstop: 168-527
> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
> > > > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
> > > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> > -----Original Message-----
> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
> > > > >>>>> > <ar...@airavata.apache.org>
> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> > > > >>>>> > To: "architecture@airavata.apache.org" <
> > > > >>>>> architecture@airavata.apache.org>,
> > > > >>>>> > dev <de...@airavata.apache.org>
> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> > > > >>>>>Airavata.
> > > > >>>>> >
> > > > >>>>> >> Hi All,
> > > > >>>>> >>
> > > > >>>>> >> As we all know Airavata has its own workflow language call
> > XWF.
> > > > >>>>>When
> > > > >>>>> XWF
> > > > >>>>> >> was introduced, main focus points are interoperability and
> > > > >>>>> convertibility.
> > > > >>>>> >> But with years of experience it is convinced that above
> > > > >>>>>requirements
> > > > >>>>> are
> > > > >>>>> >> not really useful when we come to real world use cases. And
> > XWF
> > > is
> > > > >>>>> XML
> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
> image
> > > it
> > > > >>>>> self.
> > > > >>>>> >> But
> > > > >>>>> >> with the recent changes WSDL part is being removed from XWF.
> > > > >>>>> >>
> > > > >>>>> >> It is worth to evaluate handy Scientific workflow languages
> in
> > > > >>>>> industry
> > > > >>>>> >> and
> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
> need
> > to
> > > > >>>>> come up
> > > > >>>>> >> with idea how we should improve Airavata workflow language,
> > > either
> > > > >>>>> we can
> > > > >>>>> >> improve existing XWF language, totally change to a new
> > language
> > > > >>>>> available
> > > > >>>>> >> in industry or write a new light weight language. Basic
> > > > >>>>>requirements
> > > > >>>>> that
> > > > >>>>> >> we expect from new improvement are, high usability,
> flexible,
> > > > >>>>>light
> > > > >>>>> weight
> > > > >>>>> >> and real time monitoring support. As you can see above
> > > > >>>>>requirements
> > > > >>>>> are
> > > > >>>>> >> not
> > > > >>>>> >> direct comes with workflow languages but we need workflow
> > > language
> > > > >>>>> which
> > > > >>>>> >> help to support above requirements.
> > > > >>>>> >>
> > > > >>>>> >> After reading few papers and googling, initially i have come
> > up
> > > > >>>>>with
> > > > >>>>> >> following three existing languages,
> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> > > > >>>>> >> 2. WS-BPEL
> > > > >>>>> >> 3. SIDL
> > > > >>>>> >> <
> > > http://computation.llnl.gov/casc/components/index.html#page=home
> > > > >
> > > > >>>>> >>
> > > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> > > > >>>>> Radical-SAGA
> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
> above
> > > > >>>>>three
> > > > >>>>> >> languages we can come up with simple workflow language base
> on
> > > > >>>>> json(or
> > > > >>>>> >> yaml) which support all our requirements for some extends.
> > > > >>>>> >>
> > > > >>>>> >> It would be grate if I can get more input regarding the
> > $Subject
> > > > >>>>> form the
> > > > >>>>> >> airavata community. You all are more than welcome to provide
> > any
> > > > >>>>> type of
> > > > >>>>> >> suggestions.
> > > > >>>>> >>
> > > > >>>>> >> Thanks,
> > > > >>>>> >> Shameera.
> > > > >>>>> >>
> > > > >>>>> >> 
> > > > >>>>> >>
> > > > >>>>> >> --
> > > > >>>>> >> Best Regards,
> > > > >>>>> >> Shameera Rathnayaka.
> > > > >>>>> >>
> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> The best way to predict the future is to invent it - Alan Kay
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Best Regards,
> > > > >>> Shameera Rathnayaka.
> > > > >>>
> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
> > > > >>>
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Supun Kamburugamuva
> > > > >> Member, Apache Software Foundation; http://www.apache.org
> > > > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > > > >> Blog: http://supunk.blogspot.com
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > >--
> > > > >Supun Kamburugamuva
> > > > >Member, Apache Software Foundation; http://www.apache.org
> > > > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > > > >Blog: http://supunk.blogspot.com
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Best Regards,
> > > Shameera Rathnayaka.
> > >
> > > email: shameera AT apache.org , shameerainfo AT gmail.com
> > > Blog : http://shameerarathnayaka.blogspot.com/
> > >
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>



-- 
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Supun Kamburugamuva <su...@gmail.com>.

I think I'm not suggesting to create a Workflow interpreter using Python
etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
and move it to a more higher level component. The more I think about it,
the model I'm suggesting is similar to what Hadoop, Storm etc has done for
distributed system computations. This model is proven to be successful over
the years.

Keeping what Airavata does at its core can help you to build a more robust
system. If we look at Airavata as middleware to execute applications on
computing resources we can simplify what Airavata does and focus on
improving the core functionality. All the successful systems have thrived
on defining what it does at its core and keeping it simple and being
excellent at what it does. In that regard keeping workflow aspect out of
Airavata can help you to focus on the core problem. That is to execute an
application on a remote computing resource in a fault tolerant and scalable
manner.

What I'm suggesting is to give the Orchestrator the capability to execute a
Driver program that is specified by the user. (This program can be written
in Python, Java or any other language). This driver program is similar to
what you define in a Hadoop or Storm configuration. The driver program
specifies the flow of the computation. It specifies what are the
applications needs to be executed, how to manipulate input output. The
driver program is the workflow for the user. Because the user specifies the
program he can program it to handle workflow steering etc. Every time the
user wants to execute this program he can tell Airavata to execute the
Driver program.

I'm also not 100% sure about all the details. But this can be a new way to
look at how systems like Airavata should behave. Your thoughts and
suggestions are more than welcome.

Thanks,
Supun..






On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:

> Hi Supun,
>
> I think we were in two context, because I as suggesting a way to serialize
> and deserialize the workflow description while you are suggesting to
> implement
> some kind of workflow interpreter using Python, where Python client can
> send
> thrift calls to Airavata server to run the application. I can see with your
> suggested
> approach we can control the workflow execution process from client side
> which
> make it easy to implement workflow debugger.As you mentioned this is a
> major change
> to Airavata. So we should neatly think as this will change our existing
> architecture.
>
> Still if someone need to use different language java, php, JS etc ... to
> run the same
> workflow which generated by Python, we need a language independent workflow
> description.
> My initial question was what is the best language for this?. But as I have
> explained in
> one of my previous reply, It is not matter what language we used Either we
> can use
> XML or JSON to write this description, what matters is how easy to generate
> workflow with the provided API. Hence it would be great to have set of neat
> APIs in
> different languages.
>
> Thanks,
> Shameera.
>
>
> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
> > Hi Shameera,
> >
> > Using python is a radical approach for workflow I think. But I believe
> this
> > is a very beautiful and new approach.Here is a possible scenario for
> > implementation and running using a Python based library.
> >
> > The Python library facilitates the creation of Applications and
> submitting
> > them to Airavata for execution. The underlying functionality is exactly
> > similar to what Java clients provides.The only difference is that, Python
> > library should have a more fluent API than Java for easy creation of
> > workflows. We can generate the Python clients that talk to Airavata
> server
> > using Thrift.
> >
> > Here is an example off the top of my head to a Python script created by
> > user for a Workflow. This is a very crude example and we need to come up
> > with a much better API if we are going to go along this path. First we
> need
> > to write a Python script that can execute a workflow using Airavata.
> >
> > import airavata
> >
> > host = Host("localhost", ....)
> > app1 = Application(host, ....)
> > app2 = Application(host, ....)
> >
> > # we will connect these applications as a workflow using some topology
> > builder or other constructs
> >
> > wb = WorkFlowBuilder()
> > wb.setApp("name1", app1)
> > # you can do a simple output transformation here etc
> >
> > # connects the input of app1 to app2 etc
> > wb.setApp("name2", app2).connectInput("name1")
> >
> > wb.submit()
> >
> > Now we can load this Python script from XBaya. When XBaya loads this
> script
> > the Python script can output an XML configuration of the topology, XBaya
> > can render. There are other ways like directly executing the Python
> script
> > from command line and connecting XBaya indirectly as well. Now you can
> run
> > the workflow from XBaya. Running the Workflow means just executing the
> > Python script.
> >
> > XBaya gets the notifications through messaging and update the UI
> > accordingly.
> >
> > The users need to write the Python script by hand. XBaya cannot create
> the
> > script. Because workflow language is an actual python program the
> benefits
> > are immense. For example user can do workflow steering in the workflow
> > itself by subscribing to messages from Airavata.
> >
> > Thanks,
> > Supun..
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> > shameerainfo@gmail.com> wrote:
> >
> > > Hi Supun,
> > >
> > > I meant to say JS is a well-known client side scripting language i have
> > > messed scripting part. Even we use Python, ultimately we should convert
> > > this to java model in server side, somehow we need to serialized python
> > > representation to the language which java can parse and generate that
> > > model. In this case we need to parse python script in java isn't it? I
> am
> > > not exactly clear how you suggesting to use python here. More details
> on
> > > how end system works if we use Python would be great help to clearly
> > > understand your points.
> > >
> > > Thanks,
> > > Shameera.
> > >
> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> > chris.mattmann@gmail.com>
> > > wrote:
> > >
> > > > Have you guys considered using JCC [1] as a means
> > > > to expose the workflow API currently in Java as a
> > > > Python API?
> > > >
> > > > We are exploring its use in OODT, and we have already
> > > > created a Tika [2] JCC-based python API.
> > > >
> > > > Cheers,
> > > > Chris
> > > >
> > > > [1] http://lucene.apache.org/pylucene/jcc/
> > > > [2] http://github.com/chrismattmann/tika-python/
> > > >
> > > > ------------------------
> > > > Chris Mattmann
> > > > chris.mattmann@gmail.com
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Supun Kamburugamuva <su...@gmail.com>
> > > > Reply-To: <de...@oodt.apache.org>
> > > > Date: Thursday, October 16, 2014 at 3:43 PM
> > > > To: dev <de...@airavata.apache.org>
> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> > > > <de...@oodt.apache.org>
> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> > Airavata.
> > > >
> > > > >Once we had an offline discussion about the Airavata Workflow
> language
> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
> came
> > > out
> > > > >was why we need to invent a different language when a simple library
> > > like
> > > > >Python will full fill of Airavata requirements.
> > > > >
> > > > >There are many benefits in using a Python library as the API for
> > > > >controlling Airavata workflows.
> > > > >
> > > > >1. It is a library, gives the ultimate control over the execution
> and
> > it
> > > > >can be simpler than any domain specific language that we can come
> with
> > > > >like
> > > > >XML, JSON etc
> > > > >2. Most people use python and can learn it easily than any Domain
> > > specific
> > > > >language
> > > > >3. You can easily create a Python library for Airavata because all
> the
> > > > >APIs
> > > > >are thrift based.
> > > > >4. If you design the constructs correctly you can plug an XBaya.
> > > > >
> > > > >Any thoughts?
> > > > >
> > > > >Thanks,
> > > > >Supun..
> > > > >
> > > > >
> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> > supun06@gmail.com
> > > >
> > > > >wrote:
> > > > >
> > > > >> Hi Shameera,
> > > > >>
> > > > >> Why you prefer JavaScript over a language like Python?
> > > > >>
> > > > >> Thanks,
> > > > >> Supun..
> > > > >>
> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > > > >> shameerainfo@gmail.com> wrote:
> > > > >>
> > > > >>> Hi,
> > > > >>>
> > > > >>> First of all thanks everyone for giving valuable inputs. After
> > doing
> > > > >>>some
> > > > >>> background search and talking to different people in the
> University
> > > > >>>who has
> > > > >>> used different workflow languages, I myself convinced that
> > > introducing
> > > > >>>an
> > > > >>> another workflow language is not what actually they need. By
> > changing
> > > > >>> exiting workflow language to another will not solve problems.
> What
> > > they
> > > > >>> asking is a easy way to construct the workflows. Indirectly what
> > they
> > > > >>> asking for a sort of API which they can use to generate the
> > workflows
> > > > >>>and
> > > > >>> run it. Correct me if i am wrong here.
> > > > >>>
> > > > >>> As most of above replies depict,  if we can get a simple API, as
> an
> > > > >>> example, for a web based application, JavaScript API would be a
> > good
> > > > >>> solution, and probably JSON would be a good candidate for
> language,
> > > > >>>instead
> > > > >>> of XML.
> > > > >>>
> > > > >>> Airavata community already have started to implement web base
> GUI.
> > > > >>>Hence
> > > > >>> introducing a JSON base JavaScript API would be great help. WDYT?
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Shameera.
> > > > >>>
> > > > >>>
> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> > > > >>> alekny7@gmail.com> wrote:
> > > > >>>
> > > > >>>> Hi,
> > > > >>>>
> > > > >>>> it is not dataflow instead focused on orchestrating REST
> services
> > > but
> > > > >>>> you may find it useful datapoint - we created worfklow service
> > that
> > > > >>>>uses
> > > > >>>> natively JavaScript and JSON to describe what happens during
> > > workflow
> > > > >>>> execution:
> > > > >>>>
> > > >
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> > > > >>>>
> > > > >>>> HTH,
> > > > >>>>
> > > > >>>> Alek
> > > > >>>>
> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
> smarru@apache.org>
> > > > >>>>wrote:
> > > > >>>>
> > > > >>>>> Hi Chris,
> > > > >>>>>
> > > > >>>>> Great to hear OODT community will be interested in adopting a
> > JSON
> > > > >>>>> based workflow language and potentially a web based composer as
> > > well.
> > > > >>>>> Airavata previously had BPEL support initially through a home
> > grown
> > > > >>>>> implementation [1] by Alek Slominski and later through Apache
> ODE
> > > > >>>>>[2]. Also
> > > > >>>>> a white paper [3] by Alek on this topic is an interesting read.
> > > > >>>>>
> > > > >>>>> I am of the same opinion that we should adopt something more
> > modern
> > > > >>>>>as
> > > > >>>>> the challenges from scientific workflows seems to be converging
> > > with
> > > > >>>>>the
> > > > >>>>> data flow patterns in business workflows.
> > > > >>>>>
> > > > >>>>> It will be great if we can all compile a list of potential
> > > candidates
> > > > >>>>> and hack them through.
> > > > >>>>>
> > > > >>>>> Suresh
> > > > >>>>> [1] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > > > >>>>> [2] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> > > > >>>>>time_for_eScience_workflows
> > > > >>>>> [3] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> > > > >>>>>6.pdf
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> > > > >>>>>
> > > > >>>>> > Hi Guys,
> > > > >>>>> >
> > > > >>>>> > I've been interested in this too - we don't per have a
> specific
> > > > >>>>> > OODT workflow language, but we specific workflows using XML,
> > and
> > > > >>>>> > other configuration (we are also thinking of moving to JSON
> for
> > > > >>>>> > this).
> > > > >>>>> >
> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
> > complex
> > > > >>>>> > to me.
> > > > >>>>> >
> > > > >>>>> > I wonder at the end of the day if we should adopt something
> > more
> > > > >>>>> > modern like PIG or some other data flow type of language (PIG
> > > > >>>>> > is really neat).
> > > > >>>>> >
> > > > >>>>> > Cheers,
> > > > >>>>> > Chris
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> > Chris Mattmann, Ph.D.
> > > > >>>>> > Chief Architect
> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > > >>>>> > Office: 168-519, Mailstop: 168-527
> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
> > > > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
> > > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> > -----Original Message-----
> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
> > > > >>>>> > <ar...@airavata.apache.org>
> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> > > > >>>>> > To: "architecture@airavata.apache.org" <
> > > > >>>>> architecture@airavata.apache.org>,
> > > > >>>>> > dev <de...@airavata.apache.org>
> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> > > > >>>>>Airavata.
> > > > >>>>> >
> > > > >>>>> >> Hi All,
> > > > >>>>> >>
> > > > >>>>> >> As we all know Airavata has its own workflow language call
> > XWF.
> > > > >>>>>When
> > > > >>>>> XWF
> > > > >>>>> >> was introduced, main focus points are interoperability and
> > > > >>>>> convertibility.
> > > > >>>>> >> But with years of experience it is convinced that above
> > > > >>>>>requirements
> > > > >>>>> are
> > > > >>>>> >> not really useful when we come to real world use cases. And
> > XWF
> > > is
> > > > >>>>> XML
> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
> image
> > > it
> > > > >>>>> self.
> > > > >>>>> >> But
> > > > >>>>> >> with the recent changes WSDL part is being removed from XWF.
> > > > >>>>> >>
> > > > >>>>> >> It is worth to evaluate handy Scientific workflow languages
> in
> > > > >>>>> industry
> > > > >>>>> >> and
> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
> need
> > to
> > > > >>>>> come up
> > > > >>>>> >> with idea how we should improve Airavata workflow language,
> > > either
> > > > >>>>> we can
> > > > >>>>> >> improve existing XWF language, totally change to a new
> > language
> > > > >>>>> available
> > > > >>>>> >> in industry or write a new light weight language. Basic
> > > > >>>>>requirements
> > > > >>>>> that
> > > > >>>>> >> we expect from new improvement are, high usability,
> flexible,
> > > > >>>>>light
> > > > >>>>> weight
> > > > >>>>> >> and real time monitoring support. As you can see above
> > > > >>>>>requirements
> > > > >>>>> are
> > > > >>>>> >> not
> > > > >>>>> >> direct comes with workflow languages but we need workflow
> > > language
> > > > >>>>> which
> > > > >>>>> >> help to support above requirements.
> > > > >>>>> >>
> > > > >>>>> >> After reading few papers and googling, initially i have come
> > up
> > > > >>>>>with
> > > > >>>>> >> following three existing languages,
> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> > > > >>>>> >> 2. WS-BPEL
> > > > >>>>> >> 3. SIDL
> > > > >>>>> >> <
> > > http://computation.llnl.gov/casc/components/index.html#page=home
> > > > >
> > > > >>>>> >>
> > > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> > > > >>>>> Radical-SAGA
> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
> above
> > > > >>>>>three
> > > > >>>>> >> languages we can come up with simple workflow language base
> on
> > > > >>>>> json(or
> > > > >>>>> >> yaml) which support all our requirements for some extends.
> > > > >>>>> >>
> > > > >>>>> >> It would be grate if I can get more input regarding the
> > $Subject
> > > > >>>>> form the
> > > > >>>>> >> airavata community. You all are more than welcome to provide
> > any
> > > > >>>>> type of
> > > > >>>>> >> suggestions.
> > > > >>>>> >>
> > > > >>>>> >> Thanks,
> > > > >>>>> >> Shameera.
> > > > >>>>> >>
> > > > >>>>> >> 
> > > > >>>>> >>
> > > > >>>>> >> --
> > > > >>>>> >> Best Regards,
> > > > >>>>> >> Shameera Rathnayaka.
> > > > >>>>> >>
> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> The best way to predict the future is to invent it - Alan Kay
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Best Regards,
> > > > >>> Shameera Rathnayaka.
> > > > >>>
> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
> > > > >>>
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Supun Kamburugamuva
> > > > >> Member, Apache Software Foundation; http://www.apache.org
> > > > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > > > >> Blog: http://supunk.blogspot.com
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > >--
> > > > >Supun Kamburugamuva
> > > > >Member, Apache Software Foundation; http://www.apache.org
> > > > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > > > >Blog: http://supunk.blogspot.com
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Best Regards,
> > > Shameera Rathnayaka.
> > >
> > > email: shameera AT apache.org , shameerainfo AT gmail.com
> > > Blog : http://shameerarathnayaka.blogspot.com/
> > >
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>



-- 
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Supun Kamburugamuva <su...@gmail.com>.

I think I'm not suggesting to create a Workflow interpreter using Python
etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
and move it to a more higher level component. The more I think about it,
the model I'm suggesting is similar to what Hadoop, Storm etc has done for
distributed system computations. This model is proven to be successful over
the years.

Keeping what Airavata does at its core can help you to build a more robust
system. If we look at Airavata as middleware to execute applications on
computing resources we can simplify what Airavata does and focus on
improving the core functionality. All the successful systems have thrived
on defining what it does at its core and keeping it simple and being
excellent at what it does. In that regard keeping workflow aspect out of
Airavata can help you to focus on the core problem. That is to execute an
application on a remote computing resource in a fault tolerant and scalable
manner.

What I'm suggesting is to give the Orchestrator the capability to execute a
Driver program that is specified by the user. (This program can be written
in Python, Java or any other language). This driver program is similar to
what you define in a Hadoop or Storm configuration. The driver program
specifies the flow of the computation. It specifies what are the
applications needs to be executed, how to manipulate input output. The
driver program is the workflow for the user. Because the user specifies the
program he can program it to handle workflow steering etc. Every time the
user wants to execute this program he can tell Airavata to execute the
Driver program.

I'm also not 100% sure about all the details. But this can be a new way to
look at how systems like Airavata should behave. Your thoughts and
suggestions are more than welcome.

Thanks,
Supun..






On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:

> Hi Supun,
>
> I think we were in two context, because I as suggesting a way to serialize
> and deserialize the workflow description while you are suggesting to
> implement
> some kind of workflow interpreter using Python, where Python client can
> send
> thrift calls to Airavata server to run the application. I can see with your
> suggested
> approach we can control the workflow execution process from client side
> which
> make it easy to implement workflow debugger.As you mentioned this is a
> major change
> to Airavata. So we should neatly think as this will change our existing
> architecture.
>
> Still if someone need to use different language java, php, JS etc ... to
> run the same
> workflow which generated by Python, we need a language independent workflow
> description.
> My initial question was what is the best language for this?. But as I have
> explained in
> one of my previous reply, It is not matter what language we used Either we
> can use
> XML or JSON to write this description, what matters is how easy to generate
> workflow with the provided API. Hence it would be great to have set of neat
> APIs in
> different languages.
>
> Thanks,
> Shameera.
>
>
> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
> > Hi Shameera,
> >
> > Using python is a radical approach for workflow I think. But I believe
> this
> > is a very beautiful and new approach.Here is a possible scenario for
> > implementation and running using a Python based library.
> >
> > The Python library facilitates the creation of Applications and
> submitting
> > them to Airavata for execution. The underlying functionality is exactly
> > similar to what Java clients provides.The only difference is that, Python
> > library should have a more fluent API than Java for easy creation of
> > workflows. We can generate the Python clients that talk to Airavata
> server
> > using Thrift.
> >
> > Here is an example off the top of my head to a Python script created by
> > user for a Workflow. This is a very crude example and we need to come up
> > with a much better API if we are going to go along this path. First we
> need
> > to write a Python script that can execute a workflow using Airavata.
> >
> > import airavata
> >
> > host = Host("localhost", ....)
> > app1 = Application(host, ....)
> > app2 = Application(host, ....)
> >
> > # we will connect these applications as a workflow using some topology
> > builder or other constructs
> >
> > wb = WorkFlowBuilder()
> > wb.setApp("name1", app1)
> > # you can do a simple output transformation here etc
> >
> > # connects the input of app1 to app2 etc
> > wb.setApp("name2", app2).connectInput("name1")
> >
> > wb.submit()
> >
> > Now we can load this Python script from XBaya. When XBaya loads this
> script
> > the Python script can output an XML configuration of the topology, XBaya
> > can render. There are other ways like directly executing the Python
> script
> > from command line and connecting XBaya indirectly as well. Now you can
> run
> > the workflow from XBaya. Running the Workflow means just executing the
> > Python script.
> >
> > XBaya gets the notifications through messaging and update the UI
> > accordingly.
> >
> > The users need to write the Python script by hand. XBaya cannot create
> the
> > script. Because workflow language is an actual python program the
> benefits
> > are immense. For example user can do workflow steering in the workflow
> > itself by subscribing to messages from Airavata.
> >
> > Thanks,
> > Supun..
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> > shameerainfo@gmail.com> wrote:
> >
> > > Hi Supun,
> > >
> > > I meant to say JS is a well-known client side scripting language i have
> > > messed scripting part. Even we use Python, ultimately we should convert
> > > this to java model in server side, somehow we need to serialized python
> > > representation to the language which java can parse and generate that
> > > model. In this case we need to parse python script in java isn't it? I
> am
> > > not exactly clear how you suggesting to use python here. More details
> on
> > > how end system works if we use Python would be great help to clearly
> > > understand your points.
> > >
> > > Thanks,
> > > Shameera.
> > >
> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> > chris.mattmann@gmail.com>
> > > wrote:
> > >
> > > > Have you guys considered using JCC [1] as a means
> > > > to expose the workflow API currently in Java as a
> > > > Python API?
> > > >
> > > > We are exploring its use in OODT, and we have already
> > > > created a Tika [2] JCC-based python API.
> > > >
> > > > Cheers,
> > > > Chris
> > > >
> > > > [1] http://lucene.apache.org/pylucene/jcc/
> > > > [2] http://github.com/chrismattmann/tika-python/
> > > >
> > > > ------------------------
> > > > Chris Mattmann
> > > > chris.mattmann@gmail.com
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Supun Kamburugamuva <su...@gmail.com>
> > > > Reply-To: <de...@oodt.apache.org>
> > > > Date: Thursday, October 16, 2014 at 3:43 PM
> > > > To: dev <de...@airavata.apache.org>
> > > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> > > > <sm...@apache.org>, "architecture@airavata.apache.org"
> > > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> > > > <de...@oodt.apache.org>
> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> > Airavata.
> > > >
> > > > >Once we had an offline discussion about the Airavata Workflow
> language
> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
> came
> > > out
> > > > >was why we need to invent a different language when a simple library
> > > like
> > > > >Python will full fill of Airavata requirements.
> > > > >
> > > > >There are many benefits in using a Python library as the API for
> > > > >controlling Airavata workflows.
> > > > >
> > > > >1. It is a library, gives the ultimate control over the execution
> and
> > it
> > > > >can be simpler than any domain specific language that we can come
> with
> > > > >like
> > > > >XML, JSON etc
> > > > >2. Most people use python and can learn it easily than any Domain
> > > specific
> > > > >language
> > > > >3. You can easily create a Python library for Airavata because all
> the
> > > > >APIs
> > > > >are thrift based.
> > > > >4. If you design the constructs correctly you can plug an XBaya.
> > > > >
> > > > >Any thoughts?
> > > > >
> > > > >Thanks,
> > > > >Supun..
> > > > >
> > > > >
> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> > supun06@gmail.com
> > > >
> > > > >wrote:
> > > > >
> > > > >> Hi Shameera,
> > > > >>
> > > > >> Why you prefer JavaScript over a language like Python?
> > > > >>
> > > > >> Thanks,
> > > > >> Supun..
> > > > >>
> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > > > >> shameerainfo@gmail.com> wrote:
> > > > >>
> > > > >>> Hi,
> > > > >>>
> > > > >>> First of all thanks everyone for giving valuable inputs. After
> > doing
> > > > >>>some
> > > > >>> background search and talking to different people in the
> University
> > > > >>>who has
> > > > >>> used different workflow languages, I myself convinced that
> > > introducing
> > > > >>>an
> > > > >>> another workflow language is not what actually they need. By
> > changing
> > > > >>> exiting workflow language to another will not solve problems.
> What
> > > they
> > > > >>> asking is a easy way to construct the workflows. Indirectly what
> > they
> > > > >>> asking for a sort of API which they can use to generate the
> > workflows
> > > > >>>and
> > > > >>> run it. Correct me if i am wrong here.
> > > > >>>
> > > > >>> As most of above replies depict,  if we can get a simple API, as
> an
> > > > >>> example, for a web based application, JavaScript API would be a
> > good
> > > > >>> solution, and probably JSON would be a good candidate for
> language,
> > > > >>>instead
> > > > >>> of XML.
> > > > >>>
> > > > >>> Airavata community already have started to implement web base
> GUI.
> > > > >>>Hence
> > > > >>> introducing a JSON base JavaScript API would be great help. WDYT?
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Shameera.
> > > > >>>
> > > > >>>
> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> > > > >>> alekny7@gmail.com> wrote:
> > > > >>>
> > > > >>>> Hi,
> > > > >>>>
> > > > >>>> it is not dataflow instead focused on orchestrating REST
> services
> > > but
> > > > >>>> you may find it useful datapoint - we created worfklow service
> > that
> > > > >>>>uses
> > > > >>>> natively JavaScript and JSON to describe what happens during
> > > workflow
> > > > >>>> execution:
> > > > >>>>
> > > >
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> > > > >>>>
> > > > >>>> HTH,
> > > > >>>>
> > > > >>>> Alek
> > > > >>>>
> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
> smarru@apache.org>
> > > > >>>>wrote:
> > > > >>>>
> > > > >>>>> Hi Chris,
> > > > >>>>>
> > > > >>>>> Great to hear OODT community will be interested in adopting a
> > JSON
> > > > >>>>> based workflow language and potentially a web based composer as
> > > well.
> > > > >>>>> Airavata previously had BPEL support initially through a home
> > grown
> > > > >>>>> implementation [1] by Alek Slominski and later through Apache
> ODE
> > > > >>>>>[2]. Also
> > > > >>>>> a white paper [3] by Alek on this topic is an interesting read.
> > > > >>>>>
> > > > >>>>> I am of the same opinion that we should adopt something more
> > modern
> > > > >>>>>as
> > > > >>>>> the challenges from scientific workflows seems to be converging
> > > with
> > > > >>>>>the
> > > > >>>>> data flow patterns in business workflows.
> > > > >>>>>
> > > > >>>>> It will be great if we can all compile a list of potential
> > > candidates
> > > > >>>>> and hack them through.
> > > > >>>>>
> > > > >>>>> Suresh
> > > > >>>>> [1] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > > > >>>>> [2] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> > > > >>>>>time_for_eScience_workflows
> > > > >>>>> [3] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> > > > >>>>>6.pdf
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> > > > >>>>>
> > > > >>>>> > Hi Guys,
> > > > >>>>> >
> > > > >>>>> > I've been interested in this too - we don't per have a
> specific
> > > > >>>>> > OODT workflow language, but we specific workflows using XML,
> > and
> > > > >>>>> > other configuration (we are also thinking of moving to JSON
> for
> > > > >>>>> > this).
> > > > >>>>> >
> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
> > complex
> > > > >>>>> > to me.
> > > > >>>>> >
> > > > >>>>> > I wonder at the end of the day if we should adopt something
> > more
> > > > >>>>> > modern like PIG or some other data flow type of language (PIG
> > > > >>>>> > is really neat).
> > > > >>>>> >
> > > > >>>>> > Cheers,
> > > > >>>>> > Chris
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> > Chris Mattmann, Ph.D.
> > > > >>>>> > Chief Architect
> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > > >>>>> > Office: 168-519, Mailstop: 168-527
> > > > >>>>> > Email: chris.a.mattmann@nasa.gov
> > > > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
> > > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> > -----Original Message-----
> > > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> > > > >>>>> > Reply-To: "architecture@airavata.apache.org"
> > > > >>>>> > <ar...@airavata.apache.org>
> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> > > > >>>>> > To: "architecture@airavata.apache.org" <
> > > > >>>>> architecture@airavata.apache.org>,
> > > > >>>>> > dev <de...@airavata.apache.org>
> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> > > > >>>>>Airavata.
> > > > >>>>> >
> > > > >>>>> >> Hi All,
> > > > >>>>> >>
> > > > >>>>> >> As we all know Airavata has its own workflow language call
> > XWF.
> > > > >>>>>When
> > > > >>>>> XWF
> > > > >>>>> >> was introduced, main focus points are interoperability and
> > > > >>>>> convertibility.
> > > > >>>>> >> But with years of experience it is convinced that above
> > > > >>>>>requirements
> > > > >>>>> are
> > > > >>>>> >> not really useful when we come to real world use cases. And
> > XWF
> > > is
> > > > >>>>> XML
> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
> image
> > > it
> > > > >>>>> self.
> > > > >>>>> >> But
> > > > >>>>> >> with the recent changes WSDL part is being removed from XWF.
> > > > >>>>> >>
> > > > >>>>> >> It is worth to evaluate handy Scientific workflow languages
> in
> > > > >>>>> industry
> > > > >>>>> >> and
> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
> need
> > to
> > > > >>>>> come up
> > > > >>>>> >> with idea how we should improve Airavata workflow language,
> > > either
> > > > >>>>> we can
> > > > >>>>> >> improve existing XWF language, totally change to a new
> > language
> > > > >>>>> available
> > > > >>>>> >> in industry or write a new light weight language. Basic
> > > > >>>>>requirements
> > > > >>>>> that
> > > > >>>>> >> we expect from new improvement are, high usability,
> flexible,
> > > > >>>>>light
> > > > >>>>> weight
> > > > >>>>> >> and real time monitoring support. As you can see above
> > > > >>>>>requirements
> > > > >>>>> are
> > > > >>>>> >> not
> > > > >>>>> >> direct comes with workflow languages but we need workflow
> > > language
> > > > >>>>> which
> > > > >>>>> >> help to support above requirements.
> > > > >>>>> >>
> > > > >>>>> >> After reading few papers and googling, initially i have come
> > up
> > > > >>>>>with
> > > > >>>>> >> following three existing languages,
> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> > > > >>>>> >> 2. WS-BPEL
> > > > >>>>> >> 3. SIDL
> > > > >>>>> >> <
> > > http://computation.llnl.gov/casc/components/index.html#page=home
> > > > >
> > > > >>>>> >>
> > > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> > > > >>>>> Radical-SAGA
> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
> above
> > > > >>>>>three
> > > > >>>>> >> languages we can come up with simple workflow language base
> on
> > > > >>>>> json(or
> > > > >>>>> >> yaml) which support all our requirements for some extends.
> > > > >>>>> >>
> > > > >>>>> >> It would be grate if I can get more input regarding the
> > $Subject
> > > > >>>>> form the
> > > > >>>>> >> airavata community. You all are more than welcome to provide
> > any
> > > > >>>>> type of
> > > > >>>>> >> suggestions.
> > > > >>>>> >>
> > > > >>>>> >> Thanks,
> > > > >>>>> >> Shameera.
> > > > >>>>> >>
> > > > >>>>> >> 
> > > > >>>>> >>
> > > > >>>>> >> --
> > > > >>>>> >> Best Regards,
> > > > >>>>> >> Shameera Rathnayaka.
> > > > >>>>> >>
> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> The best way to predict the future is to invent it - Alan Kay
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Best Regards,
> > > > >>> Shameera Rathnayaka.
> > > > >>>
> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
> > > > >>>
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Supun Kamburugamuva
> > > > >> Member, Apache Software Foundation; http://www.apache.org
> > > > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > > > >> Blog: http://supunk.blogspot.com
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > >--
> > > > >Supun Kamburugamuva
> > > > >Member, Apache Software Foundation; http://www.apache.org
> > > > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > > > >Blog: http://supunk.blogspot.com
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Best Regards,
> > > Shameera Rathnayaka.
> > >
> > > email: shameera AT apache.org , shameerainfo AT gmail.com
> > > Blog : http://shameerarathnayaka.blogspot.com/
> > >
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>



-- 
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Shameera Rathnayaka <sh...@gmail.com>.

Hi Supun,

I think we were in two context, because I as suggesting a way to serialize
and deserialize the workflow description while you are suggesting to
implement
some kind of workflow interpreter using Python, where Python client can
send
thrift calls to Airavata server to run the application. I can see with your
suggested
approach we can control the workflow execution process from client side
which
make it easy to implement workflow debugger.As you mentioned this is a
major change
to Airavata. So we should neatly think as this will change our existing
architecture.

Still if someone need to use different language java, php, JS etc ... to
run the same
workflow which generated by Python, we need a language independent workflow
description.
My initial question was what is the best language for this?. But as I have
explained in
one of my previous reply, It is not matter what language we used Either we
can use
XML or JSON to write this description, what matters is how easy to generate
workflow with the provided API. Hence it would be great to have set of neat
APIs in
different languages.

Thanks,
Shameera.


On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:

> Hi Shameera,
>
> Using python is a radical approach for workflow I think. But I believe this
> is a very beautiful and new approach.Here is a possible scenario for
> implementation and running using a Python based library.
>
> The Python library facilitates the creation of Applications and submitting
> them to Airavata for execution. The underlying functionality is exactly
> similar to what Java clients provides.The only difference is that, Python
> library should have a more fluent API than Java for easy creation of
> workflows. We can generate the Python clients that talk to Airavata server
> using Thrift.
>
> Here is an example off the top of my head to a Python script created by
> user for a Workflow. This is a very crude example and we need to come up
> with a much better API if we are going to go along this path. First we need
> to write a Python script that can execute a workflow using Airavata.
>
> import airavata
>
> host = Host("localhost", ....)
> app1 = Application(host, ....)
> app2 = Application(host, ....)
>
> # we will connect these applications as a workflow using some topology
> builder or other constructs
>
> wb = WorkFlowBuilder()
> wb.setApp("name1", app1)
> # you can do a simple output transformation here etc
>
> # connects the input of app1 to app2 etc
> wb.setApp("name2", app2).connectInput("name1")
>
> wb.submit()
>
> Now we can load this Python script from XBaya. When XBaya loads this script
> the Python script can output an XML configuration of the topology, XBaya
> can render. There are other ways like directly executing the Python script
> from command line and connecting XBaya indirectly as well. Now you can run
> the workflow from XBaya. Running the Workflow means just executing the
> Python script.
>
> XBaya gets the notifications through messaging and update the UI
> accordingly.
>
> The users need to write the Python script by hand. XBaya cannot create the
> script. Because workflow language is an actual python program the benefits
> are immense. For example user can do workflow steering in the workflow
> itself by subscribing to messages from Airavata.
>
> Thanks,
> Supun..
>
>
>
>
>
>
>
>
> On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
> > Hi Supun,
> >
> > I meant to say JS is a well-known client side scripting language i have
> > messed scripting part. Even we use Python, ultimately we should convert
> > this to java model in server side, somehow we need to serialized python
> > representation to the language which java can parse and generate that
> > model. In this case we need to parse python script in java isn't it? I am
> > not exactly clear how you suggesting to use python here. More details on
> > how end system works if we use Python would be great help to clearly
> > understand your points.
> >
> > Thanks,
> > Shameera.
> >
> > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> chris.mattmann@gmail.com>
> > wrote:
> >
> > > Have you guys considered using JCC [1] as a means
> > > to expose the workflow API currently in Java as a
> > > Python API?
> > >
> > > We are exploring its use in OODT, and we have already
> > > created a Tika [2] JCC-based python API.
> > >
> > > Cheers,
> > > Chris
> > >
> > > [1] http://lucene.apache.org/pylucene/jcc/
> > > [2] http://github.com/chrismattmann/tika-python/
> > >
> > > ------------------------
> > > Chris Mattmann
> > > chris.mattmann@gmail.com
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Supun Kamburugamuva <su...@gmail.com>
> > > Reply-To: <de...@oodt.apache.org>
> > > Date: Thursday, October 16, 2014 at 3:43 PM
> > > To: dev <de...@airavata.apache.org>
> > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> > > <sm...@apache.org>, "architecture@airavata.apache.org"
> > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> > > <de...@oodt.apache.org>
> > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> Airavata.
> > >
> > > >Once we had an offline discussion about the Airavata Workflow language
> > > >(with Milinda, Saliya and Shameera). In that discussion one thing came
> > out
> > > >was why we need to invent a different language when a simple library
> > like
> > > >Python will full fill of Airavata requirements.
> > > >
> > > >There are many benefits in using a Python library as the API for
> > > >controlling Airavata workflows.
> > > >
> > > >1. It is a library, gives the ultimate control over the execution and
> it
> > > >can be simpler than any domain specific language that we can come with
> > > >like
> > > >XML, JSON etc
> > > >2. Most people use python and can learn it easily than any Domain
> > specific
> > > >language
> > > >3. You can easily create a Python library for Airavata because all the
> > > >APIs
> > > >are thrift based.
> > > >4. If you design the constructs correctly you can plug an XBaya.
> > > >
> > > >Any thoughts?
> > > >
> > > >Thanks,
> > > >Supun..
> > > >
> > > >
> > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> supun06@gmail.com
> > >
> > > >wrote:
> > > >
> > > >> Hi Shameera,
> > > >>
> > > >> Why you prefer JavaScript over a language like Python?
> > > >>
> > > >> Thanks,
> > > >> Supun..
> > > >>
> > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > > >> shameerainfo@gmail.com> wrote:
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> First of all thanks everyone for giving valuable inputs. After
> doing
> > > >>>some
> > > >>> background search and talking to different people in the University
> > > >>>who has
> > > >>> used different workflow languages, I myself convinced that
> > introducing
> > > >>>an
> > > >>> another workflow language is not what actually they need. By
> changing
> > > >>> exiting workflow language to another will not solve problems. What
> > they
> > > >>> asking is a easy way to construct the workflows. Indirectly what
> they
> > > >>> asking for a sort of API which they can use to generate the
> workflows
> > > >>>and
> > > >>> run it. Correct me if i am wrong here.
> > > >>>
> > > >>> As most of above replies depict,  if we can get a simple API, as an
> > > >>> example, for a web based application, JavaScript API would be a
> good
> > > >>> solution, and probably JSON would be a good candidate for language,
> > > >>>instead
> > > >>> of XML.
> > > >>>
> > > >>> Airavata community already have started to implement web base GUI.
> > > >>>Hence
> > > >>> introducing a JSON base JavaScript API would be great help. WDYT?
> > > >>>
> > > >>> Thanks,
> > > >>> Shameera.
> > > >>>
> > > >>>
> > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> > > >>> alekny7@gmail.com> wrote:
> > > >>>
> > > >>>> Hi,
> > > >>>>
> > > >>>> it is not dataflow instead focused on orchestrating REST services
> > but
> > > >>>> you may find it useful datapoint - we created worfklow service
> that
> > > >>>>uses
> > > >>>> natively JavaScript and JSON to describe what happens during
> > workflow
> > > >>>> execution:
> > > >>>>
> > > https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> > > >>>>
> > > >>>> HTH,
> > > >>>>
> > > >>>> Alek
> > > >>>>
> > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> > > >>>>wrote:
> > > >>>>
> > > >>>>> Hi Chris,
> > > >>>>>
> > > >>>>> Great to hear OODT community will be interested in adopting a
> JSON
> > > >>>>> based workflow language and potentially a web based composer as
> > well.
> > > >>>>> Airavata previously had BPEL support initially through a home
> grown
> > > >>>>> implementation [1] by Alek Slominski and later through Apache ODE
> > > >>>>>[2]. Also
> > > >>>>> a white paper [3] by Alek on this topic is an interesting read.
> > > >>>>>
> > > >>>>> I am of the same opinion that we should adopt something more
> modern
> > > >>>>>as
> > > >>>>> the challenges from scientific workflows seems to be converging
> > with
> > > >>>>>the
> > > >>>>> data flow patterns in business workflows.
> > > >>>>>
> > > >>>>> It will be great if we can all compile a list of potential
> > candidates
> > > >>>>> and hack them through.
> > > >>>>>
> > > >>>>> Suresh
> > > >>>>> [1] -
> > > >>>>>
> > > >>>>>
> > > http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > > >>>>> [2] -
> > > >>>>>
> > > >>>>>
> > > http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> > > >>>>>time_for_eScience_workflows
> > > >>>>> [3] -
> > > >>>>>
> > > >>>>>
> > > http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> > > >>>>>6.pdf
> > > >>>>>
> > > >>>>>
> > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> > > >>>>>
> > > >>>>> > Hi Guys,
> > > >>>>> >
> > > >>>>> > I've been interested in this too - we don't per have a specific
> > > >>>>> > OODT workflow language, but we specific workflows using XML,
> and
> > > >>>>> > other configuration (we are also thinking of moving to JSON for
> > > >>>>> > this).
> > > >>>>> >
> > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
> complex
> > > >>>>> > to me.
> > > >>>>> >
> > > >>>>> > I wonder at the end of the day if we should adopt something
> more
> > > >>>>> > modern like PIG or some other data flow type of language (PIG
> > > >>>>> > is really neat).
> > > >>>>> >
> > > >>>>> > Cheers,
> > > >>>>> > Chris
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>>>> > Chris Mattmann, Ph.D.
> > > >>>>> > Chief Architect
> > > >>>>> > Instrument Software and Science Data Systems Section (398)
> > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > >>>>> > Office: 168-519, Mailstop: 168-527
> > > >>>>> > Email: chris.a.mattmann@nasa.gov
> > > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
> > > >>>>> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>>>> > Adjunct Associate Professor, Computer Science Department
> > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> > > >>>>> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> > -----Original Message-----
> > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> > > >>>>> > Reply-To: "architecture@airavata.apache.org"
> > > >>>>> > <ar...@airavata.apache.org>
> > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> > > >>>>> > To: "architecture@airavata.apache.org" <
> > > >>>>> architecture@airavata.apache.org>,
> > > >>>>> > dev <de...@airavata.apache.org>
> > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> > > >>>>>Airavata.
> > > >>>>> >
> > > >>>>> >> Hi All,
> > > >>>>> >>
> > > >>>>> >> As we all know Airavata has its own workflow language call
> XWF.
> > > >>>>>When
> > > >>>>> XWF
> > > >>>>> >> was introduced, main focus points are interoperability and
> > > >>>>> convertibility.
> > > >>>>> >> But with years of experience it is convinced that above
> > > >>>>>requirements
> > > >>>>> are
> > > >>>>> >> not really useful when we come to real world use cases. And
> XWF
> > is
> > > >>>>> XML
> > > >>>>> >> based bulky language where we attache WSDLs and Workflow image
> > it
> > > >>>>> self.
> > > >>>>> >> But
> > > >>>>> >> with the recent changes WSDL part is being removed from XWF.
> > > >>>>> >>
> > > >>>>> >> It is worth to evaluate handy Scientific workflow languages in
> > > >>>>> industry
> > > >>>>> >> and
> > > >>>>> >> find out pros and cons, at the end of this evaluation we need
> to
> > > >>>>> come up
> > > >>>>> >> with idea how we should improve Airavata workflow language,
> > either
> > > >>>>> we can
> > > >>>>> >> improve existing XWF language, totally change to a new
> language
> > > >>>>> available
> > > >>>>> >> in industry or write a new light weight language. Basic
> > > >>>>>requirements
> > > >>>>> that
> > > >>>>> >> we expect from new improvement are, high usability, flexible,
> > > >>>>>light
> > > >>>>> weight
> > > >>>>> >> and real time monitoring support. As you can see above
> > > >>>>>requirements
> > > >>>>> are
> > > >>>>> >> not
> > > >>>>> >> direct comes with workflow languages but we need workflow
> > language
> > > >>>>> which
> > > >>>>> >> help to support above requirements.
> > > >>>>> >>
> > > >>>>> >> After reading few papers and googling, initially i have come
> up
> > > >>>>>with
> > > >>>>> >> following three existing languages,
> > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> > > >>>>> >> 2. WS-BPEL
> > > >>>>> >> 3. SIDL
> > > >>>>> >> <
> > http://computation.llnl.gov/casc/components/index.html#page=home
> > > >
> > > >>>>> >>
> > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> > > >>>>> Radical-SAGA
> > > >>>>> >> also uses slightly modified version of SIDL. Other than above
> > > >>>>>three
> > > >>>>> >> languages we can come up with simple workflow language base on
> > > >>>>> json(or
> > > >>>>> >> yaml) which support all our requirements for some extends.
> > > >>>>> >>
> > > >>>>> >> It would be grate if I can get more input regarding the
> $Subject
> > > >>>>> form the
> > > >>>>> >> airavata community. You all are more than welcome to provide
> any
> > > >>>>> type of
> > > >>>>> >> suggestions.
> > > >>>>> >>
> > > >>>>> >> Thanks,
> > > >>>>> >> Shameera.
> > > >>>>> >>
> > > >>>>> >> 
> > > >>>>> >>
> > > >>>>> >> --
> > > >>>>> >> Best Regards,
> > > >>>>> >> Shameera Rathnayaka.
> > > >>>>> >>
> > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> The best way to predict the future is to invent it - Alan Kay
> > > >>>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Best Regards,
> > > >>> Shameera Rathnayaka.
> > > >>>
> > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > >>> Blog : http://shameerarathnayaka.blogspot.com/
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Supun Kamburugamuva
> > > >> Member, Apache Software Foundation; http://www.apache.org
> > > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > > >> Blog: http://supunk.blogspot.com
> > > >>
> > > >>
> > > >
> > > >
> > > >--
> > > >Supun Kamburugamuva
> > > >Member, Apache Software Foundation; http://www.apache.org
> > > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > > >Blog: http://supunk.blogspot.com
> > >
> > >
> > >
> >
> >
> > --
> > Best Regards,
> > Shameera Rathnayaka.
> >
> > email: shameera AT apache.org , shameerainfo AT gmail.com
> > Blog : http://shameerarathnayaka.blogspot.com/
> >
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>



-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Shameera Rathnayaka <sh...@gmail.com>.

Hi Supun,

I think we were in two context, because I as suggesting a way to serialize
and deserialize the workflow description while you are suggesting to
implement
some kind of workflow interpreter using Python, where Python client can
send
thrift calls to Airavata server to run the application. I can see with your
suggested
approach we can control the workflow execution process from client side
which
make it easy to implement workflow debugger.As you mentioned this is a
major change
to Airavata. So we should neatly think as this will change our existing
architecture.

Still if someone need to use different language java, php, JS etc ... to
run the same
workflow which generated by Python, we need a language independent workflow
description.
My initial question was what is the best language for this?. But as I have
explained in
one of my previous reply, It is not matter what language we used Either we
can use
XML or JSON to write this description, what matters is how easy to generate
workflow with the provided API. Hence it would be great to have set of neat
APIs in
different languages.

Thanks,
Shameera.


On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:

> Hi Shameera,
>
> Using python is a radical approach for workflow I think. But I believe this
> is a very beautiful and new approach.Here is a possible scenario for
> implementation and running using a Python based library.
>
> The Python library facilitates the creation of Applications and submitting
> them to Airavata for execution. The underlying functionality is exactly
> similar to what Java clients provides.The only difference is that, Python
> library should have a more fluent API than Java for easy creation of
> workflows. We can generate the Python clients that talk to Airavata server
> using Thrift.
>
> Here is an example off the top of my head to a Python script created by
> user for a Workflow. This is a very crude example and we need to come up
> with a much better API if we are going to go along this path. First we need
> to write a Python script that can execute a workflow using Airavata.
>
> import airavata
>
> host = Host("localhost", ....)
> app1 = Application(host, ....)
> app2 = Application(host, ....)
>
> # we will connect these applications as a workflow using some topology
> builder or other constructs
>
> wb = WorkFlowBuilder()
> wb.setApp("name1", app1)
> # you can do a simple output transformation here etc
>
> # connects the input of app1 to app2 etc
> wb.setApp("name2", app2).connectInput("name1")
>
> wb.submit()
>
> Now we can load this Python script from XBaya. When XBaya loads this script
> the Python script can output an XML configuration of the topology, XBaya
> can render. There are other ways like directly executing the Python script
> from command line and connecting XBaya indirectly as well. Now you can run
> the workflow from XBaya. Running the Workflow means just executing the
> Python script.
>
> XBaya gets the notifications through messaging and update the UI
> accordingly.
>
> The users need to write the Python script by hand. XBaya cannot create the
> script. Because workflow language is an actual python program the benefits
> are immense. For example user can do workflow steering in the workflow
> itself by subscribing to messages from Airavata.
>
> Thanks,
> Supun..
>
>
>
>
>
>
>
>
> On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
> > Hi Supun,
> >
> > I meant to say JS is a well-known client side scripting language i have
> > messed scripting part. Even we use Python, ultimately we should convert
> > this to java model in server side, somehow we need to serialized python
> > representation to the language which java can parse and generate that
> > model. In this case we need to parse python script in java isn't it? I am
> > not exactly clear how you suggesting to use python here. More details on
> > how end system works if we use Python would be great help to clearly
> > understand your points.
> >
> > Thanks,
> > Shameera.
> >
> > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> chris.mattmann@gmail.com>
> > wrote:
> >
> > > Have you guys considered using JCC [1] as a means
> > > to expose the workflow API currently in Java as a
> > > Python API?
> > >
> > > We are exploring its use in OODT, and we have already
> > > created a Tika [2] JCC-based python API.
> > >
> > > Cheers,
> > > Chris
> > >
> > > [1] http://lucene.apache.org/pylucene/jcc/
> > > [2] http://github.com/chrismattmann/tika-python/
> > >
> > > ------------------------
> > > Chris Mattmann
> > > chris.mattmann@gmail.com
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Supun Kamburugamuva <su...@gmail.com>
> > > Reply-To: <de...@oodt.apache.org>
> > > Date: Thursday, October 16, 2014 at 3:43 PM
> > > To: dev <de...@airavata.apache.org>
> > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> > > <sm...@apache.org>, "architecture@airavata.apache.org"
> > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> > > <de...@oodt.apache.org>
> > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> Airavata.
> > >
> > > >Once we had an offline discussion about the Airavata Workflow language
> > > >(with Milinda, Saliya and Shameera). In that discussion one thing came
> > out
> > > >was why we need to invent a different language when a simple library
> > like
> > > >Python will full fill of Airavata requirements.
> > > >
> > > >There are many benefits in using a Python library as the API for
> > > >controlling Airavata workflows.
> > > >
> > > >1. It is a library, gives the ultimate control over the execution and
> it
> > > >can be simpler than any domain specific language that we can come with
> > > >like
> > > >XML, JSON etc
> > > >2. Most people use python and can learn it easily than any Domain
> > specific
> > > >language
> > > >3. You can easily create a Python library for Airavata because all the
> > > >APIs
> > > >are thrift based.
> > > >4. If you design the constructs correctly you can plug an XBaya.
> > > >
> > > >Any thoughts?
> > > >
> > > >Thanks,
> > > >Supun..
> > > >
> > > >
> > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> supun06@gmail.com
> > >
> > > >wrote:
> > > >
> > > >> Hi Shameera,
> > > >>
> > > >> Why you prefer JavaScript over a language like Python?
> > > >>
> > > >> Thanks,
> > > >> Supun..
> > > >>
> > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > > >> shameerainfo@gmail.com> wrote:
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> First of all thanks everyone for giving valuable inputs. After
> doing
> > > >>>some
> > > >>> background search and talking to different people in the University
> > > >>>who has
> > > >>> used different workflow languages, I myself convinced that
> > introducing
> > > >>>an
> > > >>> another workflow language is not what actually they need. By
> changing
> > > >>> exiting workflow language to another will not solve problems. What
> > they
> > > >>> asking is a easy way to construct the workflows. Indirectly what
> they
> > > >>> asking for a sort of API which they can use to generate the
> workflows
> > > >>>and
> > > >>> run it. Correct me if i am wrong here.
> > > >>>
> > > >>> As most of above replies depict,  if we can get a simple API, as an
> > > >>> example, for a web based application, JavaScript API would be a
> good
> > > >>> solution, and probably JSON would be a good candidate for language,
> > > >>>instead
> > > >>> of XML.
> > > >>>
> > > >>> Airavata community already have started to implement web base GUI.
> > > >>>Hence
> > > >>> introducing a JSON base JavaScript API would be great help. WDYT?
> > > >>>
> > > >>> Thanks,
> > > >>> Shameera.
> > > >>>
> > > >>>
> > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> > > >>> alekny7@gmail.com> wrote:
> > > >>>
> > > >>>> Hi,
> > > >>>>
> > > >>>> it is not dataflow instead focused on orchestrating REST services
> > but
> > > >>>> you may find it useful datapoint - we created worfklow service
> that
> > > >>>>uses
> > > >>>> natively JavaScript and JSON to describe what happens during
> > workflow
> > > >>>> execution:
> > > >>>>
> > > https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> > > >>>>
> > > >>>> HTH,
> > > >>>>
> > > >>>> Alek
> > > >>>>
> > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> > > >>>>wrote:
> > > >>>>
> > > >>>>> Hi Chris,
> > > >>>>>
> > > >>>>> Great to hear OODT community will be interested in adopting a
> JSON
> > > >>>>> based workflow language and potentially a web based composer as
> > well.
> > > >>>>> Airavata previously had BPEL support initially through a home
> grown
> > > >>>>> implementation [1] by Alek Slominski and later through Apache ODE
> > > >>>>>[2]. Also
> > > >>>>> a white paper [3] by Alek on this topic is an interesting read.
> > > >>>>>
> > > >>>>> I am of the same opinion that we should adopt something more
> modern
> > > >>>>>as
> > > >>>>> the challenges from scientific workflows seems to be converging
> > with
> > > >>>>>the
> > > >>>>> data flow patterns in business workflows.
> > > >>>>>
> > > >>>>> It will be great if we can all compile a list of potential
> > candidates
> > > >>>>> and hack them through.
> > > >>>>>
> > > >>>>> Suresh
> > > >>>>> [1] -
> > > >>>>>
> > > >>>>>
> > > http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > > >>>>> [2] -
> > > >>>>>
> > > >>>>>
> > > http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> > > >>>>>time_for_eScience_workflows
> > > >>>>> [3] -
> > > >>>>>
> > > >>>>>
> > > http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> > > >>>>>6.pdf
> > > >>>>>
> > > >>>>>
> > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> > > >>>>>
> > > >>>>> > Hi Guys,
> > > >>>>> >
> > > >>>>> > I've been interested in this too - we don't per have a specific
> > > >>>>> > OODT workflow language, but we specific workflows using XML,
> and
> > > >>>>> > other configuration (we are also thinking of moving to JSON for
> > > >>>>> > this).
> > > >>>>> >
> > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
> complex
> > > >>>>> > to me.
> > > >>>>> >
> > > >>>>> > I wonder at the end of the day if we should adopt something
> more
> > > >>>>> > modern like PIG or some other data flow type of language (PIG
> > > >>>>> > is really neat).
> > > >>>>> >
> > > >>>>> > Cheers,
> > > >>>>> > Chris
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>>>> > Chris Mattmann, Ph.D.
> > > >>>>> > Chief Architect
> > > >>>>> > Instrument Software and Science Data Systems Section (398)
> > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > >>>>> > Office: 168-519, Mailstop: 168-527
> > > >>>>> > Email: chris.a.mattmann@nasa.gov
> > > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
> > > >>>>> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>>>> > Adjunct Associate Professor, Computer Science Department
> > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> > > >>>>> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> > -----Original Message-----
> > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> > > >>>>> > Reply-To: "architecture@airavata.apache.org"
> > > >>>>> > <ar...@airavata.apache.org>
> > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> > > >>>>> > To: "architecture@airavata.apache.org" <
> > > >>>>> architecture@airavata.apache.org>,
> > > >>>>> > dev <de...@airavata.apache.org>
> > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> > > >>>>>Airavata.
> > > >>>>> >
> > > >>>>> >> Hi All,
> > > >>>>> >>
> > > >>>>> >> As we all know Airavata has its own workflow language call
> XWF.
> > > >>>>>When
> > > >>>>> XWF
> > > >>>>> >> was introduced, main focus points are interoperability and
> > > >>>>> convertibility.
> > > >>>>> >> But with years of experience it is convinced that above
> > > >>>>>requirements
> > > >>>>> are
> > > >>>>> >> not really useful when we come to real world use cases. And
> XWF
> > is
> > > >>>>> XML
> > > >>>>> >> based bulky language where we attache WSDLs and Workflow image
> > it
> > > >>>>> self.
> > > >>>>> >> But
> > > >>>>> >> with the recent changes WSDL part is being removed from XWF.
> > > >>>>> >>
> > > >>>>> >> It is worth to evaluate handy Scientific workflow languages in
> > > >>>>> industry
> > > >>>>> >> and
> > > >>>>> >> find out pros and cons, at the end of this evaluation we need
> to
> > > >>>>> come up
> > > >>>>> >> with idea how we should improve Airavata workflow language,
> > either
> > > >>>>> we can
> > > >>>>> >> improve existing XWF language, totally change to a new
> language
> > > >>>>> available
> > > >>>>> >> in industry or write a new light weight language. Basic
> > > >>>>>requirements
> > > >>>>> that
> > > >>>>> >> we expect from new improvement are, high usability, flexible,
> > > >>>>>light
> > > >>>>> weight
> > > >>>>> >> and real time monitoring support. As you can see above
> > > >>>>>requirements
> > > >>>>> are
> > > >>>>> >> not
> > > >>>>> >> direct comes with workflow languages but we need workflow
> > language
> > > >>>>> which
> > > >>>>> >> help to support above requirements.
> > > >>>>> >>
> > > >>>>> >> After reading few papers and googling, initially i have come
> up
> > > >>>>>with
> > > >>>>> >> following three existing languages,
> > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> > > >>>>> >> 2. WS-BPEL
> > > >>>>> >> 3. SIDL
> > > >>>>> >> <
> > http://computation.llnl.gov/casc/components/index.html#page=home
> > > >
> > > >>>>> >>
> > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> > > >>>>> Radical-SAGA
> > > >>>>> >> also uses slightly modified version of SIDL. Other than above
> > > >>>>>three
> > > >>>>> >> languages we can come up with simple workflow language base on
> > > >>>>> json(or
> > > >>>>> >> yaml) which support all our requirements for some extends.
> > > >>>>> >>
> > > >>>>> >> It would be grate if I can get more input regarding the
> $Subject
> > > >>>>> form the
> > > >>>>> >> airavata community. You all are more than welcome to provide
> any
> > > >>>>> type of
> > > >>>>> >> suggestions.
> > > >>>>> >>
> > > >>>>> >> Thanks,
> > > >>>>> >> Shameera.
> > > >>>>> >>
> > > >>>>> >> 
> > > >>>>> >>
> > > >>>>> >> --
> > > >>>>> >> Best Regards,
> > > >>>>> >> Shameera Rathnayaka.
> > > >>>>> >>
> > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> The best way to predict the future is to invent it - Alan Kay
> > > >>>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Best Regards,
> > > >>> Shameera Rathnayaka.
> > > >>>
> > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > >>> Blog : http://shameerarathnayaka.blogspot.com/
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Supun Kamburugamuva
> > > >> Member, Apache Software Foundation; http://www.apache.org
> > > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > > >> Blog: http://supunk.blogspot.com
> > > >>
> > > >>
> > > >
> > > >
> > > >--
> > > >Supun Kamburugamuva
> > > >Member, Apache Software Foundation; http://www.apache.org
> > > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > > >Blog: http://supunk.blogspot.com
> > >
> > >
> > >
> >
> >
> > --
> > Best Regards,
> > Shameera Rathnayaka.
> >
> > email: shameera AT apache.org , shameerainfo AT gmail.com
> > Blog : http://shameerarathnayaka.blogspot.com/
> >
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>



-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Shameera Rathnayaka <sh...@gmail.com>.

Hi Supun,

I think we were in two context, because I as suggesting a way to serialize
and deserialize the workflow description while you are suggesting to
implement
some kind of workflow interpreter using Python, where Python client can
send
thrift calls to Airavata server to run the application. I can see with your
suggested
approach we can control the workflow execution process from client side
which
make it easy to implement workflow debugger.As you mentioned this is a
major change
to Airavata. So we should neatly think as this will change our existing
architecture.

Still if someone need to use different language java, php, JS etc ... to
run the same
workflow which generated by Python, we need a language independent workflow
description.
My initial question was what is the best language for this?. But as I have
explained in
one of my previous reply, It is not matter what language we used Either we
can use
XML or JSON to write this description, what matters is how easy to generate
workflow with the provided API. Hence it would be great to have set of neat
APIs in
different languages.

Thanks,
Shameera.


On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:

> Hi Shameera,
>
> Using python is a radical approach for workflow I think. But I believe this
> is a very beautiful and new approach.Here is a possible scenario for
> implementation and running using a Python based library.
>
> The Python library facilitates the creation of Applications and submitting
> them to Airavata for execution. The underlying functionality is exactly
> similar to what Java clients provides.The only difference is that, Python
> library should have a more fluent API than Java for easy creation of
> workflows. We can generate the Python clients that talk to Airavata server
> using Thrift.
>
> Here is an example off the top of my head to a Python script created by
> user for a Workflow. This is a very crude example and we need to come up
> with a much better API if we are going to go along this path. First we need
> to write a Python script that can execute a workflow using Airavata.
>
> import airavata
>
> host = Host("localhost", ....)
> app1 = Application(host, ....)
> app2 = Application(host, ....)
>
> # we will connect these applications as a workflow using some topology
> builder or other constructs
>
> wb = WorkFlowBuilder()
> wb.setApp("name1", app1)
> # you can do a simple output transformation here etc
>
> # connects the input of app1 to app2 etc
> wb.setApp("name2", app2).connectInput("name1")
>
> wb.submit()
>
> Now we can load this Python script from XBaya. When XBaya loads this script
> the Python script can output an XML configuration of the topology, XBaya
> can render. There are other ways like directly executing the Python script
> from command line and connecting XBaya indirectly as well. Now you can run
> the workflow from XBaya. Running the Workflow means just executing the
> Python script.
>
> XBaya gets the notifications through messaging and update the UI
> accordingly.
>
> The users need to write the Python script by hand. XBaya cannot create the
> script. Because workflow language is an actual python program the benefits
> are immense. For example user can do workflow steering in the workflow
> itself by subscribing to messages from Airavata.
>
> Thanks,
> Supun..
>
>
>
>
>
>
>
>
> On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
> > Hi Supun,
> >
> > I meant to say JS is a well-known client side scripting language i have
> > messed scripting part. Even we use Python, ultimately we should convert
> > this to java model in server side, somehow we need to serialized python
> > representation to the language which java can parse and generate that
> > model. In this case we need to parse python script in java isn't it? I am
> > not exactly clear how you suggesting to use python here. More details on
> > how end system works if we use Python would be great help to clearly
> > understand your points.
> >
> > Thanks,
> > Shameera.
> >
> > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> chris.mattmann@gmail.com>
> > wrote:
> >
> > > Have you guys considered using JCC [1] as a means
> > > to expose the workflow API currently in Java as a
> > > Python API?
> > >
> > > We are exploring its use in OODT, and we have already
> > > created a Tika [2] JCC-based python API.
> > >
> > > Cheers,
> > > Chris
> > >
> > > [1] http://lucene.apache.org/pylucene/jcc/
> > > [2] http://github.com/chrismattmann/tika-python/
> > >
> > > ------------------------
> > > Chris Mattmann
> > > chris.mattmann@gmail.com
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Supun Kamburugamuva <su...@gmail.com>
> > > Reply-To: <de...@oodt.apache.org>
> > > Date: Thursday, October 16, 2014 at 3:43 PM
> > > To: dev <de...@airavata.apache.org>
> > > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> > > <sm...@apache.org>, "architecture@airavata.apache.org"
> > > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> > > <de...@oodt.apache.org>
> > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> Airavata.
> > >
> > > >Once we had an offline discussion about the Airavata Workflow language
> > > >(with Milinda, Saliya and Shameera). In that discussion one thing came
> > out
> > > >was why we need to invent a different language when a simple library
> > like
> > > >Python will full fill of Airavata requirements.
> > > >
> > > >There are many benefits in using a Python library as the API for
> > > >controlling Airavata workflows.
> > > >
> > > >1. It is a library, gives the ultimate control over the execution and
> it
> > > >can be simpler than any domain specific language that we can come with
> > > >like
> > > >XML, JSON etc
> > > >2. Most people use python and can learn it easily than any Domain
> > specific
> > > >language
> > > >3. You can easily create a Python library for Airavata because all the
> > > >APIs
> > > >are thrift based.
> > > >4. If you design the constructs correctly you can plug an XBaya.
> > > >
> > > >Any thoughts?
> > > >
> > > >Thanks,
> > > >Supun..
> > > >
> > > >
> > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> supun06@gmail.com
> > >
> > > >wrote:
> > > >
> > > >> Hi Shameera,
> > > >>
> > > >> Why you prefer JavaScript over a language like Python?
> > > >>
> > > >> Thanks,
> > > >> Supun..
> > > >>
> > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > > >> shameerainfo@gmail.com> wrote:
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> First of all thanks everyone for giving valuable inputs. After
> doing
> > > >>>some
> > > >>> background search and talking to different people in the University
> > > >>>who has
> > > >>> used different workflow languages, I myself convinced that
> > introducing
> > > >>>an
> > > >>> another workflow language is not what actually they need. By
> changing
> > > >>> exiting workflow language to another will not solve problems. What
> > they
> > > >>> asking is a easy way to construct the workflows. Indirectly what
> they
> > > >>> asking for a sort of API which they can use to generate the
> workflows
> > > >>>and
> > > >>> run it. Correct me if i am wrong here.
> > > >>>
> > > >>> As most of above replies depict,  if we can get a simple API, as an
> > > >>> example, for a web based application, JavaScript API would be a
> good
> > > >>> solution, and probably JSON would be a good candidate for language,
> > > >>>instead
> > > >>> of XML.
> > > >>>
> > > >>> Airavata community already have started to implement web base GUI.
> > > >>>Hence
> > > >>> introducing a JSON base JavaScript API would be great help. WDYT?
> > > >>>
> > > >>> Thanks,
> > > >>> Shameera.
> > > >>>
> > > >>>
> > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> > > >>> alekny7@gmail.com> wrote:
> > > >>>
> > > >>>> Hi,
> > > >>>>
> > > >>>> it is not dataflow instead focused on orchestrating REST services
> > but
> > > >>>> you may find it useful datapoint - we created worfklow service
> that
> > > >>>>uses
> > > >>>> natively JavaScript and JSON to describe what happens during
> > workflow
> > > >>>> execution:
> > > >>>>
> > > https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> > > >>>>
> > > >>>> HTH,
> > > >>>>
> > > >>>> Alek
> > > >>>>
> > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> > > >>>>wrote:
> > > >>>>
> > > >>>>> Hi Chris,
> > > >>>>>
> > > >>>>> Great to hear OODT community will be interested in adopting a
> JSON
> > > >>>>> based workflow language and potentially a web based composer as
> > well.
> > > >>>>> Airavata previously had BPEL support initially through a home
> grown
> > > >>>>> implementation [1] by Alek Slominski and later through Apache ODE
> > > >>>>>[2]. Also
> > > >>>>> a white paper [3] by Alek on this topic is an interesting read.
> > > >>>>>
> > > >>>>> I am of the same opinion that we should adopt something more
> modern
> > > >>>>>as
> > > >>>>> the challenges from scientific workflows seems to be converging
> > with
> > > >>>>>the
> > > >>>>> data flow patterns in business workflows.
> > > >>>>>
> > > >>>>> It will be great if we can all compile a list of potential
> > candidates
> > > >>>>> and hack them through.
> > > >>>>>
> > > >>>>> Suresh
> > > >>>>> [1] -
> > > >>>>>
> > > >>>>>
> > > http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > > >>>>> [2] -
> > > >>>>>
> > > >>>>>
> > > http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> > > >>>>>time_for_eScience_workflows
> > > >>>>> [3] -
> > > >>>>>
> > > >>>>>
> > > http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> > > >>>>>6.pdf
> > > >>>>>
> > > >>>>>
> > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> > > >>>>>
> > > >>>>> > Hi Guys,
> > > >>>>> >
> > > >>>>> > I've been interested in this too - we don't per have a specific
> > > >>>>> > OODT workflow language, but we specific workflows using XML,
> and
> > > >>>>> > other configuration (we are also thinking of moving to JSON for
> > > >>>>> > this).
> > > >>>>> >
> > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
> complex
> > > >>>>> > to me.
> > > >>>>> >
> > > >>>>> > I wonder at the end of the day if we should adopt something
> more
> > > >>>>> > modern like PIG or some other data flow type of language (PIG
> > > >>>>> > is really neat).
> > > >>>>> >
> > > >>>>> > Cheers,
> > > >>>>> > Chris
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>>>> > Chris Mattmann, Ph.D.
> > > >>>>> > Chief Architect
> > > >>>>> > Instrument Software and Science Data Systems Section (398)
> > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > >>>>> > Office: 168-519, Mailstop: 168-527
> > > >>>>> > Email: chris.a.mattmann@nasa.gov
> > > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
> > > >>>>> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>>>> > Adjunct Associate Professor, Computer Science Department
> > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> > > >>>>> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> >
> > > >>>>> > -----Original Message-----
> > > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> > > >>>>> > Reply-To: "architecture@airavata.apache.org"
> > > >>>>> > <ar...@airavata.apache.org>
> > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> > > >>>>> > To: "architecture@airavata.apache.org" <
> > > >>>>> architecture@airavata.apache.org>,
> > > >>>>> > dev <de...@airavata.apache.org>
> > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> > > >>>>>Airavata.
> > > >>>>> >
> > > >>>>> >> Hi All,
> > > >>>>> >>
> > > >>>>> >> As we all know Airavata has its own workflow language call
> XWF.
> > > >>>>>When
> > > >>>>> XWF
> > > >>>>> >> was introduced, main focus points are interoperability and
> > > >>>>> convertibility.
> > > >>>>> >> But with years of experience it is convinced that above
> > > >>>>>requirements
> > > >>>>> are
> > > >>>>> >> not really useful when we come to real world use cases. And
> XWF
> > is
> > > >>>>> XML
> > > >>>>> >> based bulky language where we attache WSDLs and Workflow image
> > it
> > > >>>>> self.
> > > >>>>> >> But
> > > >>>>> >> with the recent changes WSDL part is being removed from XWF.
> > > >>>>> >>
> > > >>>>> >> It is worth to evaluate handy Scientific workflow languages in
> > > >>>>> industry
> > > >>>>> >> and
> > > >>>>> >> find out pros and cons, at the end of this evaluation we need
> to
> > > >>>>> come up
> > > >>>>> >> with idea how we should improve Airavata workflow language,
> > either
> > > >>>>> we can
> > > >>>>> >> improve existing XWF language, totally change to a new
> language
> > > >>>>> available
> > > >>>>> >> in industry or write a new light weight language. Basic
> > > >>>>>requirements
> > > >>>>> that
> > > >>>>> >> we expect from new improvement are, high usability, flexible,
> > > >>>>>light
> > > >>>>> weight
> > > >>>>> >> and real time monitoring support. As you can see above
> > > >>>>>requirements
> > > >>>>> are
> > > >>>>> >> not
> > > >>>>> >> direct comes with workflow languages but we need workflow
> > language
> > > >>>>> which
> > > >>>>> >> help to support above requirements.
> > > >>>>> >>
> > > >>>>> >> After reading few papers and googling, initially i have come
> up
> > > >>>>>with
> > > >>>>> >> following three existing languages,
> > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> > > >>>>> >> 2. WS-BPEL
> > > >>>>> >> 3. SIDL
> > > >>>>> >> <
> > http://computation.llnl.gov/casc/components/index.html#page=home
> > > >
> > > >>>>> >>
> > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> > > >>>>> Radical-SAGA
> > > >>>>> >> also uses slightly modified version of SIDL. Other than above
> > > >>>>>three
> > > >>>>> >> languages we can come up with simple workflow language base on
> > > >>>>> json(or
> > > >>>>> >> yaml) which support all our requirements for some extends.
> > > >>>>> >>
> > > >>>>> >> It would be grate if I can get more input regarding the
> $Subject
> > > >>>>> form the
> > > >>>>> >> airavata community. You all are more than welcome to provide
> any
> > > >>>>> type of
> > > >>>>> >> suggestions.
> > > >>>>> >>
> > > >>>>> >> Thanks,
> > > >>>>> >> Shameera.
> > > >>>>> >>
> > > >>>>> >> 
> > > >>>>> >>
> > > >>>>> >> --
> > > >>>>> >> Best Regards,
> > > >>>>> >> Shameera Rathnayaka.
> > > >>>>> >>
> > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> The best way to predict the future is to invent it - Alan Kay
> > > >>>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Best Regards,
> > > >>> Shameera Rathnayaka.
> > > >>>
> > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > >>> Blog : http://shameerarathnayaka.blogspot.com/
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Supun Kamburugamuva
> > > >> Member, Apache Software Foundation; http://www.apache.org
> > > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > > >> Blog: http://supunk.blogspot.com
> > > >>
> > > >>
> > > >
> > > >
> > > >--
> > > >Supun Kamburugamuva
> > > >Member, Apache Software Foundation; http://www.apache.org
> > > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > > >Blog: http://supunk.blogspot.com
> > >
> > >
> > >
> >
> >
> > --
> > Best Regards,
> > Shameera Rathnayaka.
> >
> > email: shameera AT apache.org , shameerainfo AT gmail.com
> > Blog : http://shameerarathnayaka.blogspot.com/
> >
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>



-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Supun Kamburugamuva <su...@gmail.com>.

Hi Shameera,

Using python is a radical approach for workflow I think. But I believe this
is a very beautiful and new approach.Here is a possible scenario for
implementation and running using a Python based library.

The Python library facilitates the creation of Applications and submitting
them to Airavata for execution. The underlying functionality is exactly
similar to what Java clients provides.The only difference is that, Python
library should have a more fluent API than Java for easy creation of
workflows. We can generate the Python clients that talk to Airavata server
using Thrift.

Here is an example off the top of my head to a Python script created by
user for a Workflow. This is a very crude example and we need to come up
with a much better API if we are going to go along this path. First we need
to write a Python script that can execute a workflow using Airavata.

import airavata

host = Host("localhost", ....)
app1 = Application(host, ....)
app2 = Application(host, ....)

# we will connect these applications as a workflow using some topology
builder or other constructs

wb = WorkFlowBuilder()
wb.setApp("name1", app1)
# you can do a simple output transformation here etc

# connects the input of app1 to app2 etc
wb.setApp("name2", app2).connectInput("name1")

wb.submit()

Now we can load this Python script from XBaya. When XBaya loads this script
the Python script can output an XML configuration of the topology, XBaya
can render. There are other ways like directly executing the Python script
from command line and connecting XBaya indirectly as well. Now you can run
the workflow from XBaya. Running the Workflow means just executing the
Python script.

XBaya gets the notifications through messaging and update the UI
accordingly.

The users need to write the Python script by hand. XBaya cannot create the
script. Because workflow language is an actual python program the benefits
are immense. For example user can do workflow steering in the workflow
itself by subscribing to messages from Airavata.

Thanks,
Supun..








On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:

> Hi Supun,
>
> I meant to say JS is a well-known client side scripting language i have
> messed scripting part. Even we use Python, ultimately we should convert
> this to java model in server side, somehow we need to serialized python
> representation to the language which java can parse and generate that
> model. In this case we need to parse python script in java isn't it? I am
> not exactly clear how you suggesting to use python here. More details on
> how end system works if we use Python would be great help to clearly
> understand your points.
>
> Thanks,
> Shameera.
>
> On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <ch...@gmail.com>
> wrote:
>
> > Have you guys considered using JCC [1] as a means
> > to expose the workflow API currently in Java as a
> > Python API?
> >
> > We are exploring its use in OODT, and we have already
> > created a Tika [2] JCC-based python API.
> >
> > Cheers,
> > Chris
> >
> > [1] http://lucene.apache.org/pylucene/jcc/
> > [2] http://github.com/chrismattmann/tika-python/
> >
> > ------------------------
> > Chris Mattmann
> > chris.mattmann@gmail.com
> >
> >
> >
> >
> > -----Original Message-----
> > From: Supun Kamburugamuva <su...@gmail.com>
> > Reply-To: <de...@oodt.apache.org>
> > Date: Thursday, October 16, 2014 at 3:43 PM
> > To: dev <de...@airavata.apache.org>
> > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> > <sm...@apache.org>, "architecture@airavata.apache.org"
> > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> > <de...@oodt.apache.org>
> > Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
> >
> > >Once we had an offline discussion about the Airavata Workflow language
> > >(with Milinda, Saliya and Shameera). In that discussion one thing came
> out
> > >was why we need to invent a different language when a simple library
> like
> > >Python will full fill of Airavata requirements.
> > >
> > >There are many benefits in using a Python library as the API for
> > >controlling Airavata workflows.
> > >
> > >1. It is a library, gives the ultimate control over the execution and it
> > >can be simpler than any domain specific language that we can come with
> > >like
> > >XML, JSON etc
> > >2. Most people use python and can learn it easily than any Domain
> specific
> > >language
> > >3. You can easily create a Python library for Airavata because all the
> > >APIs
> > >are thrift based.
> > >4. If you design the constructs correctly you can plug an XBaya.
> > >
> > >Any thoughts?
> > >
> > >Thanks,
> > >Supun..
> > >
> > >
> > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <supun06@gmail.com
> >
> > >wrote:
> > >
> > >> Hi Shameera,
> > >>
> > >> Why you prefer JavaScript over a language like Python?
> > >>
> > >> Thanks,
> > >> Supun..
> > >>
> > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > >> shameerainfo@gmail.com> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> First of all thanks everyone for giving valuable inputs. After doing
> > >>>some
> > >>> background search and talking to different people in the University
> > >>>who has
> > >>> used different workflow languages, I myself convinced that
> introducing
> > >>>an
> > >>> another workflow language is not what actually they need. By changing
> > >>> exiting workflow language to another will not solve problems. What
> they
> > >>> asking is a easy way to construct the workflows. Indirectly what they
> > >>> asking for a sort of API which they can use to generate the workflows
> > >>>and
> > >>> run it. Correct me if i am wrong here.
> > >>>
> > >>> As most of above replies depict,  if we can get a simple API, as an
> > >>> example, for a web based application, JavaScript API would be a good
> > >>> solution, and probably JSON would be a good candidate for language,
> > >>>instead
> > >>> of XML.
> > >>>
> > >>> Airavata community already have started to implement web base GUI.
> > >>>Hence
> > >>> introducing a JSON base JavaScript API would be great help. WDYT?
> > >>>
> > >>> Thanks,
> > >>> Shameera.
> > >>>
> > >>>
> > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> > >>> alekny7@gmail.com> wrote:
> > >>>
> > >>>> Hi,
> > >>>>
> > >>>> it is not dataflow instead focused on orchestrating REST services
> but
> > >>>> you may find it useful datapoint - we created worfklow service that
> > >>>>uses
> > >>>> natively JavaScript and JSON to describe what happens during
> workflow
> > >>>> execution:
> > >>>>
> > https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> > >>>>
> > >>>> HTH,
> > >>>>
> > >>>> Alek
> > >>>>
> > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> > >>>>wrote:
> > >>>>
> > >>>>> Hi Chris,
> > >>>>>
> > >>>>> Great to hear OODT community will be interested in adopting a JSON
> > >>>>> based workflow language and potentially a web based composer as
> well.
> > >>>>> Airavata previously had BPEL support initially through a home grown
> > >>>>> implementation [1] by Alek Slominski and later through Apache ODE
> > >>>>>[2]. Also
> > >>>>> a white paper [3] by Alek on this topic is an interesting read.
> > >>>>>
> > >>>>> I am of the same opinion that we should adopt something more modern
> > >>>>>as
> > >>>>> the challenges from scientific workflows seems to be converging
> with
> > >>>>>the
> > >>>>> data flow patterns in business workflows.
> > >>>>>
> > >>>>> It will be great if we can all compile a list of potential
> candidates
> > >>>>> and hack them through.
> > >>>>>
> > >>>>> Suresh
> > >>>>> [1] -
> > >>>>>
> > >>>>>
> > http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > >>>>> [2] -
> > >>>>>
> > >>>>>
> > http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> > >>>>>time_for_eScience_workflows
> > >>>>> [3] -
> > >>>>>
> > >>>>>
> > http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> > >>>>>6.pdf
> > >>>>>
> > >>>>>
> > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> > >>>>>
> > >>>>> > Hi Guys,
> > >>>>> >
> > >>>>> > I've been interested in this too - we don't per have a specific
> > >>>>> > OODT workflow language, but we specific workflows using XML, and
> > >>>>> > other configuration (we are also thinking of moving to JSON for
> > >>>>> > this).
> > >>>>> >
> > >>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> > >>>>> > to me.
> > >>>>> >
> > >>>>> > I wonder at the end of the day if we should adopt something more
> > >>>>> > modern like PIG or some other data flow type of language (PIG
> > >>>>> > is really neat).
> > >>>>> >
> > >>>>> > Cheers,
> > >>>>> > Chris
> > >>>>> >
> > >>>>> >
> > >>>>> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>>>> > Chris Mattmann, Ph.D.
> > >>>>> > Chief Architect
> > >>>>> > Instrument Software and Science Data Systems Section (398)
> > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > >>>>> > Office: 168-519, Mailstop: 168-527
> > >>>>> > Email: chris.a.mattmann@nasa.gov
> > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
> > >>>>> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>>>> > Adjunct Associate Professor, Computer Science Department
> > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> > >>>>> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> > -----Original Message-----
> > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> > >>>>> > Reply-To: "architecture@airavata.apache.org"
> > >>>>> > <ar...@airavata.apache.org>
> > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> > >>>>> > To: "architecture@airavata.apache.org" <
> > >>>>> architecture@airavata.apache.org>,
> > >>>>> > dev <de...@airavata.apache.org>
> > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> > >>>>>Airavata.
> > >>>>> >
> > >>>>> >> Hi All,
> > >>>>> >>
> > >>>>> >> As we all know Airavata has its own workflow language call XWF.
> > >>>>>When
> > >>>>> XWF
> > >>>>> >> was introduced, main focus points are interoperability and
> > >>>>> convertibility.
> > >>>>> >> But with years of experience it is convinced that above
> > >>>>>requirements
> > >>>>> are
> > >>>>> >> not really useful when we come to real world use cases. And XWF
> is
> > >>>>> XML
> > >>>>> >> based bulky language where we attache WSDLs and Workflow image
> it
> > >>>>> self.
> > >>>>> >> But
> > >>>>> >> with the recent changes WSDL part is being removed from XWF.
> > >>>>> >>
> > >>>>> >> It is worth to evaluate handy Scientific workflow languages in
> > >>>>> industry
> > >>>>> >> and
> > >>>>> >> find out pros and cons, at the end of this evaluation we need to
> > >>>>> come up
> > >>>>> >> with idea how we should improve Airavata workflow language,
> either
> > >>>>> we can
> > >>>>> >> improve existing XWF language, totally change to a new language
> > >>>>> available
> > >>>>> >> in industry or write a new light weight language. Basic
> > >>>>>requirements
> > >>>>> that
> > >>>>> >> we expect from new improvement are, high usability, flexible,
> > >>>>>light
> > >>>>> weight
> > >>>>> >> and real time monitoring support. As you can see above
> > >>>>>requirements
> > >>>>> are
> > >>>>> >> not
> > >>>>> >> direct comes with workflow languages but we need workflow
> language
> > >>>>> which
> > >>>>> >> help to support above requirements.
> > >>>>> >>
> > >>>>> >> After reading few papers and googling, initially i have come up
> > >>>>>with
> > >>>>> >> following three existing languages,
> > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> > >>>>> >> 2. WS-BPEL
> > >>>>> >> 3. SIDL
> > >>>>> >> <
> http://computation.llnl.gov/casc/components/index.html#page=home
> > >
> > >>>>> >>
> > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> > >>>>> Radical-SAGA
> > >>>>> >> also uses slightly modified version of SIDL. Other than above
> > >>>>>three
> > >>>>> >> languages we can come up with simple workflow language base on
> > >>>>> json(or
> > >>>>> >> yaml) which support all our requirements for some extends.
> > >>>>> >>
> > >>>>> >> It would be grate if I can get more input regarding the $Subject
> > >>>>> form the
> > >>>>> >> airavata community. You all are more than welcome to provide any
> > >>>>> type of
> > >>>>> >> suggestions.
> > >>>>> >>
> > >>>>> >> Thanks,
> > >>>>> >> Shameera.
> > >>>>> >>
> > >>>>> >> 
> > >>>>> >>
> > >>>>> >> --
> > >>>>> >> Best Regards,
> > >>>>> >> Shameera Rathnayaka.
> > >>>>> >>
> > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> > >>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> The best way to predict the future is to invent it - Alan Kay
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Best Regards,
> > >>> Shameera Rathnayaka.
> > >>>
> > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > >>> Blog : http://shameerarathnayaka.blogspot.com/
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Supun Kamburugamuva
> > >> Member, Apache Software Foundation; http://www.apache.org
> > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > >> Blog: http://supunk.blogspot.com
> > >>
> > >>
> > >
> > >
> > >--
> > >Supun Kamburugamuva
> > >Member, Apache Software Foundation; http://www.apache.org
> > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > >Blog: http://supunk.blogspot.com
> >
> >
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>



-- 
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Supun Kamburugamuva <su...@gmail.com>.

Hi Shameera,

Using python is a radical approach for workflow I think. But I believe this
is a very beautiful and new approach.Here is a possible scenario for
implementation and running using a Python based library.

The Python library facilitates the creation of Applications and submitting
them to Airavata for execution. The underlying functionality is exactly
similar to what Java clients provides.The only difference is that, Python
library should have a more fluent API than Java for easy creation of
workflows. We can generate the Python clients that talk to Airavata server
using Thrift.

Here is an example off the top of my head to a Python script created by
user for a Workflow. This is a very crude example and we need to come up
with a much better API if we are going to go along this path. First we need
to write a Python script that can execute a workflow using Airavata.

import airavata

host = Host("localhost", ....)
app1 = Application(host, ....)
app2 = Application(host, ....)

# we will connect these applications as a workflow using some topology
builder or other constructs

wb = WorkFlowBuilder()
wb.setApp("name1", app1)
# you can do a simple output transformation here etc

# connects the input of app1 to app2 etc
wb.setApp("name2", app2).connectInput("name1")

wb.submit()

Now we can load this Python script from XBaya. When XBaya loads this script
the Python script can output an XML configuration of the topology, XBaya
can render. There are other ways like directly executing the Python script
from command line and connecting XBaya indirectly as well. Now you can run
the workflow from XBaya. Running the Workflow means just executing the
Python script.

XBaya gets the notifications through messaging and update the UI
accordingly.

The users need to write the Python script by hand. XBaya cannot create the
script. Because workflow language is an actual python program the benefits
are immense. For example user can do workflow steering in the workflow
itself by subscribing to messages from Airavata.

Thanks,
Supun..








On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:

> Hi Supun,
>
> I meant to say JS is a well-known client side scripting language i have
> messed scripting part. Even we use Python, ultimately we should convert
> this to java model in server side, somehow we need to serialized python
> representation to the language which java can parse and generate that
> model. In this case we need to parse python script in java isn't it? I am
> not exactly clear how you suggesting to use python here. More details on
> how end system works if we use Python would be great help to clearly
> understand your points.
>
> Thanks,
> Shameera.
>
> On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <ch...@gmail.com>
> wrote:
>
> > Have you guys considered using JCC [1] as a means
> > to expose the workflow API currently in Java as a
> > Python API?
> >
> > We are exploring its use in OODT, and we have already
> > created a Tika [2] JCC-based python API.
> >
> > Cheers,
> > Chris
> >
> > [1] http://lucene.apache.org/pylucene/jcc/
> > [2] http://github.com/chrismattmann/tika-python/
> >
> > ------------------------
> > Chris Mattmann
> > chris.mattmann@gmail.com
> >
> >
> >
> >
> > -----Original Message-----
> > From: Supun Kamburugamuva <su...@gmail.com>
> > Reply-To: <de...@oodt.apache.org>
> > Date: Thursday, October 16, 2014 at 3:43 PM
> > To: dev <de...@airavata.apache.org>
> > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> > <sm...@apache.org>, "architecture@airavata.apache.org"
> > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> > <de...@oodt.apache.org>
> > Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
> >
> > >Once we had an offline discussion about the Airavata Workflow language
> > >(with Milinda, Saliya and Shameera). In that discussion one thing came
> out
> > >was why we need to invent a different language when a simple library
> like
> > >Python will full fill of Airavata requirements.
> > >
> > >There are many benefits in using a Python library as the API for
> > >controlling Airavata workflows.
> > >
> > >1. It is a library, gives the ultimate control over the execution and it
> > >can be simpler than any domain specific language that we can come with
> > >like
> > >XML, JSON etc
> > >2. Most people use python and can learn it easily than any Domain
> specific
> > >language
> > >3. You can easily create a Python library for Airavata because all the
> > >APIs
> > >are thrift based.
> > >4. If you design the constructs correctly you can plug an XBaya.
> > >
> > >Any thoughts?
> > >
> > >Thanks,
> > >Supun..
> > >
> > >
> > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <supun06@gmail.com
> >
> > >wrote:
> > >
> > >> Hi Shameera,
> > >>
> > >> Why you prefer JavaScript over a language like Python?
> > >>
> > >> Thanks,
> > >> Supun..
> > >>
> > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > >> shameerainfo@gmail.com> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> First of all thanks everyone for giving valuable inputs. After doing
> > >>>some
> > >>> background search and talking to different people in the University
> > >>>who has
> > >>> used different workflow languages, I myself convinced that
> introducing
> > >>>an
> > >>> another workflow language is not what actually they need. By changing
> > >>> exiting workflow language to another will not solve problems. What
> they
> > >>> asking is a easy way to construct the workflows. Indirectly what they
> > >>> asking for a sort of API which they can use to generate the workflows
> > >>>and
> > >>> run it. Correct me if i am wrong here.
> > >>>
> > >>> As most of above replies depict,  if we can get a simple API, as an
> > >>> example, for a web based application, JavaScript API would be a good
> > >>> solution, and probably JSON would be a good candidate for language,
> > >>>instead
> > >>> of XML.
> > >>>
> > >>> Airavata community already have started to implement web base GUI.
> > >>>Hence
> > >>> introducing a JSON base JavaScript API would be great help. WDYT?
> > >>>
> > >>> Thanks,
> > >>> Shameera.
> > >>>
> > >>>
> > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> > >>> alekny7@gmail.com> wrote:
> > >>>
> > >>>> Hi,
> > >>>>
> > >>>> it is not dataflow instead focused on orchestrating REST services
> but
> > >>>> you may find it useful datapoint - we created worfklow service that
> > >>>>uses
> > >>>> natively JavaScript and JSON to describe what happens during
> workflow
> > >>>> execution:
> > >>>>
> > https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> > >>>>
> > >>>> HTH,
> > >>>>
> > >>>> Alek
> > >>>>
> > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> > >>>>wrote:
> > >>>>
> > >>>>> Hi Chris,
> > >>>>>
> > >>>>> Great to hear OODT community will be interested in adopting a JSON
> > >>>>> based workflow language and potentially a web based composer as
> well.
> > >>>>> Airavata previously had BPEL support initially through a home grown
> > >>>>> implementation [1] by Alek Slominski and later through Apache ODE
> > >>>>>[2]. Also
> > >>>>> a white paper [3] by Alek on this topic is an interesting read.
> > >>>>>
> > >>>>> I am of the same opinion that we should adopt something more modern
> > >>>>>as
> > >>>>> the challenges from scientific workflows seems to be converging
> with
> > >>>>>the
> > >>>>> data flow patterns in business workflows.
> > >>>>>
> > >>>>> It will be great if we can all compile a list of potential
> candidates
> > >>>>> and hack them through.
> > >>>>>
> > >>>>> Suresh
> > >>>>> [1] -
> > >>>>>
> > >>>>>
> > http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > >>>>> [2] -
> > >>>>>
> > >>>>>
> > http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> > >>>>>time_for_eScience_workflows
> > >>>>> [3] -
> > >>>>>
> > >>>>>
> > http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> > >>>>>6.pdf
> > >>>>>
> > >>>>>
> > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> > >>>>>
> > >>>>> > Hi Guys,
> > >>>>> >
> > >>>>> > I've been interested in this too - we don't per have a specific
> > >>>>> > OODT workflow language, but we specific workflows using XML, and
> > >>>>> > other configuration (we are also thinking of moving to JSON for
> > >>>>> > this).
> > >>>>> >
> > >>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> > >>>>> > to me.
> > >>>>> >
> > >>>>> > I wonder at the end of the day if we should adopt something more
> > >>>>> > modern like PIG or some other data flow type of language (PIG
> > >>>>> > is really neat).
> > >>>>> >
> > >>>>> > Cheers,
> > >>>>> > Chris
> > >>>>> >
> > >>>>> >
> > >>>>> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>>>> > Chris Mattmann, Ph.D.
> > >>>>> > Chief Architect
> > >>>>> > Instrument Software and Science Data Systems Section (398)
> > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > >>>>> > Office: 168-519, Mailstop: 168-527
> > >>>>> > Email: chris.a.mattmann@nasa.gov
> > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
> > >>>>> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>>>> > Adjunct Associate Professor, Computer Science Department
> > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> > >>>>> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> > -----Original Message-----
> > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> > >>>>> > Reply-To: "architecture@airavata.apache.org"
> > >>>>> > <ar...@airavata.apache.org>
> > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> > >>>>> > To: "architecture@airavata.apache.org" <
> > >>>>> architecture@airavata.apache.org>,
> > >>>>> > dev <de...@airavata.apache.org>
> > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> > >>>>>Airavata.
> > >>>>> >
> > >>>>> >> Hi All,
> > >>>>> >>
> > >>>>> >> As we all know Airavata has its own workflow language call XWF.
> > >>>>>When
> > >>>>> XWF
> > >>>>> >> was introduced, main focus points are interoperability and
> > >>>>> convertibility.
> > >>>>> >> But with years of experience it is convinced that above
> > >>>>>requirements
> > >>>>> are
> > >>>>> >> not really useful when we come to real world use cases. And XWF
> is
> > >>>>> XML
> > >>>>> >> based bulky language where we attache WSDLs and Workflow image
> it
> > >>>>> self.
> > >>>>> >> But
> > >>>>> >> with the recent changes WSDL part is being removed from XWF.
> > >>>>> >>
> > >>>>> >> It is worth to evaluate handy Scientific workflow languages in
> > >>>>> industry
> > >>>>> >> and
> > >>>>> >> find out pros and cons, at the end of this evaluation we need to
> > >>>>> come up
> > >>>>> >> with idea how we should improve Airavata workflow language,
> either
> > >>>>> we can
> > >>>>> >> improve existing XWF language, totally change to a new language
> > >>>>> available
> > >>>>> >> in industry or write a new light weight language. Basic
> > >>>>>requirements
> > >>>>> that
> > >>>>> >> we expect from new improvement are, high usability, flexible,
> > >>>>>light
> > >>>>> weight
> > >>>>> >> and real time monitoring support. As you can see above
> > >>>>>requirements
> > >>>>> are
> > >>>>> >> not
> > >>>>> >> direct comes with workflow languages but we need workflow
> language
> > >>>>> which
> > >>>>> >> help to support above requirements.
> > >>>>> >>
> > >>>>> >> After reading few papers and googling, initially i have come up
> > >>>>>with
> > >>>>> >> following three existing languages,
> > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> > >>>>> >> 2. WS-BPEL
> > >>>>> >> 3. SIDL
> > >>>>> >> <
> http://computation.llnl.gov/casc/components/index.html#page=home
> > >
> > >>>>> >>
> > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> > >>>>> Radical-SAGA
> > >>>>> >> also uses slightly modified version of SIDL. Other than above
> > >>>>>three
> > >>>>> >> languages we can come up with simple workflow language base on
> > >>>>> json(or
> > >>>>> >> yaml) which support all our requirements for some extends.
> > >>>>> >>
> > >>>>> >> It would be grate if I can get more input regarding the $Subject
> > >>>>> form the
> > >>>>> >> airavata community. You all are more than welcome to provide any
> > >>>>> type of
> > >>>>> >> suggestions.
> > >>>>> >>
> > >>>>> >> Thanks,
> > >>>>> >> Shameera.
> > >>>>> >>
> > >>>>> >> 
> > >>>>> >>
> > >>>>> >> --
> > >>>>> >> Best Regards,
> > >>>>> >> Shameera Rathnayaka.
> > >>>>> >>
> > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> > >>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> The best way to predict the future is to invent it - Alan Kay
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Best Regards,
> > >>> Shameera Rathnayaka.
> > >>>
> > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > >>> Blog : http://shameerarathnayaka.blogspot.com/
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Supun Kamburugamuva
> > >> Member, Apache Software Foundation; http://www.apache.org
> > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > >> Blog: http://supunk.blogspot.com
> > >>
> > >>
> > >
> > >
> > >--
> > >Supun Kamburugamuva
> > >Member, Apache Software Foundation; http://www.apache.org
> > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > >Blog: http://supunk.blogspot.com
> >
> >
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>



-- 
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Supun Kamburugamuva <su...@gmail.com>.

Hi Shameera,

Using python is a radical approach for workflow I think. But I believe this
is a very beautiful and new approach.Here is a possible scenario for
implementation and running using a Python based library.

The Python library facilitates the creation of Applications and submitting
them to Airavata for execution. The underlying functionality is exactly
similar to what Java clients provides.The only difference is that, Python
library should have a more fluent API than Java for easy creation of
workflows. We can generate the Python clients that talk to Airavata server
using Thrift.

Here is an example off the top of my head to a Python script created by
user for a Workflow. This is a very crude example and we need to come up
with a much better API if we are going to go along this path. First we need
to write a Python script that can execute a workflow using Airavata.

import airavata

host = Host("localhost", ....)
app1 = Application(host, ....)
app2 = Application(host, ....)

# we will connect these applications as a workflow using some topology
builder or other constructs

wb = WorkFlowBuilder()
wb.setApp("name1", app1)
# you can do a simple output transformation here etc

# connects the input of app1 to app2 etc
wb.setApp("name2", app2).connectInput("name1")

wb.submit()

Now we can load this Python script from XBaya. When XBaya loads this script
the Python script can output an XML configuration of the topology, XBaya
can render. There are other ways like directly executing the Python script
from command line and connecting XBaya indirectly as well. Now you can run
the workflow from XBaya. Running the Workflow means just executing the
Python script.

XBaya gets the notifications through messaging and update the UI
accordingly.

The users need to write the Python script by hand. XBaya cannot create the
script. Because workflow language is an actual python program the benefits
are immense. For example user can do workflow steering in the workflow
itself by subscribing to messages from Airavata.

Thanks,
Supun..








On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:

> Hi Supun,
>
> I meant to say JS is a well-known client side scripting language i have
> messed scripting part. Even we use Python, ultimately we should convert
> this to java model in server side, somehow we need to serialized python
> representation to the language which java can parse and generate that
> model. In this case we need to parse python script in java isn't it? I am
> not exactly clear how you suggesting to use python here. More details on
> how end system works if we use Python would be great help to clearly
> understand your points.
>
> Thanks,
> Shameera.
>
> On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <ch...@gmail.com>
> wrote:
>
> > Have you guys considered using JCC [1] as a means
> > to expose the workflow API currently in Java as a
> > Python API?
> >
> > We are exploring its use in OODT, and we have already
> > created a Tika [2] JCC-based python API.
> >
> > Cheers,
> > Chris
> >
> > [1] http://lucene.apache.org/pylucene/jcc/
> > [2] http://github.com/chrismattmann/tika-python/
> >
> > ------------------------
> > Chris Mattmann
> > chris.mattmann@gmail.com
> >
> >
> >
> >
> > -----Original Message-----
> > From: Supun Kamburugamuva <su...@gmail.com>
> > Reply-To: <de...@oodt.apache.org>
> > Date: Thursday, October 16, 2014 at 3:43 PM
> > To: dev <de...@airavata.apache.org>
> > Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> > <sm...@apache.org>, "architecture@airavata.apache.org"
> > <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> > <de...@oodt.apache.org>
> > Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
> >
> > >Once we had an offline discussion about the Airavata Workflow language
> > >(with Milinda, Saliya and Shameera). In that discussion one thing came
> out
> > >was why we need to invent a different language when a simple library
> like
> > >Python will full fill of Airavata requirements.
> > >
> > >There are many benefits in using a Python library as the API for
> > >controlling Airavata workflows.
> > >
> > >1. It is a library, gives the ultimate control over the execution and it
> > >can be simpler than any domain specific language that we can come with
> > >like
> > >XML, JSON etc
> > >2. Most people use python and can learn it easily than any Domain
> specific
> > >language
> > >3. You can easily create a Python library for Airavata because all the
> > >APIs
> > >are thrift based.
> > >4. If you design the constructs correctly you can plug an XBaya.
> > >
> > >Any thoughts?
> > >
> > >Thanks,
> > >Supun..
> > >
> > >
> > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <supun06@gmail.com
> >
> > >wrote:
> > >
> > >> Hi Shameera,
> > >>
> > >> Why you prefer JavaScript over a language like Python?
> > >>
> > >> Thanks,
> > >> Supun..
> > >>
> > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > >> shameerainfo@gmail.com> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> First of all thanks everyone for giving valuable inputs. After doing
> > >>>some
> > >>> background search and talking to different people in the University
> > >>>who has
> > >>> used different workflow languages, I myself convinced that
> introducing
> > >>>an
> > >>> another workflow language is not what actually they need. By changing
> > >>> exiting workflow language to another will not solve problems. What
> they
> > >>> asking is a easy way to construct the workflows. Indirectly what they
> > >>> asking for a sort of API which they can use to generate the workflows
> > >>>and
> > >>> run it. Correct me if i am wrong here.
> > >>>
> > >>> As most of above replies depict,  if we can get a simple API, as an
> > >>> example, for a web based application, JavaScript API would be a good
> > >>> solution, and probably JSON would be a good candidate for language,
> > >>>instead
> > >>> of XML.
> > >>>
> > >>> Airavata community already have started to implement web base GUI.
> > >>>Hence
> > >>> introducing a JSON base JavaScript API would be great help. WDYT?
> > >>>
> > >>> Thanks,
> > >>> Shameera.
> > >>>
> > >>>
> > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> > >>> alekny7@gmail.com> wrote:
> > >>>
> > >>>> Hi,
> > >>>>
> > >>>> it is not dataflow instead focused on orchestrating REST services
> but
> > >>>> you may find it useful datapoint - we created worfklow service that
> > >>>>uses
> > >>>> natively JavaScript and JSON to describe what happens during
> workflow
> > >>>> execution:
> > >>>>
> > https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> > >>>>
> > >>>> HTH,
> > >>>>
> > >>>> Alek
> > >>>>
> > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> > >>>>wrote:
> > >>>>
> > >>>>> Hi Chris,
> > >>>>>
> > >>>>> Great to hear OODT community will be interested in adopting a JSON
> > >>>>> based workflow language and potentially a web based composer as
> well.
> > >>>>> Airavata previously had BPEL support initially through a home grown
> > >>>>> implementation [1] by Alek Slominski and later through Apache ODE
> > >>>>>[2]. Also
> > >>>>> a white paper [3] by Alek on this topic is an interesting read.
> > >>>>>
> > >>>>> I am of the same opinion that we should adopt something more modern
> > >>>>>as
> > >>>>> the challenges from scientific workflows seems to be converging
> with
> > >>>>>the
> > >>>>> data flow patterns in business workflows.
> > >>>>>
> > >>>>> It will be great if we can all compile a list of potential
> candidates
> > >>>>> and hack them through.
> > >>>>>
> > >>>>> Suresh
> > >>>>> [1] -
> > >>>>>
> > >>>>>
> > http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > >>>>> [2] -
> > >>>>>
> > >>>>>
> > http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> > >>>>>time_for_eScience_workflows
> > >>>>> [3] -
> > >>>>>
> > >>>>>
> > http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> > >>>>>6.pdf
> > >>>>>
> > >>>>>
> > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> > >>>>>
> > >>>>> > Hi Guys,
> > >>>>> >
> > >>>>> > I've been interested in this too - we don't per have a specific
> > >>>>> > OODT workflow language, but we specific workflows using XML, and
> > >>>>> > other configuration (we are also thinking of moving to JSON for
> > >>>>> > this).
> > >>>>> >
> > >>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> > >>>>> > to me.
> > >>>>> >
> > >>>>> > I wonder at the end of the day if we should adopt something more
> > >>>>> > modern like PIG or some other data flow type of language (PIG
> > >>>>> > is really neat).
> > >>>>> >
> > >>>>> > Cheers,
> > >>>>> > Chris
> > >>>>> >
> > >>>>> >
> > >>>>> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>>>> > Chris Mattmann, Ph.D.
> > >>>>> > Chief Architect
> > >>>>> > Instrument Software and Science Data Systems Section (398)
> > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > >>>>> > Office: 168-519, Mailstop: 168-527
> > >>>>> > Email: chris.a.mattmann@nasa.gov
> > >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
> > >>>>> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>>>> > Adjunct Associate Professor, Computer Science Department
> > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> > >>>>> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> > -----Original Message-----
> > >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> > >>>>> > Reply-To: "architecture@airavata.apache.org"
> > >>>>> > <ar...@airavata.apache.org>
> > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> > >>>>> > To: "architecture@airavata.apache.org" <
> > >>>>> architecture@airavata.apache.org>,
> > >>>>> > dev <de...@airavata.apache.org>
> > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> > >>>>>Airavata.
> > >>>>> >
> > >>>>> >> Hi All,
> > >>>>> >>
> > >>>>> >> As we all know Airavata has its own workflow language call XWF.
> > >>>>>When
> > >>>>> XWF
> > >>>>> >> was introduced, main focus points are interoperability and
> > >>>>> convertibility.
> > >>>>> >> But with years of experience it is convinced that above
> > >>>>>requirements
> > >>>>> are
> > >>>>> >> not really useful when we come to real world use cases. And XWF
> is
> > >>>>> XML
> > >>>>> >> based bulky language where we attache WSDLs and Workflow image
> it
> > >>>>> self.
> > >>>>> >> But
> > >>>>> >> with the recent changes WSDL part is being removed from XWF.
> > >>>>> >>
> > >>>>> >> It is worth to evaluate handy Scientific workflow languages in
> > >>>>> industry
> > >>>>> >> and
> > >>>>> >> find out pros and cons, at the end of this evaluation we need to
> > >>>>> come up
> > >>>>> >> with idea how we should improve Airavata workflow language,
> either
> > >>>>> we can
> > >>>>> >> improve existing XWF language, totally change to a new language
> > >>>>> available
> > >>>>> >> in industry or write a new light weight language. Basic
> > >>>>>requirements
> > >>>>> that
> > >>>>> >> we expect from new improvement are, high usability, flexible,
> > >>>>>light
> > >>>>> weight
> > >>>>> >> and real time monitoring support. As you can see above
> > >>>>>requirements
> > >>>>> are
> > >>>>> >> not
> > >>>>> >> direct comes with workflow languages but we need workflow
> language
> > >>>>> which
> > >>>>> >> help to support above requirements.
> > >>>>> >>
> > >>>>> >> After reading few papers and googling, initially i have come up
> > >>>>>with
> > >>>>> >> following three existing languages,
> > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> > >>>>> >> 2. WS-BPEL
> > >>>>> >> 3. SIDL
> > >>>>> >> <
> http://computation.llnl.gov/casc/components/index.html#page=home
> > >
> > >>>>> >>
> > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> > >>>>> Radical-SAGA
> > >>>>> >> also uses slightly modified version of SIDL. Other than above
> > >>>>>three
> > >>>>> >> languages we can come up with simple workflow language base on
> > >>>>> json(or
> > >>>>> >> yaml) which support all our requirements for some extends.
> > >>>>> >>
> > >>>>> >> It would be grate if I can get more input regarding the $Subject
> > >>>>> form the
> > >>>>> >> airavata community. You all are more than welcome to provide any
> > >>>>> type of
> > >>>>> >> suggestions.
> > >>>>> >>
> > >>>>> >> Thanks,
> > >>>>> >> Shameera.
> > >>>>> >>
> > >>>>> >> 
> > >>>>> >>
> > >>>>> >> --
> > >>>>> >> Best Regards,
> > >>>>> >> Shameera Rathnayaka.
> > >>>>> >>
> > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> > >>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> The best way to predict the future is to invent it - Alan Kay
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Best Regards,
> > >>> Shameera Rathnayaka.
> > >>>
> > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > >>> Blog : http://shameerarathnayaka.blogspot.com/
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Supun Kamburugamuva
> > >> Member, Apache Software Foundation; http://www.apache.org
> > >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > >> Blog: http://supunk.blogspot.com
> > >>
> > >>
> > >
> > >
> > >--
> > >Supun Kamburugamuva
> > >Member, Apache Software Foundation; http://www.apache.org
> > >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > >Blog: http://supunk.blogspot.com
> >
> >
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>



-- 
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Shameera Rathnayaka <sh...@gmail.com>.

Hi Supun,

I meant to say JS is a well-known client side scripting language i have
messed scripting part. Even we use Python, ultimately we should convert
this to java model in server side, somehow we need to serialized python
representation to the language which java can parse and generate that
model. In this case we need to parse python script in java isn't it? I am
not exactly clear how you suggesting to use python here. More details on
how end system works if we use Python would be great help to clearly
understand your points.

Thanks,
Shameera.

On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <ch...@gmail.com>
wrote:

> Have you guys considered using JCC [1] as a means
> to expose the workflow API currently in Java as a
> Python API?
>
> We are exploring its use in OODT, and we have already
> created a Tika [2] JCC-based python API.
>
> Cheers,
> Chris
>
> [1] http://lucene.apache.org/pylucene/jcc/
> [2] http://github.com/chrismattmann/tika-python/
>
> ------------------------
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
> -----Original Message-----
> From: Supun Kamburugamuva <su...@gmail.com>
> Reply-To: <de...@oodt.apache.org>
> Date: Thursday, October 16, 2014 at 3:43 PM
> To: dev <de...@airavata.apache.org>
> Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> <sm...@apache.org>, "architecture@airavata.apache.org"
> <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> <de...@oodt.apache.org>
> Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
>
> >Once we had an offline discussion about the Airavata Workflow language
> >(with Milinda, Saliya and Shameera). In that discussion one thing came out
> >was why we need to invent a different language when a simple library like
> >Python will full fill of Airavata requirements.
> >
> >There are many benefits in using a Python library as the API for
> >controlling Airavata workflows.
> >
> >1. It is a library, gives the ultimate control over the execution and it
> >can be simpler than any domain specific language that we can come with
> >like
> >XML, JSON etc
> >2. Most people use python and can learn it easily than any Domain specific
> >language
> >3. You can easily create a Python library for Airavata because all the
> >APIs
> >are thrift based.
> >4. If you design the constructs correctly you can plug an XBaya.
> >
> >Any thoughts?
> >
> >Thanks,
> >Supun..
> >
> >
> >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> >wrote:
> >
> >> Hi Shameera,
> >>
> >> Why you prefer JavaScript over a language like Python?
> >>
> >> Thanks,
> >> Supun..
> >>
> >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> >> shameerainfo@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> First of all thanks everyone for giving valuable inputs. After doing
> >>>some
> >>> background search and talking to different people in the University
> >>>who has
> >>> used different workflow languages, I myself convinced that introducing
> >>>an
> >>> another workflow language is not what actually they need. By changing
> >>> exiting workflow language to another will not solve problems. What they
> >>> asking is a easy way to construct the workflows. Indirectly what they
> >>> asking for a sort of API which they can use to generate the workflows
> >>>and
> >>> run it. Correct me if i am wrong here.
> >>>
> >>> As most of above replies depict,  if we can get a simple API, as an
> >>> example, for a web based application, JavaScript API would be a good
> >>> solution, and probably JSON would be a good candidate for language,
> >>>instead
> >>> of XML.
> >>>
> >>> Airavata community already have started to implement web base GUI.
> >>>Hence
> >>> introducing a JSON base JavaScript API would be great help. WDYT?
> >>>
> >>> Thanks,
> >>> Shameera.
> >>>
> >>>
> >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >>> alekny7@gmail.com> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> it is not dataflow instead focused on orchestrating REST services but
> >>>> you may find it useful datapoint - we created worfklow service that
> >>>>uses
> >>>> natively JavaScript and JSON to describe what happens during workflow
> >>>> execution:
> >>>>
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>>
> >>>> HTH,
> >>>>
> >>>> Alek
> >>>>
> >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> >>>>wrote:
> >>>>
> >>>>> Hi Chris,
> >>>>>
> >>>>> Great to hear OODT community will be interested in adopting a JSON
> >>>>> based workflow language and potentially a web based composer as well.
> >>>>> Airavata previously had BPEL support initially through a home grown
> >>>>> implementation [1] by Alek Slominski and later through Apache ODE
> >>>>>[2]. Also
> >>>>> a white paper [3] by Alek on this topic is an interesting read.
> >>>>>
> >>>>> I am of the same opinion that we should adopt something more modern
> >>>>>as
> >>>>> the challenges from scientific workflows seems to be converging with
> >>>>>the
> >>>>> data flow patterns in business workflows.
> >>>>>
> >>>>> It will be great if we can all compile a list of potential candidates
> >>>>> and hack them through.
> >>>>>
> >>>>> Suresh
> >>>>> [1] -
> >>>>>
> >>>>>
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>>>> [2] -
> >>>>>
> >>>>>
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> >>>>>time_for_eScience_workflows
> >>>>> [3] -
> >>>>>
> >>>>>
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> >>>>>6.pdf
> >>>>>
> >>>>>
> >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>>>>
> >>>>> > Hi Guys,
> >>>>> >
> >>>>> > I've been interested in this too - we don't per have a specific
> >>>>> > OODT workflow language, but we specific workflows using XML, and
> >>>>> > other configuration (we are also thinking of moving to JSON for
> >>>>> > this).
> >>>>> >
> >>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> >>>>> > to me.
> >>>>> >
> >>>>> > I wonder at the end of the day if we should adopt something more
> >>>>> > modern like PIG or some other data flow type of language (PIG
> >>>>> > is really neat).
> >>>>> >
> >>>>> > Cheers,
> >>>>> > Chris
> >>>>> >
> >>>>> >
> >>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>> > Chris Mattmann, Ph.D.
> >>>>> > Chief Architect
> >>>>> > Instrument Software and Science Data Systems Section (398)
> >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>>>> > Office: 168-519, Mailstop: 168-527
> >>>>> > Email: chris.a.mattmann@nasa.gov
> >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
> >>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>> > Adjunct Associate Professor, Computer Science Department
> >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> >>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > -----Original Message-----
> >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>>>> > Reply-To: "architecture@airavata.apache.org"
> >>>>> > <ar...@airavata.apache.org>
> >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>>>> > To: "architecture@airavata.apache.org" <
> >>>>> architecture@airavata.apache.org>,
> >>>>> > dev <de...@airavata.apache.org>
> >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> >>>>>Airavata.
> >>>>> >
> >>>>> >> Hi All,
> >>>>> >>
> >>>>> >> As we all know Airavata has its own workflow language call XWF.
> >>>>>When
> >>>>> XWF
> >>>>> >> was introduced, main focus points are interoperability and
> >>>>> convertibility.
> >>>>> >> But with years of experience it is convinced that above
> >>>>>requirements
> >>>>> are
> >>>>> >> not really useful when we come to real world use cases. And XWF is
> >>>>> XML
> >>>>> >> based bulky language where we attache WSDLs and Workflow image it
> >>>>> self.
> >>>>> >> But
> >>>>> >> with the recent changes WSDL part is being removed from XWF.
> >>>>> >>
> >>>>> >> It is worth to evaluate handy Scientific workflow languages in
> >>>>> industry
> >>>>> >> and
> >>>>> >> find out pros and cons, at the end of this evaluation we need to
> >>>>> come up
> >>>>> >> with idea how we should improve Airavata workflow language, either
> >>>>> we can
> >>>>> >> improve existing XWF language, totally change to a new language
> >>>>> available
> >>>>> >> in industry or write a new light weight language. Basic
> >>>>>requirements
> >>>>> that
> >>>>> >> we expect from new improvement are, high usability, flexible,
> >>>>>light
> >>>>> weight
> >>>>> >> and real time monitoring support. As you can see above
> >>>>>requirements
> >>>>> are
> >>>>> >> not
> >>>>> >> direct comes with workflow languages but we need workflow language
> >>>>> which
> >>>>> >> help to support above requirements.
> >>>>> >>
> >>>>> >> After reading few papers and googling, initially i have come up
> >>>>>with
> >>>>> >> following three existing languages,
> >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>>>> >> 2. WS-BPEL
> >>>>> >> 3. SIDL
> >>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home
> >
> >>>>> >>
> >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> >>>>> Radical-SAGA
> >>>>> >> also uses slightly modified version of SIDL. Other than above
> >>>>>three
> >>>>> >> languages we can come up with simple workflow language base on
> >>>>> json(or
> >>>>> >> yaml) which support all our requirements for some extends.
> >>>>> >>
> >>>>> >> It would be grate if I can get more input regarding the $Subject
> >>>>> form the
> >>>>> >> airavata community. You all are more than welcome to provide any
> >>>>> type of
> >>>>> >> suggestions.
> >>>>> >>
> >>>>> >> Thanks,
> >>>>> >> Shameera.
> >>>>> >>
> >>>>> >> 
> >>>>> >>
> >>>>> >> --
> >>>>> >> Best Regards,
> >>>>> >> Shameera Rathnayaka.
> >>>>> >>
> >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> The best way to predict the future is to invent it - Alan Kay
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards,
> >>> Shameera Rathnayaka.
> >>>
> >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> Blog : http://shameerarathnayaka.blogspot.com/
> >>>
> >>
> >>
> >>
> >> --
> >> Supun Kamburugamuva
> >> Member, Apache Software Foundation; http://www.apache.org
> >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >> Blog: http://supunk.blogspot.com
> >>
> >>
> >
> >
> >--
> >Supun Kamburugamuva
> >Member, Apache Software Foundation; http://www.apache.org
> >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >Blog: http://supunk.blogspot.com
>
>
>


-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Shameera Rathnayaka <sh...@gmail.com>.

Hi Supun,

I meant to say JS is a well-known client side scripting language i have
messed scripting part. Even we use Python, ultimately we should convert
this to java model in server side, somehow we need to serialized python
representation to the language which java can parse and generate that
model. In this case we need to parse python script in java isn't it? I am
not exactly clear how you suggesting to use python here. More details on
how end system works if we use Python would be great help to clearly
understand your points.

Thanks,
Shameera.

On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <ch...@gmail.com>
wrote:

> Have you guys considered using JCC [1] as a means
> to expose the workflow API currently in Java as a
> Python API?
>
> We are exploring its use in OODT, and we have already
> created a Tika [2] JCC-based python API.
>
> Cheers,
> Chris
>
> [1] http://lucene.apache.org/pylucene/jcc/
> [2] http://github.com/chrismattmann/tika-python/
>
> ------------------------
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
> -----Original Message-----
> From: Supun Kamburugamuva <su...@gmail.com>
> Reply-To: <de...@oodt.apache.org>
> Date: Thursday, October 16, 2014 at 3:43 PM
> To: dev <de...@airavata.apache.org>
> Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> <sm...@apache.org>, "architecture@airavata.apache.org"
> <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> <de...@oodt.apache.org>
> Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
>
> >Once we had an offline discussion about the Airavata Workflow language
> >(with Milinda, Saliya and Shameera). In that discussion one thing came out
> >was why we need to invent a different language when a simple library like
> >Python will full fill of Airavata requirements.
> >
> >There are many benefits in using a Python library as the API for
> >controlling Airavata workflows.
> >
> >1. It is a library, gives the ultimate control over the execution and it
> >can be simpler than any domain specific language that we can come with
> >like
> >XML, JSON etc
> >2. Most people use python and can learn it easily than any Domain specific
> >language
> >3. You can easily create a Python library for Airavata because all the
> >APIs
> >are thrift based.
> >4. If you design the constructs correctly you can plug an XBaya.
> >
> >Any thoughts?
> >
> >Thanks,
> >Supun..
> >
> >
> >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> >wrote:
> >
> >> Hi Shameera,
> >>
> >> Why you prefer JavaScript over a language like Python?
> >>
> >> Thanks,
> >> Supun..
> >>
> >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> >> shameerainfo@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> First of all thanks everyone for giving valuable inputs. After doing
> >>>some
> >>> background search and talking to different people in the University
> >>>who has
> >>> used different workflow languages, I myself convinced that introducing
> >>>an
> >>> another workflow language is not what actually they need. By changing
> >>> exiting workflow language to another will not solve problems. What they
> >>> asking is a easy way to construct the workflows. Indirectly what they
> >>> asking for a sort of API which they can use to generate the workflows
> >>>and
> >>> run it. Correct me if i am wrong here.
> >>>
> >>> As most of above replies depict,  if we can get a simple API, as an
> >>> example, for a web based application, JavaScript API would be a good
> >>> solution, and probably JSON would be a good candidate for language,
> >>>instead
> >>> of XML.
> >>>
> >>> Airavata community already have started to implement web base GUI.
> >>>Hence
> >>> introducing a JSON base JavaScript API would be great help. WDYT?
> >>>
> >>> Thanks,
> >>> Shameera.
> >>>
> >>>
> >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >>> alekny7@gmail.com> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> it is not dataflow instead focused on orchestrating REST services but
> >>>> you may find it useful datapoint - we created worfklow service that
> >>>>uses
> >>>> natively JavaScript and JSON to describe what happens during workflow
> >>>> execution:
> >>>>
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>>
> >>>> HTH,
> >>>>
> >>>> Alek
> >>>>
> >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> >>>>wrote:
> >>>>
> >>>>> Hi Chris,
> >>>>>
> >>>>> Great to hear OODT community will be interested in adopting a JSON
> >>>>> based workflow language and potentially a web based composer as well.
> >>>>> Airavata previously had BPEL support initially through a home grown
> >>>>> implementation [1] by Alek Slominski and later through Apache ODE
> >>>>>[2]. Also
> >>>>> a white paper [3] by Alek on this topic is an interesting read.
> >>>>>
> >>>>> I am of the same opinion that we should adopt something more modern
> >>>>>as
> >>>>> the challenges from scientific workflows seems to be converging with
> >>>>>the
> >>>>> data flow patterns in business workflows.
> >>>>>
> >>>>> It will be great if we can all compile a list of potential candidates
> >>>>> and hack them through.
> >>>>>
> >>>>> Suresh
> >>>>> [1] -
> >>>>>
> >>>>>
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>>>> [2] -
> >>>>>
> >>>>>
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> >>>>>time_for_eScience_workflows
> >>>>> [3] -
> >>>>>
> >>>>>
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> >>>>>6.pdf
> >>>>>
> >>>>>
> >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>>>>
> >>>>> > Hi Guys,
> >>>>> >
> >>>>> > I've been interested in this too - we don't per have a specific
> >>>>> > OODT workflow language, but we specific workflows using XML, and
> >>>>> > other configuration (we are also thinking of moving to JSON for
> >>>>> > this).
> >>>>> >
> >>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> >>>>> > to me.
> >>>>> >
> >>>>> > I wonder at the end of the day if we should adopt something more
> >>>>> > modern like PIG or some other data flow type of language (PIG
> >>>>> > is really neat).
> >>>>> >
> >>>>> > Cheers,
> >>>>> > Chris
> >>>>> >
> >>>>> >
> >>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>> > Chris Mattmann, Ph.D.
> >>>>> > Chief Architect
> >>>>> > Instrument Software and Science Data Systems Section (398)
> >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>>>> > Office: 168-519, Mailstop: 168-527
> >>>>> > Email: chris.a.mattmann@nasa.gov
> >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
> >>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>> > Adjunct Associate Professor, Computer Science Department
> >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> >>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > -----Original Message-----
> >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>>>> > Reply-To: "architecture@airavata.apache.org"
> >>>>> > <ar...@airavata.apache.org>
> >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>>>> > To: "architecture@airavata.apache.org" <
> >>>>> architecture@airavata.apache.org>,
> >>>>> > dev <de...@airavata.apache.org>
> >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> >>>>>Airavata.
> >>>>> >
> >>>>> >> Hi All,
> >>>>> >>
> >>>>> >> As we all know Airavata has its own workflow language call XWF.
> >>>>>When
> >>>>> XWF
> >>>>> >> was introduced, main focus points are interoperability and
> >>>>> convertibility.
> >>>>> >> But with years of experience it is convinced that above
> >>>>>requirements
> >>>>> are
> >>>>> >> not really useful when we come to real world use cases. And XWF is
> >>>>> XML
> >>>>> >> based bulky language where we attache WSDLs and Workflow image it
> >>>>> self.
> >>>>> >> But
> >>>>> >> with the recent changes WSDL part is being removed from XWF.
> >>>>> >>
> >>>>> >> It is worth to evaluate handy Scientific workflow languages in
> >>>>> industry
> >>>>> >> and
> >>>>> >> find out pros and cons, at the end of this evaluation we need to
> >>>>> come up
> >>>>> >> with idea how we should improve Airavata workflow language, either
> >>>>> we can
> >>>>> >> improve existing XWF language, totally change to a new language
> >>>>> available
> >>>>> >> in industry or write a new light weight language. Basic
> >>>>>requirements
> >>>>> that
> >>>>> >> we expect from new improvement are, high usability, flexible,
> >>>>>light
> >>>>> weight
> >>>>> >> and real time monitoring support. As you can see above
> >>>>>requirements
> >>>>> are
> >>>>> >> not
> >>>>> >> direct comes with workflow languages but we need workflow language
> >>>>> which
> >>>>> >> help to support above requirements.
> >>>>> >>
> >>>>> >> After reading few papers and googling, initially i have come up
> >>>>>with
> >>>>> >> following three existing languages,
> >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>>>> >> 2. WS-BPEL
> >>>>> >> 3. SIDL
> >>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home
> >
> >>>>> >>
> >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> >>>>> Radical-SAGA
> >>>>> >> also uses slightly modified version of SIDL. Other than above
> >>>>>three
> >>>>> >> languages we can come up with simple workflow language base on
> >>>>> json(or
> >>>>> >> yaml) which support all our requirements for some extends.
> >>>>> >>
> >>>>> >> It would be grate if I can get more input regarding the $Subject
> >>>>> form the
> >>>>> >> airavata community. You all are more than welcome to provide any
> >>>>> type of
> >>>>> >> suggestions.
> >>>>> >>
> >>>>> >> Thanks,
> >>>>> >> Shameera.
> >>>>> >>
> >>>>> >> 
> >>>>> >>
> >>>>> >> --
> >>>>> >> Best Regards,
> >>>>> >> Shameera Rathnayaka.
> >>>>> >>
> >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> The best way to predict the future is to invent it - Alan Kay
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards,
> >>> Shameera Rathnayaka.
> >>>
> >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> Blog : http://shameerarathnayaka.blogspot.com/
> >>>
> >>
> >>
> >>
> >> --
> >> Supun Kamburugamuva
> >> Member, Apache Software Foundation; http://www.apache.org
> >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >> Blog: http://supunk.blogspot.com
> >>
> >>
> >
> >
> >--
> >Supun Kamburugamuva
> >Member, Apache Software Foundation; http://www.apache.org
> >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >Blog: http://supunk.blogspot.com
>
>
>


-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Shameera Rathnayaka <sh...@gmail.com>.

Hi Supun,

I meant to say JS is a well-known client side scripting language i have
messed scripting part. Even we use Python, ultimately we should convert
this to java model in server side, somehow we need to serialized python
representation to the language which java can parse and generate that
model. In this case we need to parse python script in java isn't it? I am
not exactly clear how you suggesting to use python here. More details on
how end system works if we use Python would be great help to clearly
understand your points.

Thanks,
Shameera.

On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <ch...@gmail.com>
wrote:

> Have you guys considered using JCC [1] as a means
> to expose the workflow API currently in Java as a
> Python API?
>
> We are exploring its use in OODT, and we have already
> created a Tika [2] JCC-based python API.
>
> Cheers,
> Chris
>
> [1] http://lucene.apache.org/pylucene/jcc/
> [2] http://github.com/chrismattmann/tika-python/
>
> ------------------------
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
> -----Original Message-----
> From: Supun Kamburugamuva <su...@gmail.com>
> Reply-To: <de...@oodt.apache.org>
> Date: Thursday, October 16, 2014 at 3:43 PM
> To: dev <de...@airavata.apache.org>
> Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
> <sm...@apache.org>, "architecture@airavata.apache.org"
> <ar...@airavata.apache.org>, "dev@oodt.apache.org"
> <de...@oodt.apache.org>
> Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
>
> >Once we had an offline discussion about the Airavata Workflow language
> >(with Milinda, Saliya and Shameera). In that discussion one thing came out
> >was why we need to invent a different language when a simple library like
> >Python will full fill of Airavata requirements.
> >
> >There are many benefits in using a Python library as the API for
> >controlling Airavata workflows.
> >
> >1. It is a library, gives the ultimate control over the execution and it
> >can be simpler than any domain specific language that we can come with
> >like
> >XML, JSON etc
> >2. Most people use python and can learn it easily than any Domain specific
> >language
> >3. You can easily create a Python library for Airavata because all the
> >APIs
> >are thrift based.
> >4. If you design the constructs correctly you can plug an XBaya.
> >
> >Any thoughts?
> >
> >Thanks,
> >Supun..
> >
> >
> >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> >wrote:
> >
> >> Hi Shameera,
> >>
> >> Why you prefer JavaScript over a language like Python?
> >>
> >> Thanks,
> >> Supun..
> >>
> >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> >> shameerainfo@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> First of all thanks everyone for giving valuable inputs. After doing
> >>>some
> >>> background search and talking to different people in the University
> >>>who has
> >>> used different workflow languages, I myself convinced that introducing
> >>>an
> >>> another workflow language is not what actually they need. By changing
> >>> exiting workflow language to another will not solve problems. What they
> >>> asking is a easy way to construct the workflows. Indirectly what they
> >>> asking for a sort of API which they can use to generate the workflows
> >>>and
> >>> run it. Correct me if i am wrong here.
> >>>
> >>> As most of above replies depict,  if we can get a simple API, as an
> >>> example, for a web based application, JavaScript API would be a good
> >>> solution, and probably JSON would be a good candidate for language,
> >>>instead
> >>> of XML.
> >>>
> >>> Airavata community already have started to implement web base GUI.
> >>>Hence
> >>> introducing a JSON base JavaScript API would be great help. WDYT?
> >>>
> >>> Thanks,
> >>> Shameera.
> >>>
> >>>
> >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >>> alekny7@gmail.com> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> it is not dataflow instead focused on orchestrating REST services but
> >>>> you may find it useful datapoint - we created worfklow service that
> >>>>uses
> >>>> natively JavaScript and JSON to describe what happens during workflow
> >>>> execution:
> >>>>
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>>
> >>>> HTH,
> >>>>
> >>>> Alek
> >>>>
> >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> >>>>wrote:
> >>>>
> >>>>> Hi Chris,
> >>>>>
> >>>>> Great to hear OODT community will be interested in adopting a JSON
> >>>>> based workflow language and potentially a web based composer as well.
> >>>>> Airavata previously had BPEL support initially through a home grown
> >>>>> implementation [1] by Alek Slominski and later through Apache ODE
> >>>>>[2]. Also
> >>>>> a white paper [3] by Alek on this topic is an interesting read.
> >>>>>
> >>>>> I am of the same opinion that we should adopt something more modern
> >>>>>as
> >>>>> the challenges from scientific workflows seems to be converging with
> >>>>>the
> >>>>> data flow patterns in business workflows.
> >>>>>
> >>>>> It will be great if we can all compile a list of potential candidates
> >>>>> and hack them through.
> >>>>>
> >>>>> Suresh
> >>>>> [1] -
> >>>>>
> >>>>>
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>>>> [2] -
> >>>>>
> >>>>>
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> >>>>>time_for_eScience_workflows
> >>>>> [3] -
> >>>>>
> >>>>>
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> >>>>>6.pdf
> >>>>>
> >>>>>
> >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>>>>
> >>>>> > Hi Guys,
> >>>>> >
> >>>>> > I've been interested in this too - we don't per have a specific
> >>>>> > OODT workflow language, but we specific workflows using XML, and
> >>>>> > other configuration (we are also thinking of moving to JSON for
> >>>>> > this).
> >>>>> >
> >>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> >>>>> > to me.
> >>>>> >
> >>>>> > I wonder at the end of the day if we should adopt something more
> >>>>> > modern like PIG or some other data flow type of language (PIG
> >>>>> > is really neat).
> >>>>> >
> >>>>> > Cheers,
> >>>>> > Chris
> >>>>> >
> >>>>> >
> >>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>> > Chris Mattmann, Ph.D.
> >>>>> > Chief Architect
> >>>>> > Instrument Software and Science Data Systems Section (398)
> >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>>>> > Office: 168-519, Mailstop: 168-527
> >>>>> > Email: chris.a.mattmann@nasa.gov
> >>>>> > WWW:  http://sunset.usc.edu/~mattmann/
> >>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>> > Adjunct Associate Professor, Computer Science Department
> >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> >>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > -----Original Message-----
> >>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>>>> > Reply-To: "architecture@airavata.apache.org"
> >>>>> > <ar...@airavata.apache.org>
> >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>>>> > To: "architecture@airavata.apache.org" <
> >>>>> architecture@airavata.apache.org>,
> >>>>> > dev <de...@airavata.apache.org>
> >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> >>>>>Airavata.
> >>>>> >
> >>>>> >> Hi All,
> >>>>> >>
> >>>>> >> As we all know Airavata has its own workflow language call XWF.
> >>>>>When
> >>>>> XWF
> >>>>> >> was introduced, main focus points are interoperability and
> >>>>> convertibility.
> >>>>> >> But with years of experience it is convinced that above
> >>>>>requirements
> >>>>> are
> >>>>> >> not really useful when we come to real world use cases. And XWF is
> >>>>> XML
> >>>>> >> based bulky language where we attache WSDLs and Workflow image it
> >>>>> self.
> >>>>> >> But
> >>>>> >> with the recent changes WSDL part is being removed from XWF.
> >>>>> >>
> >>>>> >> It is worth to evaluate handy Scientific workflow languages in
> >>>>> industry
> >>>>> >> and
> >>>>> >> find out pros and cons, at the end of this evaluation we need to
> >>>>> come up
> >>>>> >> with idea how we should improve Airavata workflow language, either
> >>>>> we can
> >>>>> >> improve existing XWF language, totally change to a new language
> >>>>> available
> >>>>> >> in industry or write a new light weight language. Basic
> >>>>>requirements
> >>>>> that
> >>>>> >> we expect from new improvement are, high usability, flexible,
> >>>>>light
> >>>>> weight
> >>>>> >> and real time monitoring support. As you can see above
> >>>>>requirements
> >>>>> are
> >>>>> >> not
> >>>>> >> direct comes with workflow languages but we need workflow language
> >>>>> which
> >>>>> >> help to support above requirements.
> >>>>> >>
> >>>>> >> After reading few papers and googling, initially i have come up
> >>>>>with
> >>>>> >> following three existing languages,
> >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>>>> >> 2. WS-BPEL
> >>>>> >> 3. SIDL
> >>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home
> >
> >>>>> >>
> >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> >>>>> Radical-SAGA
> >>>>> >> also uses slightly modified version of SIDL. Other than above
> >>>>>three
> >>>>> >> languages we can come up with simple workflow language base on
> >>>>> json(or
> >>>>> >> yaml) which support all our requirements for some extends.
> >>>>> >>
> >>>>> >> It would be grate if I can get more input regarding the $Subject
> >>>>> form the
> >>>>> >> airavata community. You all are more than welcome to provide any
> >>>>> type of
> >>>>> >> suggestions.
> >>>>> >>
> >>>>> >> Thanks,
> >>>>> >> Shameera.
> >>>>> >>
> >>>>> >> 
> >>>>> >>
> >>>>> >> --
> >>>>> >> Best Regards,
> >>>>> >> Shameera Rathnayaka.
> >>>>> >>
> >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> The best way to predict the future is to invent it - Alan Kay
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards,
> >>> Shameera Rathnayaka.
> >>>
> >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>> Blog : http://shameerarathnayaka.blogspot.com/
> >>>
> >>
> >>
> >>
> >> --
> >> Supun Kamburugamuva
> >> Member, Apache Software Foundation; http://www.apache.org
> >> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >> Blog: http://supunk.blogspot.com
> >>
> >>
> >
> >
> >--
> >Supun Kamburugamuva
> >Member, Apache Software Foundation; http://www.apache.org
> >E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> >Blog: http://supunk.blogspot.com
>
>
>


-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Chris Mattmann <ch...@gmail.com>.

Have you guys considered using JCC [1] as a means
to expose the workflow API currently in Java as a
Python API?

We are exploring its use in OODT, and we have already
created a Tika [2] JCC-based python API.

Cheers,
Chris

[1] http://lucene.apache.org/pylucene/jcc/
[2] http://github.com/chrismattmann/tika-python/

------------------------
Chris Mattmann
chris.mattmann@gmail.com




-----Original Message-----
From: Supun Kamburugamuva <su...@gmail.com>
Reply-To: <de...@oodt.apache.org>
Date: Thursday, October 16, 2014 at 3:43 PM
To: dev <de...@airavata.apache.org>
Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
<sm...@apache.org>, "architecture@airavata.apache.org"
<ar...@airavata.apache.org>, "dev@oodt.apache.org"
<de...@oodt.apache.org>
Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.

>Once we had an offline discussion about the Airavata Workflow language
>(with Milinda, Saliya and Shameera). In that discussion one thing came out
>was why we need to invent a different language when a simple library like
>Python will full fill of Airavata requirements.
>
>There are many benefits in using a Python library as the API for
>controlling Airavata workflows.
>
>1. It is a library, gives the ultimate control over the execution and it
>can be simpler than any domain specific language that we can come with
>like
>XML, JSON etc
>2. Most people use python and can learn it easily than any Domain specific
>language
>3. You can easily create a Python library for Airavata because all the
>APIs
>are thrift based.
>4. If you design the constructs correctly you can plug an XBaya.
>
>Any thoughts?
>
>Thanks,
>Supun..
>
>
>On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
>wrote:
>
>> Hi Shameera,
>>
>> Why you prefer JavaScript over a language like Python?
>>
>> Thanks,
>> Supun..
>>
>> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> First of all thanks everyone for giving valuable inputs. After doing
>>>some
>>> background search and talking to different people in the University
>>>who has
>>> used different workflow languages, I myself convinced that introducing
>>>an
>>> another workflow language is not what actually they need. By changing
>>> exiting workflow language to another will not solve problems. What they
>>> asking is a easy way to construct the workflows. Indirectly what they
>>> asking for a sort of API which they can use to generate the workflows
>>>and
>>> run it. Correct me if i am wrong here.
>>>
>>> As most of above replies depict,  if we can get a simple API, as an
>>> example, for a web based application, JavaScript API would be a good
>>> solution, and probably JSON would be a good candidate for language,
>>>instead
>>> of XML.
>>>
>>> Airavata community already have started to implement web base GUI.
>>>Hence
>>> introducing a JSON base JavaScript API would be great help. WDYT?
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> alekny7@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> it is not dataflow instead focused on orchestrating REST services but
>>>> you may find it useful datapoint - we created worfklow service that
>>>>uses
>>>> natively JavaScript and JSON to describe what happens during workflow
>>>> execution:
>>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>>
>>>> HTH,
>>>>
>>>> Alek
>>>>
>>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
>>>>wrote:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>> Great to hear OODT community will be interested in adopting a JSON
>>>>> based workflow language and potentially a web based composer as well.
>>>>> Airavata previously had BPEL support initially through a home grown
>>>>> implementation [1] by Alek Slominski and later through Apache ODE
>>>>>[2]. Also
>>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>>
>>>>> I am of the same opinion that we should adopt something more modern
>>>>>as
>>>>> the challenges from scientific workflows seems to be converging with
>>>>>the
>>>>> data flow patterns in business workflows.
>>>>>
>>>>> It will be great if we can all compile a list of potential candidates
>>>>> and hack them through.
>>>>>
>>>>> Suresh
>>>>> [1] -
>>>>> 
>>>>>http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>>> [2] -
>>>>> 
>>>>>http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>>>>>time_for_eScience_workflows
>>>>> [3] -
>>>>> 
>>>>>http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>>>>>6.pdf
>>>>>
>>>>>
>>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>>
>>>>> > Hi Guys,
>>>>> >
>>>>> > I've been interested in this too - we don't per have a specific
>>>>> > OODT workflow language, but we specific workflows using XML, and
>>>>> > other configuration (we are also thinking of moving to JSON for
>>>>> > this).
>>>>> >
>>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>>> > to me.
>>>>> >
>>>>> > I wonder at the end of the day if we should adopt something more
>>>>> > modern like PIG or some other data flow type of language (PIG
>>>>> > is really neat).
>>>>> >
>>>>> > Cheers,
>>>>> > Chris
>>>>> >
>>>>> >
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Chris Mattmann, Ph.D.
>>>>> > Chief Architect
>>>>> > Instrument Software and Science Data Systems Section (398)
>>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>> > Office: 168-519, Mailstop: 168-527
>>>>> > Email: chris.a.mattmann@nasa.gov
>>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Adjunct Associate Professor, Computer Science Department
>>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > -----Original Message-----
>>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>>> > Reply-To: "architecture@airavata.apache.org"
>>>>> > <ar...@airavata.apache.org>
>>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>>> > To: "architecture@airavata.apache.org" <
>>>>> architecture@airavata.apache.org>,
>>>>> > dev <de...@airavata.apache.org>
>>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
>>>>>Airavata.
>>>>> >
>>>>> >> Hi All,
>>>>> >>
>>>>> >> As we all know Airavata has its own workflow language call XWF.
>>>>>When
>>>>> XWF
>>>>> >> was introduced, main focus points are interoperability and
>>>>> convertibility.
>>>>> >> But with years of experience it is convinced that above
>>>>>requirements
>>>>> are
>>>>> >> not really useful when we come to real world use cases. And XWF is
>>>>> XML
>>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>>> self.
>>>>> >> But
>>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>>> >>
>>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>>> industry
>>>>> >> and
>>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>>> come up
>>>>> >> with idea how we should improve Airavata workflow language, either
>>>>> we can
>>>>> >> improve existing XWF language, totally change to a new language
>>>>> available
>>>>> >> in industry or write a new light weight language. Basic
>>>>>requirements
>>>>> that
>>>>> >> we expect from new improvement are, high usability, flexible,
>>>>>light
>>>>> weight
>>>>> >> and real time monitoring support. As you can see above
>>>>>requirements
>>>>> are
>>>>> >> not
>>>>> >> direct comes with workflow languages but we need workflow language
>>>>> which
>>>>> >> help to support above requirements.
>>>>> >>
>>>>> >> After reading few papers and googling, initially i have come up
>>>>>with
>>>>> >> following three existing languages,
>>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>>> >> 2. WS-BPEL
>>>>> >> 3. SIDL
>>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>>> >>
>>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>>> Radical-SAGA
>>>>> >> also uses slightly modified version of SIDL. Other than above
>>>>>three
>>>>> >> languages we can come up with simple workflow language base on
>>>>> json(or
>>>>> >> yaml) which support all our requirements for some extends.
>>>>> >>
>>>>> >> It would be grate if I can get more input regarding the $Subject
>>>>> form the
>>>>> >> airavata community. You all are more than welcome to provide any
>>>>> type of
>>>>> >> suggestions.
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Shameera.
>>>>> >>
>>>>> >> 
>>>>> >>
>>>>> >> --
>>>>> >> Best Regards,
>>>>> >> Shameera Rathnayaka.
>>>>> >>
>>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> The best way to predict the future is to invent it - Alan Kay
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>>
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>
>
>-- 
>Supun Kamburugamuva
>Member, Apache Software Foundation; http://www.apache.org
>E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>Blog: http://supunk.blogspot.com

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Chris Mattmann <ch...@gmail.com>.

Have you guys considered using JCC [1] as a means
to expose the workflow API currently in Java as a
Python API?

We are exploring its use in OODT, and we have already
created a Tika [2] JCC-based python API.

Cheers,
Chris

[1] http://lucene.apache.org/pylucene/jcc/
[2] http://github.com/chrismattmann/tika-python/

------------------------
Chris Mattmann
chris.mattmann@gmail.com




-----Original Message-----
From: Supun Kamburugamuva <su...@gmail.com>
Reply-To: <de...@oodt.apache.org>
Date: Thursday, October 16, 2014 at 3:43 PM
To: dev <de...@airavata.apache.org>
Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
<sm...@apache.org>, "architecture@airavata.apache.org"
<ar...@airavata.apache.org>, "dev@oodt.apache.org"
<de...@oodt.apache.org>
Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.

>Once we had an offline discussion about the Airavata Workflow language
>(with Milinda, Saliya and Shameera). In that discussion one thing came out
>was why we need to invent a different language when a simple library like
>Python will full fill of Airavata requirements.
>
>There are many benefits in using a Python library as the API for
>controlling Airavata workflows.
>
>1. It is a library, gives the ultimate control over the execution and it
>can be simpler than any domain specific language that we can come with
>like
>XML, JSON etc
>2. Most people use python and can learn it easily than any Domain specific
>language
>3. You can easily create a Python library for Airavata because all the
>APIs
>are thrift based.
>4. If you design the constructs correctly you can plug an XBaya.
>
>Any thoughts?
>
>Thanks,
>Supun..
>
>
>On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
>wrote:
>
>> Hi Shameera,
>>
>> Why you prefer JavaScript over a language like Python?
>>
>> Thanks,
>> Supun..
>>
>> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> First of all thanks everyone for giving valuable inputs. After doing
>>>some
>>> background search and talking to different people in the University
>>>who has
>>> used different workflow languages, I myself convinced that introducing
>>>an
>>> another workflow language is not what actually they need. By changing
>>> exiting workflow language to another will not solve problems. What they
>>> asking is a easy way to construct the workflows. Indirectly what they
>>> asking for a sort of API which they can use to generate the workflows
>>>and
>>> run it. Correct me if i am wrong here.
>>>
>>> As most of above replies depict,  if we can get a simple API, as an
>>> example, for a web based application, JavaScript API would be a good
>>> solution, and probably JSON would be a good candidate for language,
>>>instead
>>> of XML.
>>>
>>> Airavata community already have started to implement web base GUI.
>>>Hence
>>> introducing a JSON base JavaScript API would be great help. WDYT?
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> alekny7@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> it is not dataflow instead focused on orchestrating REST services but
>>>> you may find it useful datapoint - we created worfklow service that
>>>>uses
>>>> natively JavaScript and JSON to describe what happens during workflow
>>>> execution:
>>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>>
>>>> HTH,
>>>>
>>>> Alek
>>>>
>>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
>>>>wrote:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>> Great to hear OODT community will be interested in adopting a JSON
>>>>> based workflow language and potentially a web based composer as well.
>>>>> Airavata previously had BPEL support initially through a home grown
>>>>> implementation [1] by Alek Slominski and later through Apache ODE
>>>>>[2]. Also
>>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>>
>>>>> I am of the same opinion that we should adopt something more modern
>>>>>as
>>>>> the challenges from scientific workflows seems to be converging with
>>>>>the
>>>>> data flow patterns in business workflows.
>>>>>
>>>>> It will be great if we can all compile a list of potential candidates
>>>>> and hack them through.
>>>>>
>>>>> Suresh
>>>>> [1] -
>>>>> 
>>>>>http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>>> [2] -
>>>>> 
>>>>>http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>>>>>time_for_eScience_workflows
>>>>> [3] -
>>>>> 
>>>>>http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>>>>>6.pdf
>>>>>
>>>>>
>>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>>
>>>>> > Hi Guys,
>>>>> >
>>>>> > I've been interested in this too - we don't per have a specific
>>>>> > OODT workflow language, but we specific workflows using XML, and
>>>>> > other configuration (we are also thinking of moving to JSON for
>>>>> > this).
>>>>> >
>>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>>> > to me.
>>>>> >
>>>>> > I wonder at the end of the day if we should adopt something more
>>>>> > modern like PIG or some other data flow type of language (PIG
>>>>> > is really neat).
>>>>> >
>>>>> > Cheers,
>>>>> > Chris
>>>>> >
>>>>> >
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Chris Mattmann, Ph.D.
>>>>> > Chief Architect
>>>>> > Instrument Software and Science Data Systems Section (398)
>>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>> > Office: 168-519, Mailstop: 168-527
>>>>> > Email: chris.a.mattmann@nasa.gov
>>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Adjunct Associate Professor, Computer Science Department
>>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > -----Original Message-----
>>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>>> > Reply-To: "architecture@airavata.apache.org"
>>>>> > <ar...@airavata.apache.org>
>>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>>> > To: "architecture@airavata.apache.org" <
>>>>> architecture@airavata.apache.org>,
>>>>> > dev <de...@airavata.apache.org>
>>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
>>>>>Airavata.
>>>>> >
>>>>> >> Hi All,
>>>>> >>
>>>>> >> As we all know Airavata has its own workflow language call XWF.
>>>>>When
>>>>> XWF
>>>>> >> was introduced, main focus points are interoperability and
>>>>> convertibility.
>>>>> >> But with years of experience it is convinced that above
>>>>>requirements
>>>>> are
>>>>> >> not really useful when we come to real world use cases. And XWF is
>>>>> XML
>>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>>> self.
>>>>> >> But
>>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>>> >>
>>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>>> industry
>>>>> >> and
>>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>>> come up
>>>>> >> with idea how we should improve Airavata workflow language, either
>>>>> we can
>>>>> >> improve existing XWF language, totally change to a new language
>>>>> available
>>>>> >> in industry or write a new light weight language. Basic
>>>>>requirements
>>>>> that
>>>>> >> we expect from new improvement are, high usability, flexible,
>>>>>light
>>>>> weight
>>>>> >> and real time monitoring support. As you can see above
>>>>>requirements
>>>>> are
>>>>> >> not
>>>>> >> direct comes with workflow languages but we need workflow language
>>>>> which
>>>>> >> help to support above requirements.
>>>>> >>
>>>>> >> After reading few papers and googling, initially i have come up
>>>>>with
>>>>> >> following three existing languages,
>>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>>> >> 2. WS-BPEL
>>>>> >> 3. SIDL
>>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>>> >>
>>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>>> Radical-SAGA
>>>>> >> also uses slightly modified version of SIDL. Other than above
>>>>>three
>>>>> >> languages we can come up with simple workflow language base on
>>>>> json(or
>>>>> >> yaml) which support all our requirements for some extends.
>>>>> >>
>>>>> >> It would be grate if I can get more input regarding the $Subject
>>>>> form the
>>>>> >> airavata community. You all are more than welcome to provide any
>>>>> type of
>>>>> >> suggestions.
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Shameera.
>>>>> >>
>>>>> >> 
>>>>> >>
>>>>> >> --
>>>>> >> Best Regards,
>>>>> >> Shameera Rathnayaka.
>>>>> >>
>>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> The best way to predict the future is to invent it - Alan Kay
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>>
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>
>
>-- 
>Supun Kamburugamuva
>Member, Apache Software Foundation; http://www.apache.org
>E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>Blog: http://supunk.blogspot.com

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Chris Mattmann <ch...@gmail.com>.

Have you guys considered using JCC [1] as a means
to expose the workflow API currently in Java as a
Python API?

We are exploring its use in OODT, and we have already
created a Tika [2] JCC-based python API.

Cheers,
Chris

[1] http://lucene.apache.org/pylucene/jcc/
[2] http://github.com/chrismattmann/tika-python/

------------------------
Chris Mattmann
chris.mattmann@gmail.com




-----Original Message-----
From: Supun Kamburugamuva <su...@gmail.com>
Reply-To: <de...@oodt.apache.org>
Date: Thursday, October 16, 2014 at 3:43 PM
To: dev <de...@airavata.apache.org>
Cc: "Alek Jones (Indiana)" <al...@gmail.com>, Suresh Marru
<sm...@apache.org>, "architecture@airavata.apache.org"
<ar...@airavata.apache.org>, "dev@oodt.apache.org"
<de...@oodt.apache.org>
Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.

>Once we had an offline discussion about the Airavata Workflow language
>(with Milinda, Saliya and Shameera). In that discussion one thing came out
>was why we need to invent a different language when a simple library like
>Python will full fill of Airavata requirements.
>
>There are many benefits in using a Python library as the API for
>controlling Airavata workflows.
>
>1. It is a library, gives the ultimate control over the execution and it
>can be simpler than any domain specific language that we can come with
>like
>XML, JSON etc
>2. Most people use python and can learn it easily than any Domain specific
>language
>3. You can easily create a Python library for Airavata because all the
>APIs
>are thrift based.
>4. If you design the constructs correctly you can plug an XBaya.
>
>Any thoughts?
>
>Thanks,
>Supun..
>
>
>On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
>wrote:
>
>> Hi Shameera,
>>
>> Why you prefer JavaScript over a language like Python?
>>
>> Thanks,
>> Supun..
>>
>> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> First of all thanks everyone for giving valuable inputs. After doing
>>>some
>>> background search and talking to different people in the University
>>>who has
>>> used different workflow languages, I myself convinced that introducing
>>>an
>>> another workflow language is not what actually they need. By changing
>>> exiting workflow language to another will not solve problems. What they
>>> asking is a easy way to construct the workflows. Indirectly what they
>>> asking for a sort of API which they can use to generate the workflows
>>>and
>>> run it. Correct me if i am wrong here.
>>>
>>> As most of above replies depict,  if we can get a simple API, as an
>>> example, for a web based application, JavaScript API would be a good
>>> solution, and probably JSON would be a good candidate for language,
>>>instead
>>> of XML.
>>>
>>> Airavata community already have started to implement web base GUI.
>>>Hence
>>> introducing a JSON base JavaScript API would be great help. WDYT?
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> alekny7@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> it is not dataflow instead focused on orchestrating REST services but
>>>> you may find it useful datapoint - we created worfklow service that
>>>>uses
>>>> natively JavaScript and JSON to describe what happens during workflow
>>>> execution:
>>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>>
>>>> HTH,
>>>>
>>>> Alek
>>>>
>>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
>>>>wrote:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>> Great to hear OODT community will be interested in adopting a JSON
>>>>> based workflow language and potentially a web based composer as well.
>>>>> Airavata previously had BPEL support initially through a home grown
>>>>> implementation [1] by Alek Slominski and later through Apache ODE
>>>>>[2]. Also
>>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>>
>>>>> I am of the same opinion that we should adopt something more modern
>>>>>as
>>>>> the challenges from scientific workflows seems to be converging with
>>>>>the
>>>>> data flow patterns in business workflows.
>>>>>
>>>>> It will be great if we can all compile a list of potential candidates
>>>>> and hack them through.
>>>>>
>>>>> Suresh
>>>>> [1] -
>>>>> 
>>>>>http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>>> [2] -
>>>>> 
>>>>>http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>>>>>time_for_eScience_workflows
>>>>> [3] -
>>>>> 
>>>>>http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>>>>>6.pdf
>>>>>
>>>>>
>>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>>
>>>>> > Hi Guys,
>>>>> >
>>>>> > I've been interested in this too - we don't per have a specific
>>>>> > OODT workflow language, but we specific workflows using XML, and
>>>>> > other configuration (we are also thinking of moving to JSON for
>>>>> > this).
>>>>> >
>>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>>> > to me.
>>>>> >
>>>>> > I wonder at the end of the day if we should adopt something more
>>>>> > modern like PIG or some other data flow type of language (PIG
>>>>> > is really neat).
>>>>> >
>>>>> > Cheers,
>>>>> > Chris
>>>>> >
>>>>> >
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Chris Mattmann, Ph.D.
>>>>> > Chief Architect
>>>>> > Instrument Software and Science Data Systems Section (398)
>>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>> > Office: 168-519, Mailstop: 168-527
>>>>> > Email: chris.a.mattmann@nasa.gov
>>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Adjunct Associate Professor, Computer Science Department
>>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > -----Original Message-----
>>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>>> > Reply-To: "architecture@airavata.apache.org"
>>>>> > <ar...@airavata.apache.org>
>>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>>> > To: "architecture@airavata.apache.org" <
>>>>> architecture@airavata.apache.org>,
>>>>> > dev <de...@airavata.apache.org>
>>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
>>>>>Airavata.
>>>>> >
>>>>> >> Hi All,
>>>>> >>
>>>>> >> As we all know Airavata has its own workflow language call XWF.
>>>>>When
>>>>> XWF
>>>>> >> was introduced, main focus points are interoperability and
>>>>> convertibility.
>>>>> >> But with years of experience it is convinced that above
>>>>>requirements
>>>>> are
>>>>> >> not really useful when we come to real world use cases. And XWF is
>>>>> XML
>>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>>> self.
>>>>> >> But
>>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>>> >>
>>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>>> industry
>>>>> >> and
>>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>>> come up
>>>>> >> with idea how we should improve Airavata workflow language, either
>>>>> we can
>>>>> >> improve existing XWF language, totally change to a new language
>>>>> available
>>>>> >> in industry or write a new light weight language. Basic
>>>>>requirements
>>>>> that
>>>>> >> we expect from new improvement are, high usability, flexible,
>>>>>light
>>>>> weight
>>>>> >> and real time monitoring support. As you can see above
>>>>>requirements
>>>>> are
>>>>> >> not
>>>>> >> direct comes with workflow languages but we need workflow language
>>>>> which
>>>>> >> help to support above requirements.
>>>>> >>
>>>>> >> After reading few papers and googling, initially i have come up
>>>>>with
>>>>> >> following three existing languages,
>>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>>> >> 2. WS-BPEL
>>>>> >> 3. SIDL
>>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>>> >>
>>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>>> Radical-SAGA
>>>>> >> also uses slightly modified version of SIDL. Other than above
>>>>>three
>>>>> >> languages we can come up with simple workflow language base on
>>>>> json(or
>>>>> >> yaml) which support all our requirements for some extends.
>>>>> >>
>>>>> >> It would be grate if I can get more input regarding the $Subject
>>>>> form the
>>>>> >> airavata community. You all are more than welcome to provide any
>>>>> type of
>>>>> >> suggestions.
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Shameera.
>>>>> >>
>>>>> >> 
>>>>> >>
>>>>> >> --
>>>>> >> Best Regards,
>>>>> >> Shameera Rathnayaka.
>>>>> >>
>>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> The best way to predict the future is to invent it - Alan Kay
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>>
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>
>
>-- 
>Supun Kamburugamuva
>Member, Apache Software Foundation; http://www.apache.org
>E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>Blog: http://supunk.blogspot.com

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Supun Kamburugamuva <su...@gmail.com>.

Once we had an offline discussion about the Airavata Workflow language
(with Milinda, Saliya and Shameera). In that discussion one thing came out
was why we need to invent a different language when a simple library like
Python will full fill of Airavata requirements.

There are many benefits in using a Python library as the API for
controlling Airavata workflows.

1. It is a library, gives the ultimate control over the execution and it
can be simpler than any domain specific language that we can come with like
XML, JSON etc
2. Most people use python and can learn it easily than any Domain specific
language
3. You can easily create a Python library for Airavata because all the APIs
are thrift based.
4. If you design the constructs correctly you can plug an XBaya.

Any thoughts?

Thanks,
Supun..


On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:

> Hi Shameera,
>
> Why you prefer JavaScript over a language like Python?
>
> Thanks,
> Supun..
>
> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi,
>>
>> First of all thanks everyone for giving valuable inputs. After doing some
>> background search and talking to different people in the University who has
>> used different workflow languages, I myself convinced that introducing an
>> another workflow language is not what actually they need. By changing
>> exiting workflow language to another will not solve problems. What they
>> asking is a easy way to construct the workflows. Indirectly what they
>> asking for a sort of API which they can use to generate the workflows and
>> run it. Correct me if i am wrong here.
>>
>> As most of above replies depict,  if we can get a simple API, as an
>> example, for a web based application, JavaScript API would be a good
>> solution, and probably JSON would be a good candidate for language, instead
>> of XML.
>>
>> Airavata community already have started to implement web base GUI. Hence
>> introducing a JSON base JavaScript API would be great help. WDYT?
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> alekny7@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> it is not dataflow instead focused on orchestrating REST services but
>>> you may find it useful datapoint - we created worfklow service that uses
>>> natively JavaScript and JSON to describe what happens during workflow
>>> execution:
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>
>>> HTH,
>>>
>>> Alek
>>>
>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>>>
>>>> Hi Chris,
>>>>
>>>> Great to hear OODT community will be interested in adopting a JSON
>>>> based workflow language and potentially a web based composer as well.
>>>> Airavata previously had BPEL support initially through a home grown
>>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>
>>>> I am of the same opinion that we should adopt something more modern as
>>>> the challenges from scientific workflows seems to be converging with the
>>>> data flow patterns in business workflows.
>>>>
>>>> It will be great if we can all compile a list of potential candidates
>>>> and hack them through.
>>>>
>>>> Suresh
>>>> [1] -
>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>> [2] -
>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>>> [3] -
>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>>
>>>>
>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>
>>>> > Hi Guys,
>>>> >
>>>> > I've been interested in this too - we don't per have a specific
>>>> > OODT workflow language, but we specific workflows using XML, and
>>>> > other configuration (we are also thinking of moving to JSON for
>>>> > this).
>>>> >
>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>> > to me.
>>>> >
>>>> > I wonder at the end of the day if we should adopt something more
>>>> > modern like PIG or some other data flow type of language (PIG
>>>> > is really neat).
>>>> >
>>>> > Cheers,
>>>> > Chris
>>>> >
>>>> >
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Chris Mattmann, Ph.D.
>>>> > Chief Architect
>>>> > Instrument Software and Science Data Systems Section (398)
>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> > Office: 168-519, Mailstop: 168-527
>>>> > Email: chris.a.mattmann@nasa.gov
>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Adjunct Associate Professor, Computer Science Department
>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > -----Original Message-----
>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>> > Reply-To: "architecture@airavata.apache.org"
>>>> > <ar...@airavata.apache.org>
>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>> > To: "architecture@airavata.apache.org" <
>>>> architecture@airavata.apache.org>,
>>>> > dev <de...@airavata.apache.org>
>>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>>> >
>>>> >> Hi All,
>>>> >>
>>>> >> As we all know Airavata has its own workflow language call XWF. When
>>>> XWF
>>>> >> was introduced, main focus points are interoperability and
>>>> convertibility.
>>>> >> But with years of experience it is convinced that above requirements
>>>> are
>>>> >> not really useful when we come to real world use cases. And XWF is
>>>> XML
>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>> self.
>>>> >> But
>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>> >>
>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>> industry
>>>> >> and
>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>> come up
>>>> >> with idea how we should improve Airavata workflow language, either
>>>> we can
>>>> >> improve existing XWF language, totally change to a new language
>>>> available
>>>> >> in industry or write a new light weight language. Basic requirements
>>>> that
>>>> >> we expect from new improvement are, high usability, flexible, light
>>>> weight
>>>> >> and real time monitoring support. As you can see above requirements
>>>> are
>>>> >> not
>>>> >> direct comes with workflow languages but we need workflow language
>>>> which
>>>> >> help to support above requirements.
>>>> >>
>>>> >> After reading few papers and googling, initially i have come up with
>>>> >> following three existing languages,
>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>> >> 2. WS-BPEL
>>>> >> 3. SIDL
>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>> >>
>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>> Radical-SAGA
>>>> >> also uses slightly modified version of SIDL. Other than above three
>>>> >> languages we can come up with simple workflow language base on
>>>> json(or
>>>> >> yaml) which support all our requirements for some extends.
>>>> >>
>>>> >> It would be grate if I can get more input regarding the $Subject
>>>> form the
>>>> >> airavata community. You all are more than welcome to provide any
>>>> type of
>>>> >> suggestions.
>>>> >>
>>>> >> Thanks,
>>>> >> Shameera.
>>>> >>
>>>> >> 
>>>> >>
>>>> >> --
>>>> >> Best Regards,
>>>> >> Shameera Rathnayaka.
>>>> >>
>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>
>>>>
>>>
>>>
>>> --
>>> The best way to predict the future is to invent it - Alan Kay
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>


-- 
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by K Yoshimoto <ke...@sdsc.edu>.

I prefer python myself, but if the API is clean, it should be
usable from a large number of languages on the client side.

On Thu, Oct 16, 2014 at 06:46:36PM -0400, Shameera Rathnayaka wrote:
> Hi Supun,
> 
> Considering we are going to provide web base GUI support and  JavaScript is
> a well-known client side language, I would select JavaScript over Python. I
> am not much familiar with Python, so would like to know the advantages we
> get by selecting python here.
> 
> 
> Thanks,
> Shameera.
> 
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
> 
> > Hi Shameera,
> >
> > Why you prefer JavaScript over a language like Python?
> >
> > Thanks,
> > Supun..
> >
> > On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > shameerainfo@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> First of all thanks everyone for giving valuable inputs. After doing some
> >> background search and talking to different people in the University who has
> >> used different workflow languages, I myself convinced that introducing an
> >> another workflow language is not what actually they need. By changing
> >> exiting workflow language to another will not solve problems. What they
> >> asking is a easy way to construct the workflows. Indirectly what they
> >> asking for a sort of API which they can use to generate the workflows and
> >> run it. Correct me if i am wrong here.
> >>
> >> As most of above replies depict,  if we can get a simple API, as an
> >> example, for a web based application, JavaScript API would be a good
> >> solution, and probably JSON would be a good candidate for language, instead
> >> of XML.
> >>
> >> Airavata community already have started to implement web base GUI. Hence
> >> introducing a JSON base JavaScript API would be great help. WDYT?
> >>
> >> Thanks,
> >> Shameera.
> >>
> >>
> >> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >> alekny7@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> it is not dataflow instead focused on orchestrating REST services but
> >>> you may find it useful datapoint - we created worfklow service that uses
> >>> natively JavaScript and JSON to describe what happens during workflow
> >>> execution:
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>
> >>> HTH,
> >>>
> >>> Alek
> >>>
> >>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
> >>>
> >>>> Hi Chris,
> >>>>
> >>>> Great to hear OODT community will be interested in adopting a JSON
> >>>> based workflow language and potentially a web based composer as well.
> >>>> Airavata previously had BPEL support initially through a home grown
> >>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
> >>>> a white paper [3] by Alek on this topic is an interesting read.
> >>>>
> >>>> I am of the same opinion that we should adopt something more modern as
> >>>> the challenges from scientific workflows seems to be converging with the
> >>>> data flow patterns in business workflows.
> >>>>
> >>>> It will be great if we can all compile a list of potential candidates
> >>>> and hack them through.
> >>>>
> >>>> Suresh
> >>>> [1] -
> >>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>>> [2] -
> >>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
> >>>> [3] -
> >>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
> >>>>
> >>>>
> >>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>>>
> >>>> > Hi Guys,
> >>>> >
> >>>> > I've been interested in this too - we don't per have a specific
> >>>> > OODT workflow language, but we specific workflows using XML, and
> >>>> > other configuration (we are also thinking of moving to JSON for
> >>>> > this).
> >>>> >
> >>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> >>>> > to me.
> >>>> >
> >>>> > I wonder at the end of the day if we should adopt something more
> >>>> > modern like PIG or some other data flow type of language (PIG
> >>>> > is really neat).
> >>>> >
> >>>> > Cheers,
> >>>> > Chris
> >>>> >
> >>>> >
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Chris Mattmann, Ph.D.
> >>>> > Chief Architect
> >>>> > Instrument Software and Science Data Systems Section (398)
> >>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>>> > Office: 168-519, Mailstop: 168-527
> >>>> > Email: chris.a.mattmann@nasa.gov
> >>>> > WWW:  http://sunset.usc.edu/~mattmann/
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Adjunct Associate Professor, Computer Science Department
> >>>> > University of Southern California, Los Angeles, CA 90089 USA
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > -----Original Message-----
> >>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>>> > Reply-To: "architecture@airavata.apache.org"
> >>>> > <ar...@airavata.apache.org>
> >>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>>> > To: "architecture@airavata.apache.org" <
> >>>> architecture@airavata.apache.org>,
> >>>> > dev <de...@airavata.apache.org>
> >>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> >>>> >
> >>>> >> Hi All,
> >>>> >>
> >>>> >> As we all know Airavata has its own workflow language call XWF. When
> >>>> XWF
> >>>> >> was introduced, main focus points are interoperability and
> >>>> convertibility.
> >>>> >> But with years of experience it is convinced that above requirements
> >>>> are
> >>>> >> not really useful when we come to real world use cases. And XWF is
> >>>> XML
> >>>> >> based bulky language where we attache WSDLs and Workflow image it
> >>>> self.
> >>>> >> But
> >>>> >> with the recent changes WSDL part is being removed from XWF.
> >>>> >>
> >>>> >> It is worth to evaluate handy Scientific workflow languages in
> >>>> industry
> >>>> >> and
> >>>> >> find out pros and cons, at the end of this evaluation we need to
> >>>> come up
> >>>> >> with idea how we should improve Airavata workflow language, either
> >>>> we can
> >>>> >> improve existing XWF language, totally change to a new language
> >>>> available
> >>>> >> in industry or write a new light weight language. Basic requirements
> >>>> that
> >>>> >> we expect from new improvement are, high usability, flexible, light
> >>>> weight
> >>>> >> and real time monitoring support. As you can see above requirements
> >>>> are
> >>>> >> not
> >>>> >> direct comes with workflow languages but we need workflow language
> >>>> which
> >>>> >> help to support above requirements.
> >>>> >>
> >>>> >> After reading few papers and googling, initially i have come up with
> >>>> >> following three existing languages,
> >>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>>> >> 2. WS-BPEL
> >>>> >> 3. SIDL
> >>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
> >>>> >>
> >>>> >> In my opinion SIDL is more familiar with scientific domain,
> >>>> Radical-SAGA
> >>>> >> also uses slightly modified version of SIDL. Other than above three
> >>>> >> languages we can come up with simple workflow language base on
> >>>> json(or
> >>>> >> yaml) which support all our requirements for some extends.
> >>>> >>
> >>>> >> It would be grate if I can get more input regarding the $Subject
> >>>> form the
> >>>> >> airavata community. You all are more than welcome to provide any
> >>>> type of
> >>>> >> suggestions.
> >>>> >>
> >>>> >> Thanks,
> >>>> >> Shameera.
> >>>> >>
> >>>> >> 
> >>>> >>
> >>>> >> --
> >>>> >> Best Regards,
> >>>> >> Shameera Rathnayaka.
> >>>> >>
> >>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> The best way to predict the future is to invent it - Alan Kay
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Shameera Rathnayaka.
> >>
> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
> >
> 
> 
> -- 
> Best Regards,
> Shameera Rathnayaka.
> 
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Nipurn Doshi <ni...@umail.iu.edu>.

Hi,

How about I suggest a third option where I have the current code setup
which has access to thrift classes, airavata interface classes and a
complete framework ( Laravel ) that supports rest apis to generate json
data that we are looking at?

I am suggesting this only because I have done this in previous projects I
have worked on using Laravel.

Although, it's in PHP.
 On 16-Oct-2014 6:47 pm, "Shameera Rathnayaka" <sh...@gmail.com>
wrote:

> Hi Supun,
>
> Considering we are going to provide web base GUI support and  JavaScript
> is a well-known client side language, I would select JavaScript over
> Python. I am not much familiar with Python, so would like to know the
> advantages we get by selecting python here.
>
>
> Thanks,
> Shameera.
>
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
>> Hi Shameera,
>>
>> Why you prefer JavaScript over a language like Python?
>>
>> Thanks,
>> Supun..
>>
>> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> First of all thanks everyone for giving valuable inputs. After doing
>>> some background search and talking to different people in the University
>>> who has used different workflow languages, I myself convinced that
>>> introducing an another workflow language is not what actually they need. By
>>> changing exiting workflow language to another will not solve problems. What
>>> they asking is a easy way to construct the workflows. Indirectly what they
>>> asking for a sort of API which they can use to generate the workflows and
>>> run it. Correct me if i am wrong here.
>>>
>>> As most of above replies depict,  if we can get a simple API, as an
>>> example, for a web based application, JavaScript API would be a good
>>> solution, and probably JSON would be a good candidate for language, instead
>>> of XML.
>>>
>>> Airavata community already have started to implement web base GUI. Hence
>>> introducing a JSON base JavaScript API would be great help. WDYT?
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> alekny7@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> it is not dataflow instead focused on orchestrating REST services but
>>>> you may find it useful datapoint - we created worfklow service that uses
>>>> natively JavaScript and JSON to describe what happens during workflow
>>>> execution:
>>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>>
>>>> HTH,
>>>>
>>>> Alek
>>>>
>>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>> Great to hear OODT community will be interested in adopting a JSON
>>>>> based workflow language and potentially a web based composer as well.
>>>>> Airavata previously had BPEL support initially through a home grown
>>>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
>>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>>
>>>>> I am of the same opinion that we should adopt something more modern as
>>>>> the challenges from scientific workflows seems to be converging with the
>>>>> data flow patterns in business workflows.
>>>>>
>>>>> It will be great if we can all compile a list of potential candidates
>>>>> and hack them through.
>>>>>
>>>>> Suresh
>>>>> [1] -
>>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>>> [2] -
>>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>>>> [3] -
>>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>>>
>>>>>
>>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>>
>>>>> > Hi Guys,
>>>>> >
>>>>> > I've been interested in this too - we don't per have a specific
>>>>> > OODT workflow language, but we specific workflows using XML, and
>>>>> > other configuration (we are also thinking of moving to JSON for
>>>>> > this).
>>>>> >
>>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>>> > to me.
>>>>> >
>>>>> > I wonder at the end of the day if we should adopt something more
>>>>> > modern like PIG or some other data flow type of language (PIG
>>>>> > is really neat).
>>>>> >
>>>>> > Cheers,
>>>>> > Chris
>>>>> >
>>>>> >
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Chris Mattmann, Ph.D.
>>>>> > Chief Architect
>>>>> > Instrument Software and Science Data Systems Section (398)
>>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>> > Office: 168-519, Mailstop: 168-527
>>>>> > Email: chris.a.mattmann@nasa.gov
>>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Adjunct Associate Professor, Computer Science Department
>>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > -----Original Message-----
>>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>>> > Reply-To: "architecture@airavata.apache.org"
>>>>> > <ar...@airavata.apache.org>
>>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>>> > To: "architecture@airavata.apache.org" <
>>>>> architecture@airavata.apache.org>,
>>>>> > dev <de...@airavata.apache.org>
>>>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>>>> >
>>>>> >> Hi All,
>>>>> >>
>>>>> >> As we all know Airavata has its own workflow language call XWF.
>>>>> When XWF
>>>>> >> was introduced, main focus points are interoperability and
>>>>> convertibility.
>>>>> >> But with years of experience it is convinced that above
>>>>> requirements are
>>>>> >> not really useful when we come to real world use cases. And XWF is
>>>>> XML
>>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>>> self.
>>>>> >> But
>>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>>> >>
>>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>>> industry
>>>>> >> and
>>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>>> come up
>>>>> >> with idea how we should improve Airavata workflow language, either
>>>>> we can
>>>>> >> improve existing XWF language, totally change to a new language
>>>>> available
>>>>> >> in industry or write a new light weight language. Basic
>>>>> requirements that
>>>>> >> we expect from new improvement are, high usability, flexible, light
>>>>> weight
>>>>> >> and real time monitoring support. As you can see above requirements
>>>>> are
>>>>> >> not
>>>>> >> direct comes with workflow languages but we need workflow language
>>>>> which
>>>>> >> help to support above requirements.
>>>>> >>
>>>>> >> After reading few papers and googling, initially i have come up with
>>>>> >> following three existing languages,
>>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>>> >> 2. WS-BPEL
>>>>> >> 3. SIDL
>>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>>> >>
>>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>>> Radical-SAGA
>>>>> >> also uses slightly modified version of SIDL. Other than above three
>>>>> >> languages we can come up with simple workflow language base on
>>>>> json(or
>>>>> >> yaml) which support all our requirements for some extends.
>>>>> >>
>>>>> >> It would be grate if I can get more input regarding the $Subject
>>>>> form the
>>>>> >> airavata community. You all are more than welcome to provide any
>>>>> type of
>>>>> >> suggestions.
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Shameera.
>>>>> >>
>>>>> >> 
>>>>> >>
>>>>> >> --
>>>>> >> Best Regards,
>>>>> >> Shameera Rathnayaka.
>>>>> >>
>>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> The best way to predict the future is to invent it - Alan Kay
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>>
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by K Yoshimoto <ke...@sdsc.edu>.

I prefer python myself, but if the API is clean, it should be
usable from a large number of languages on the client side.

On Thu, Oct 16, 2014 at 06:46:36PM -0400, Shameera Rathnayaka wrote:
> Hi Supun,
> 
> Considering we are going to provide web base GUI support and  JavaScript is
> a well-known client side language, I would select JavaScript over Python. I
> am not much familiar with Python, so would like to know the advantages we
> get by selecting python here.
> 
> 
> Thanks,
> Shameera.
> 
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
> 
> > Hi Shameera,
> >
> > Why you prefer JavaScript over a language like Python?
> >
> > Thanks,
> > Supun..
> >
> > On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > shameerainfo@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> First of all thanks everyone for giving valuable inputs. After doing some
> >> background search and talking to different people in the University who has
> >> used different workflow languages, I myself convinced that introducing an
> >> another workflow language is not what actually they need. By changing
> >> exiting workflow language to another will not solve problems. What they
> >> asking is a easy way to construct the workflows. Indirectly what they
> >> asking for a sort of API which they can use to generate the workflows and
> >> run it. Correct me if i am wrong here.
> >>
> >> As most of above replies depict,  if we can get a simple API, as an
> >> example, for a web based application, JavaScript API would be a good
> >> solution, and probably JSON would be a good candidate for language, instead
> >> of XML.
> >>
> >> Airavata community already have started to implement web base GUI. Hence
> >> introducing a JSON base JavaScript API would be great help. WDYT?
> >>
> >> Thanks,
> >> Shameera.
> >>
> >>
> >> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >> alekny7@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> it is not dataflow instead focused on orchestrating REST services but
> >>> you may find it useful datapoint - we created worfklow service that uses
> >>> natively JavaScript and JSON to describe what happens during workflow
> >>> execution:
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>
> >>> HTH,
> >>>
> >>> Alek
> >>>
> >>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
> >>>
> >>>> Hi Chris,
> >>>>
> >>>> Great to hear OODT community will be interested in adopting a JSON
> >>>> based workflow language and potentially a web based composer as well.
> >>>> Airavata previously had BPEL support initially through a home grown
> >>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
> >>>> a white paper [3] by Alek on this topic is an interesting read.
> >>>>
> >>>> I am of the same opinion that we should adopt something more modern as
> >>>> the challenges from scientific workflows seems to be converging with the
> >>>> data flow patterns in business workflows.
> >>>>
> >>>> It will be great if we can all compile a list of potential candidates
> >>>> and hack them through.
> >>>>
> >>>> Suresh
> >>>> [1] -
> >>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>>> [2] -
> >>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
> >>>> [3] -
> >>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
> >>>>
> >>>>
> >>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>>>
> >>>> > Hi Guys,
> >>>> >
> >>>> > I've been interested in this too - we don't per have a specific
> >>>> > OODT workflow language, but we specific workflows using XML, and
> >>>> > other configuration (we are also thinking of moving to JSON for
> >>>> > this).
> >>>> >
> >>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> >>>> > to me.
> >>>> >
> >>>> > I wonder at the end of the day if we should adopt something more
> >>>> > modern like PIG or some other data flow type of language (PIG
> >>>> > is really neat).
> >>>> >
> >>>> > Cheers,
> >>>> > Chris
> >>>> >
> >>>> >
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Chris Mattmann, Ph.D.
> >>>> > Chief Architect
> >>>> > Instrument Software and Science Data Systems Section (398)
> >>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>>> > Office: 168-519, Mailstop: 168-527
> >>>> > Email: chris.a.mattmann@nasa.gov
> >>>> > WWW:  http://sunset.usc.edu/~mattmann/
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Adjunct Associate Professor, Computer Science Department
> >>>> > University of Southern California, Los Angeles, CA 90089 USA
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > -----Original Message-----
> >>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>>> > Reply-To: "architecture@airavata.apache.org"
> >>>> > <ar...@airavata.apache.org>
> >>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>>> > To: "architecture@airavata.apache.org" <
> >>>> architecture@airavata.apache.org>,
> >>>> > dev <de...@airavata.apache.org>
> >>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> >>>> >
> >>>> >> Hi All,
> >>>> >>
> >>>> >> As we all know Airavata has its own workflow language call XWF. When
> >>>> XWF
> >>>> >> was introduced, main focus points are interoperability and
> >>>> convertibility.
> >>>> >> But with years of experience it is convinced that above requirements
> >>>> are
> >>>> >> not really useful when we come to real world use cases. And XWF is
> >>>> XML
> >>>> >> based bulky language where we attache WSDLs and Workflow image it
> >>>> self.
> >>>> >> But
> >>>> >> with the recent changes WSDL part is being removed from XWF.
> >>>> >>
> >>>> >> It is worth to evaluate handy Scientific workflow languages in
> >>>> industry
> >>>> >> and
> >>>> >> find out pros and cons, at the end of this evaluation we need to
> >>>> come up
> >>>> >> with idea how we should improve Airavata workflow language, either
> >>>> we can
> >>>> >> improve existing XWF language, totally change to a new language
> >>>> available
> >>>> >> in industry or write a new light weight language. Basic requirements
> >>>> that
> >>>> >> we expect from new improvement are, high usability, flexible, light
> >>>> weight
> >>>> >> and real time monitoring support. As you can see above requirements
> >>>> are
> >>>> >> not
> >>>> >> direct comes with workflow languages but we need workflow language
> >>>> which
> >>>> >> help to support above requirements.
> >>>> >>
> >>>> >> After reading few papers and googling, initially i have come up with
> >>>> >> following three existing languages,
> >>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>>> >> 2. WS-BPEL
> >>>> >> 3. SIDL
> >>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
> >>>> >>
> >>>> >> In my opinion SIDL is more familiar with scientific domain,
> >>>> Radical-SAGA
> >>>> >> also uses slightly modified version of SIDL. Other than above three
> >>>> >> languages we can come up with simple workflow language base on
> >>>> json(or
> >>>> >> yaml) which support all our requirements for some extends.
> >>>> >>
> >>>> >> It would be grate if I can get more input regarding the $Subject
> >>>> form the
> >>>> >> airavata community. You all are more than welcome to provide any
> >>>> type of
> >>>> >> suggestions.
> >>>> >>
> >>>> >> Thanks,
> >>>> >> Shameera.
> >>>> >>
> >>>> >> 
> >>>> >>
> >>>> >> --
> >>>> >> Best Regards,
> >>>> >> Shameera Rathnayaka.
> >>>> >>
> >>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> The best way to predict the future is to invent it - Alan Kay
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Shameera Rathnayaka.
> >>
> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
> >
> 
> 
> -- 
> Best Regards,
> Shameera Rathnayaka.
> 
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Supun Kamburugamuva <su...@gmail.com>.

Hi Shameera,

To create a web based UI I don't see an immediate advantage of using
JavaScript over a Python API. Because Airavata's API is Thrift (not any
language based API) you can generate code for any language (including
JavaScript). Anyway most probably you will not use direct Airavata API
calls for creating a web UI. I'm also not sure what you exactly mean by
JavaScript language as well. If you explain in more detail that would be
really helpful.

If you just give a Python API to create workflows, you really don't even
need XBaya to create/run workflows. You can create them as scripts and run
them from command line. You can debug your workflows written in Python
using normal python debuggers. You can generate the XBaya workflow by using
the structure you create in Python as well. Then you can use XBaya to
monitor the workflow. These things are hard to do with an JavaScript API.
As a language also I would pick Python over JavaScript.

Thanks,
Supun..


On Thu, Oct 16, 2014 at 6:46 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:

> Hi Supun,
>
> Considering we are going to provide web base GUI support and  JavaScript is
> a well-known client side language, I would select JavaScript over Python. I
> am not much familiar with Python, so would like to know the advantages we
> get by selecting python here.
>
>
> Thanks,
> Shameera.
>
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
> > Hi Shameera,
> >
> > Why you prefer JavaScript over a language like Python?
> >
> > Thanks,
> > Supun..
> >
> > On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > shameerainfo@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> First of all thanks everyone for giving valuable inputs. After doing
> some
> >> background search and talking to different people in the University who
> has
> >> used different workflow languages, I myself convinced that introducing
> an
> >> another workflow language is not what actually they need. By changing
> >> exiting workflow language to another will not solve problems. What they
> >> asking is a easy way to construct the workflows. Indirectly what they
> >> asking for a sort of API which they can use to generate the workflows
> and
> >> run it. Correct me if i am wrong here.
> >>
> >> As most of above replies depict,  if we can get a simple API, as an
> >> example, for a web based application, JavaScript API would be a good
> >> solution, and probably JSON would be a good candidate for language,
> instead
> >> of XML.
> >>
> >> Airavata community already have started to implement web base GUI. Hence
> >> introducing a JSON base JavaScript API would be great help. WDYT?
> >>
> >> Thanks,
> >> Shameera.
> >>
> >>
> >> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >> alekny7@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> it is not dataflow instead focused on orchestrating REST services but
> >>> you may find it useful datapoint - we created worfklow service that
> uses
> >>> natively JavaScript and JSON to describe what happens during workflow
> >>> execution:
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>
> >>> HTH,
> >>>
> >>> Alek
> >>>
> >>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> wrote:
> >>>
> >>>> Hi Chris,
> >>>>
> >>>> Great to hear OODT community will be interested in adopting a JSON
> >>>> based workflow language and potentially a web based composer as well.
> >>>> Airavata previously had BPEL support initially through a home grown
> >>>> implementation [1] by Alek Slominski and later through Apache ODE
> [2]. Also
> >>>> a white paper [3] by Alek on this topic is an interesting read.
> >>>>
> >>>> I am of the same opinion that we should adopt something more modern as
> >>>> the challenges from scientific workflows seems to be converging with
> the
> >>>> data flow patterns in business workflows.
> >>>>
> >>>> It will be great if we can all compile a list of potential candidates
> >>>> and hack them through.
> >>>>
> >>>> Suresh
> >>>> [1] -
> >>>>
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>>> [2] -
> >>>>
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
> >>>> [3] -
> >>>>
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
> >>>>
> >>>>
> >>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>>>
> >>>> > Hi Guys,
> >>>> >
> >>>> > I've been interested in this too - we don't per have a specific
> >>>> > OODT workflow language, but we specific workflows using XML, and
> >>>> > other configuration (we are also thinking of moving to JSON for
> >>>> > this).
> >>>> >
> >>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> >>>> > to me.
> >>>> >
> >>>> > I wonder at the end of the day if we should adopt something more
> >>>> > modern like PIG or some other data flow type of language (PIG
> >>>> > is really neat).
> >>>> >
> >>>> > Cheers,
> >>>> > Chris
> >>>> >
> >>>> >
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Chris Mattmann, Ph.D.
> >>>> > Chief Architect
> >>>> > Instrument Software and Science Data Systems Section (398)
> >>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>>> > Office: 168-519, Mailstop: 168-527
> >>>> > Email: chris.a.mattmann@nasa.gov
> >>>> > WWW:  http://sunset.usc.edu/~mattmann/
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Adjunct Associate Professor, Computer Science Department
> >>>> > University of Southern California, Los Angeles, CA 90089 USA
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > -----Original Message-----
> >>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>>> > Reply-To: "architecture@airavata.apache.org"
> >>>> > <ar...@airavata.apache.org>
> >>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>>> > To: "architecture@airavata.apache.org" <
> >>>> architecture@airavata.apache.org>,
> >>>> > dev <de...@airavata.apache.org>
> >>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> Airavata.
> >>>> >
> >>>> >> Hi All,
> >>>> >>
> >>>> >> As we all know Airavata has its own workflow language call XWF.
> When
> >>>> XWF
> >>>> >> was introduced, main focus points are interoperability and
> >>>> convertibility.
> >>>> >> But with years of experience it is convinced that above
> requirements
> >>>> are
> >>>> >> not really useful when we come to real world use cases. And XWF is
> >>>> XML
> >>>> >> based bulky language where we attache WSDLs and Workflow image it
> >>>> self.
> >>>> >> But
> >>>> >> with the recent changes WSDL part is being removed from XWF.
> >>>> >>
> >>>> >> It is worth to evaluate handy Scientific workflow languages in
> >>>> industry
> >>>> >> and
> >>>> >> find out pros and cons, at the end of this evaluation we need to
> >>>> come up
> >>>> >> with idea how we should improve Airavata workflow language, either
> >>>> we can
> >>>> >> improve existing XWF language, totally change to a new language
> >>>> available
> >>>> >> in industry or write a new light weight language. Basic
> requirements
> >>>> that
> >>>> >> we expect from new improvement are, high usability, flexible, light
> >>>> weight
> >>>> >> and real time monitoring support. As you can see above requirements
> >>>> are
> >>>> >> not
> >>>> >> direct comes with workflow languages but we need workflow language
> >>>> which
> >>>> >> help to support above requirements.
> >>>> >>
> >>>> >> After reading few papers and googling, initially i have come up
> with
> >>>> >> following three existing languages,
> >>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>>> >> 2. WS-BPEL
> >>>> >> 3. SIDL
> >>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
> >>>> >>
> >>>> >> In my opinion SIDL is more familiar with scientific domain,
> >>>> Radical-SAGA
> >>>> >> also uses slightly modified version of SIDL. Other than above three
> >>>> >> languages we can come up with simple workflow language base on
> >>>> json(or
> >>>> >> yaml) which support all our requirements for some extends.
> >>>> >>
> >>>> >> It would be grate if I can get more input regarding the $Subject
> >>>> form the
> >>>> >> airavata community. You all are more than welcome to provide any
> >>>> type of
> >>>> >> suggestions.
> >>>> >>
> >>>> >> Thanks,
> >>>> >> Shameera.
> >>>> >>
> >>>> >> 
> >>>> >>
> >>>> >> --
> >>>> >> Best Regards,
> >>>> >> Shameera Rathnayaka.
> >>>> >>
> >>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> The best way to predict the future is to invent it - Alan Kay
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Shameera Rathnayaka.
> >>
> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>



-- 
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Supun Kamburugamuva <su...@gmail.com>.

Hi Shameera,

To create a web based UI I don't see an immediate advantage of using
JavaScript over a Python API. Because Airavata's API is Thrift (not any
language based API) you can generate code for any language (including
JavaScript). Anyway most probably you will not use direct Airavata API
calls for creating a web UI. I'm also not sure what you exactly mean by
JavaScript language as well. If you explain in more detail that would be
really helpful.

If you just give a Python API to create workflows, you really don't even
need XBaya to create/run workflows. You can create them as scripts and run
them from command line. You can debug your workflows written in Python
using normal python debuggers. You can generate the XBaya workflow by using
the structure you create in Python as well. Then you can use XBaya to
monitor the workflow. These things are hard to do with an JavaScript API.
As a language also I would pick Python over JavaScript.

Thanks,
Supun..


On Thu, Oct 16, 2014 at 6:46 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:

> Hi Supun,
>
> Considering we are going to provide web base GUI support and  JavaScript is
> a well-known client side language, I would select JavaScript over Python. I
> am not much familiar with Python, so would like to know the advantages we
> get by selecting python here.
>
>
> Thanks,
> Shameera.
>
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
> > Hi Shameera,
> >
> > Why you prefer JavaScript over a language like Python?
> >
> > Thanks,
> > Supun..
> >
> > On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > shameerainfo@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> First of all thanks everyone for giving valuable inputs. After doing
> some
> >> background search and talking to different people in the University who
> has
> >> used different workflow languages, I myself convinced that introducing
> an
> >> another workflow language is not what actually they need. By changing
> >> exiting workflow language to another will not solve problems. What they
> >> asking is a easy way to construct the workflows. Indirectly what they
> >> asking for a sort of API which they can use to generate the workflows
> and
> >> run it. Correct me if i am wrong here.
> >>
> >> As most of above replies depict,  if we can get a simple API, as an
> >> example, for a web based application, JavaScript API would be a good
> >> solution, and probably JSON would be a good candidate for language,
> instead
> >> of XML.
> >>
> >> Airavata community already have started to implement web base GUI. Hence
> >> introducing a JSON base JavaScript API would be great help. WDYT?
> >>
> >> Thanks,
> >> Shameera.
> >>
> >>
> >> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >> alekny7@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> it is not dataflow instead focused on orchestrating REST services but
> >>> you may find it useful datapoint - we created worfklow service that
> uses
> >>> natively JavaScript and JSON to describe what happens during workflow
> >>> execution:
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>
> >>> HTH,
> >>>
> >>> Alek
> >>>
> >>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> wrote:
> >>>
> >>>> Hi Chris,
> >>>>
> >>>> Great to hear OODT community will be interested in adopting a JSON
> >>>> based workflow language and potentially a web based composer as well.
> >>>> Airavata previously had BPEL support initially through a home grown
> >>>> implementation [1] by Alek Slominski and later through Apache ODE
> [2]. Also
> >>>> a white paper [3] by Alek on this topic is an interesting read.
> >>>>
> >>>> I am of the same opinion that we should adopt something more modern as
> >>>> the challenges from scientific workflows seems to be converging with
> the
> >>>> data flow patterns in business workflows.
> >>>>
> >>>> It will be great if we can all compile a list of potential candidates
> >>>> and hack them through.
> >>>>
> >>>> Suresh
> >>>> [1] -
> >>>>
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>>> [2] -
> >>>>
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
> >>>> [3] -
> >>>>
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
> >>>>
> >>>>
> >>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>>>
> >>>> > Hi Guys,
> >>>> >
> >>>> > I've been interested in this too - we don't per have a specific
> >>>> > OODT workflow language, but we specific workflows using XML, and
> >>>> > other configuration (we are also thinking of moving to JSON for
> >>>> > this).
> >>>> >
> >>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> >>>> > to me.
> >>>> >
> >>>> > I wonder at the end of the day if we should adopt something more
> >>>> > modern like PIG or some other data flow type of language (PIG
> >>>> > is really neat).
> >>>> >
> >>>> > Cheers,
> >>>> > Chris
> >>>> >
> >>>> >
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Chris Mattmann, Ph.D.
> >>>> > Chief Architect
> >>>> > Instrument Software and Science Data Systems Section (398)
> >>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>>> > Office: 168-519, Mailstop: 168-527
> >>>> > Email: chris.a.mattmann@nasa.gov
> >>>> > WWW:  http://sunset.usc.edu/~mattmann/
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Adjunct Associate Professor, Computer Science Department
> >>>> > University of Southern California, Los Angeles, CA 90089 USA
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > -----Original Message-----
> >>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>>> > Reply-To: "architecture@airavata.apache.org"
> >>>> > <ar...@airavata.apache.org>
> >>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>>> > To: "architecture@airavata.apache.org" <
> >>>> architecture@airavata.apache.org>,
> >>>> > dev <de...@airavata.apache.org>
> >>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> Airavata.
> >>>> >
> >>>> >> Hi All,
> >>>> >>
> >>>> >> As we all know Airavata has its own workflow language call XWF.
> When
> >>>> XWF
> >>>> >> was introduced, main focus points are interoperability and
> >>>> convertibility.
> >>>> >> But with years of experience it is convinced that above
> requirements
> >>>> are
> >>>> >> not really useful when we come to real world use cases. And XWF is
> >>>> XML
> >>>> >> based bulky language where we attache WSDLs and Workflow image it
> >>>> self.
> >>>> >> But
> >>>> >> with the recent changes WSDL part is being removed from XWF.
> >>>> >>
> >>>> >> It is worth to evaluate handy Scientific workflow languages in
> >>>> industry
> >>>> >> and
> >>>> >> find out pros and cons, at the end of this evaluation we need to
> >>>> come up
> >>>> >> with idea how we should improve Airavata workflow language, either
> >>>> we can
> >>>> >> improve existing XWF language, totally change to a new language
> >>>> available
> >>>> >> in industry or write a new light weight language. Basic
> requirements
> >>>> that
> >>>> >> we expect from new improvement are, high usability, flexible, light
> >>>> weight
> >>>> >> and real time monitoring support. As you can see above requirements
> >>>> are
> >>>> >> not
> >>>> >> direct comes with workflow languages but we need workflow language
> >>>> which
> >>>> >> help to support above requirements.
> >>>> >>
> >>>> >> After reading few papers and googling, initially i have come up
> with
> >>>> >> following three existing languages,
> >>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>>> >> 2. WS-BPEL
> >>>> >> 3. SIDL
> >>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
> >>>> >>
> >>>> >> In my opinion SIDL is more familiar with scientific domain,
> >>>> Radical-SAGA
> >>>> >> also uses slightly modified version of SIDL. Other than above three
> >>>> >> languages we can come up with simple workflow language base on
> >>>> json(or
> >>>> >> yaml) which support all our requirements for some extends.
> >>>> >>
> >>>> >> It would be grate if I can get more input regarding the $Subject
> >>>> form the
> >>>> >> airavata community. You all are more than welcome to provide any
> >>>> type of
> >>>> >> suggestions.
> >>>> >>
> >>>> >> Thanks,
> >>>> >> Shameera.
> >>>> >>
> >>>> >> 
> >>>> >>
> >>>> >> --
> >>>> >> Best Regards,
> >>>> >> Shameera Rathnayaka.
> >>>> >>
> >>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> The best way to predict the future is to invent it - Alan Kay
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Shameera Rathnayaka.
> >>
> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>



-- 
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Nipurn Doshi <ni...@umail.iu.edu>.

Hi,

How about I suggest a third option where I have the current code setup
which has access to thrift classes, airavata interface classes and a
complete framework ( Laravel ) that supports rest apis to generate json
data that we are looking at?

I am suggesting this only because I have done this in previous projects I
have worked on using Laravel.

Although, it's in PHP.
 On 16-Oct-2014 6:47 pm, "Shameera Rathnayaka" <sh...@gmail.com>
wrote:

> Hi Supun,
>
> Considering we are going to provide web base GUI support and  JavaScript
> is a well-known client side language, I would select JavaScript over
> Python. I am not much familiar with Python, so would like to know the
> advantages we get by selecting python here.
>
>
> Thanks,
> Shameera.
>
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
>> Hi Shameera,
>>
>> Why you prefer JavaScript over a language like Python?
>>
>> Thanks,
>> Supun..
>>
>> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> First of all thanks everyone for giving valuable inputs. After doing
>>> some background search and talking to different people in the University
>>> who has used different workflow languages, I myself convinced that
>>> introducing an another workflow language is not what actually they need. By
>>> changing exiting workflow language to another will not solve problems. What
>>> they asking is a easy way to construct the workflows. Indirectly what they
>>> asking for a sort of API which they can use to generate the workflows and
>>> run it. Correct me if i am wrong here.
>>>
>>> As most of above replies depict,  if we can get a simple API, as an
>>> example, for a web based application, JavaScript API would be a good
>>> solution, and probably JSON would be a good candidate for language, instead
>>> of XML.
>>>
>>> Airavata community already have started to implement web base GUI. Hence
>>> introducing a JSON base JavaScript API would be great help. WDYT?
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> alekny7@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> it is not dataflow instead focused on orchestrating REST services but
>>>> you may find it useful datapoint - we created worfklow service that uses
>>>> natively JavaScript and JSON to describe what happens during workflow
>>>> execution:
>>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>>
>>>> HTH,
>>>>
>>>> Alek
>>>>
>>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>> Great to hear OODT community will be interested in adopting a JSON
>>>>> based workflow language and potentially a web based composer as well.
>>>>> Airavata previously had BPEL support initially through a home grown
>>>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
>>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>>
>>>>> I am of the same opinion that we should adopt something more modern as
>>>>> the challenges from scientific workflows seems to be converging with the
>>>>> data flow patterns in business workflows.
>>>>>
>>>>> It will be great if we can all compile a list of potential candidates
>>>>> and hack them through.
>>>>>
>>>>> Suresh
>>>>> [1] -
>>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>>> [2] -
>>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>>>> [3] -
>>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>>>
>>>>>
>>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>>
>>>>> > Hi Guys,
>>>>> >
>>>>> > I've been interested in this too - we don't per have a specific
>>>>> > OODT workflow language, but we specific workflows using XML, and
>>>>> > other configuration (we are also thinking of moving to JSON for
>>>>> > this).
>>>>> >
>>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>>> > to me.
>>>>> >
>>>>> > I wonder at the end of the day if we should adopt something more
>>>>> > modern like PIG or some other data flow type of language (PIG
>>>>> > is really neat).
>>>>> >
>>>>> > Cheers,
>>>>> > Chris
>>>>> >
>>>>> >
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Chris Mattmann, Ph.D.
>>>>> > Chief Architect
>>>>> > Instrument Software and Science Data Systems Section (398)
>>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>> > Office: 168-519, Mailstop: 168-527
>>>>> > Email: chris.a.mattmann@nasa.gov
>>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Adjunct Associate Professor, Computer Science Department
>>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > -----Original Message-----
>>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>>> > Reply-To: "architecture@airavata.apache.org"
>>>>> > <ar...@airavata.apache.org>
>>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>>> > To: "architecture@airavata.apache.org" <
>>>>> architecture@airavata.apache.org>,
>>>>> > dev <de...@airavata.apache.org>
>>>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>>>> >
>>>>> >> Hi All,
>>>>> >>
>>>>> >> As we all know Airavata has its own workflow language call XWF.
>>>>> When XWF
>>>>> >> was introduced, main focus points are interoperability and
>>>>> convertibility.
>>>>> >> But with years of experience it is convinced that above
>>>>> requirements are
>>>>> >> not really useful when we come to real world use cases. And XWF is
>>>>> XML
>>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>>> self.
>>>>> >> But
>>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>>> >>
>>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>>> industry
>>>>> >> and
>>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>>> come up
>>>>> >> with idea how we should improve Airavata workflow language, either
>>>>> we can
>>>>> >> improve existing XWF language, totally change to a new language
>>>>> available
>>>>> >> in industry or write a new light weight language. Basic
>>>>> requirements that
>>>>> >> we expect from new improvement are, high usability, flexible, light
>>>>> weight
>>>>> >> and real time monitoring support. As you can see above requirements
>>>>> are
>>>>> >> not
>>>>> >> direct comes with workflow languages but we need workflow language
>>>>> which
>>>>> >> help to support above requirements.
>>>>> >>
>>>>> >> After reading few papers and googling, initially i have come up with
>>>>> >> following three existing languages,
>>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>>> >> 2. WS-BPEL
>>>>> >> 3. SIDL
>>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>>> >>
>>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>>> Radical-SAGA
>>>>> >> also uses slightly modified version of SIDL. Other than above three
>>>>> >> languages we can come up with simple workflow language base on
>>>>> json(or
>>>>> >> yaml) which support all our requirements for some extends.
>>>>> >>
>>>>> >> It would be grate if I can get more input regarding the $Subject
>>>>> form the
>>>>> >> airavata community. You all are more than welcome to provide any
>>>>> type of
>>>>> >> suggestions.
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Shameera.
>>>>> >>
>>>>> >> 
>>>>> >>
>>>>> >> --
>>>>> >> Best Regards,
>>>>> >> Shameera Rathnayaka.
>>>>> >>
>>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> The best way to predict the future is to invent it - Alan Kay
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>>
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by K Yoshimoto <ke...@sdsc.edu>.

I prefer python myself, but if the API is clean, it should be
usable from a large number of languages on the client side.

On Thu, Oct 16, 2014 at 06:46:36PM -0400, Shameera Rathnayaka wrote:
> Hi Supun,
> 
> Considering we are going to provide web base GUI support and  JavaScript is
> a well-known client side language, I would select JavaScript over Python. I
> am not much familiar with Python, so would like to know the advantages we
> get by selecting python here.
> 
> 
> Thanks,
> Shameera.
> 
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
> 
> > Hi Shameera,
> >
> > Why you prefer JavaScript over a language like Python?
> >
> > Thanks,
> > Supun..
> >
> > On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > shameerainfo@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> First of all thanks everyone for giving valuable inputs. After doing some
> >> background search and talking to different people in the University who has
> >> used different workflow languages, I myself convinced that introducing an
> >> another workflow language is not what actually they need. By changing
> >> exiting workflow language to another will not solve problems. What they
> >> asking is a easy way to construct the workflows. Indirectly what they
> >> asking for a sort of API which they can use to generate the workflows and
> >> run it. Correct me if i am wrong here.
> >>
> >> As most of above replies depict,  if we can get a simple API, as an
> >> example, for a web based application, JavaScript API would be a good
> >> solution, and probably JSON would be a good candidate for language, instead
> >> of XML.
> >>
> >> Airavata community already have started to implement web base GUI. Hence
> >> introducing a JSON base JavaScript API would be great help. WDYT?
> >>
> >> Thanks,
> >> Shameera.
> >>
> >>
> >> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >> alekny7@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> it is not dataflow instead focused on orchestrating REST services but
> >>> you may find it useful datapoint - we created worfklow service that uses
> >>> natively JavaScript and JSON to describe what happens during workflow
> >>> execution:
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>
> >>> HTH,
> >>>
> >>> Alek
> >>>
> >>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
> >>>
> >>>> Hi Chris,
> >>>>
> >>>> Great to hear OODT community will be interested in adopting a JSON
> >>>> based workflow language and potentially a web based composer as well.
> >>>> Airavata previously had BPEL support initially through a home grown
> >>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
> >>>> a white paper [3] by Alek on this topic is an interesting read.
> >>>>
> >>>> I am of the same opinion that we should adopt something more modern as
> >>>> the challenges from scientific workflows seems to be converging with the
> >>>> data flow patterns in business workflows.
> >>>>
> >>>> It will be great if we can all compile a list of potential candidates
> >>>> and hack them through.
> >>>>
> >>>> Suresh
> >>>> [1] -
> >>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>>> [2] -
> >>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
> >>>> [3] -
> >>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
> >>>>
> >>>>
> >>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>>>
> >>>> > Hi Guys,
> >>>> >
> >>>> > I've been interested in this too - we don't per have a specific
> >>>> > OODT workflow language, but we specific workflows using XML, and
> >>>> > other configuration (we are also thinking of moving to JSON for
> >>>> > this).
> >>>> >
> >>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> >>>> > to me.
> >>>> >
> >>>> > I wonder at the end of the day if we should adopt something more
> >>>> > modern like PIG or some other data flow type of language (PIG
> >>>> > is really neat).
> >>>> >
> >>>> > Cheers,
> >>>> > Chris
> >>>> >
> >>>> >
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Chris Mattmann, Ph.D.
> >>>> > Chief Architect
> >>>> > Instrument Software and Science Data Systems Section (398)
> >>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>>> > Office: 168-519, Mailstop: 168-527
> >>>> > Email: chris.a.mattmann@nasa.gov
> >>>> > WWW:  http://sunset.usc.edu/~mattmann/
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Adjunct Associate Professor, Computer Science Department
> >>>> > University of Southern California, Los Angeles, CA 90089 USA
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > -----Original Message-----
> >>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>>> > Reply-To: "architecture@airavata.apache.org"
> >>>> > <ar...@airavata.apache.org>
> >>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>>> > To: "architecture@airavata.apache.org" <
> >>>> architecture@airavata.apache.org>,
> >>>> > dev <de...@airavata.apache.org>
> >>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> >>>> >
> >>>> >> Hi All,
> >>>> >>
> >>>> >> As we all know Airavata has its own workflow language call XWF. When
> >>>> XWF
> >>>> >> was introduced, main focus points are interoperability and
> >>>> convertibility.
> >>>> >> But with years of experience it is convinced that above requirements
> >>>> are
> >>>> >> not really useful when we come to real world use cases. And XWF is
> >>>> XML
> >>>> >> based bulky language where we attache WSDLs and Workflow image it
> >>>> self.
> >>>> >> But
> >>>> >> with the recent changes WSDL part is being removed from XWF.
> >>>> >>
> >>>> >> It is worth to evaluate handy Scientific workflow languages in
> >>>> industry
> >>>> >> and
> >>>> >> find out pros and cons, at the end of this evaluation we need to
> >>>> come up
> >>>> >> with idea how we should improve Airavata workflow language, either
> >>>> we can
> >>>> >> improve existing XWF language, totally change to a new language
> >>>> available
> >>>> >> in industry or write a new light weight language. Basic requirements
> >>>> that
> >>>> >> we expect from new improvement are, high usability, flexible, light
> >>>> weight
> >>>> >> and real time monitoring support. As you can see above requirements
> >>>> are
> >>>> >> not
> >>>> >> direct comes with workflow languages but we need workflow language
> >>>> which
> >>>> >> help to support above requirements.
> >>>> >>
> >>>> >> After reading few papers and googling, initially i have come up with
> >>>> >> following three existing languages,
> >>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>>> >> 2. WS-BPEL
> >>>> >> 3. SIDL
> >>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
> >>>> >>
> >>>> >> In my opinion SIDL is more familiar with scientific domain,
> >>>> Radical-SAGA
> >>>> >> also uses slightly modified version of SIDL. Other than above three
> >>>> >> languages we can come up with simple workflow language base on
> >>>> json(or
> >>>> >> yaml) which support all our requirements for some extends.
> >>>> >>
> >>>> >> It would be grate if I can get more input regarding the $Subject
> >>>> form the
> >>>> >> airavata community. You all are more than welcome to provide any
> >>>> type of
> >>>> >> suggestions.
> >>>> >>
> >>>> >> Thanks,
> >>>> >> Shameera.
> >>>> >>
> >>>> >> 
> >>>> >>
> >>>> >> --
> >>>> >> Best Regards,
> >>>> >> Shameera Rathnayaka.
> >>>> >>
> >>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> The best way to predict the future is to invent it - Alan Kay
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Shameera Rathnayaka.
> >>
> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
> >
> 
> 
> -- 
> Best Regards,
> Shameera Rathnayaka.
> 
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Supun Kamburugamuva <su...@gmail.com>.

Hi Shameera,

To create a web based UI I don't see an immediate advantage of using
JavaScript over a Python API. Because Airavata's API is Thrift (not any
language based API) you can generate code for any language (including
JavaScript). Anyway most probably you will not use direct Airavata API
calls for creating a web UI. I'm also not sure what you exactly mean by
JavaScript language as well. If you explain in more detail that would be
really helpful.

If you just give a Python API to create workflows, you really don't even
need XBaya to create/run workflows. You can create them as scripts and run
them from command line. You can debug your workflows written in Python
using normal python debuggers. You can generate the XBaya workflow by using
the structure you create in Python as well. Then you can use XBaya to
monitor the workflow. These things are hard to do with an JavaScript API.
As a language also I would pick Python over JavaScript.

Thanks,
Supun..


On Thu, Oct 16, 2014 at 6:46 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:

> Hi Supun,
>
> Considering we are going to provide web base GUI support and  JavaScript is
> a well-known client side language, I would select JavaScript over Python. I
> am not much familiar with Python, so would like to know the advantages we
> get by selecting python here.
>
>
> Thanks,
> Shameera.
>
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
> > Hi Shameera,
> >
> > Why you prefer JavaScript over a language like Python?
> >
> > Thanks,
> > Supun..
> >
> > On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > shameerainfo@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> First of all thanks everyone for giving valuable inputs. After doing
> some
> >> background search and talking to different people in the University who
> has
> >> used different workflow languages, I myself convinced that introducing
> an
> >> another workflow language is not what actually they need. By changing
> >> exiting workflow language to another will not solve problems. What they
> >> asking is a easy way to construct the workflows. Indirectly what they
> >> asking for a sort of API which they can use to generate the workflows
> and
> >> run it. Correct me if i am wrong here.
> >>
> >> As most of above replies depict,  if we can get a simple API, as an
> >> example, for a web based application, JavaScript API would be a good
> >> solution, and probably JSON would be a good candidate for language,
> instead
> >> of XML.
> >>
> >> Airavata community already have started to implement web base GUI. Hence
> >> introducing a JSON base JavaScript API would be great help. WDYT?
> >>
> >> Thanks,
> >> Shameera.
> >>
> >>
> >> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >> alekny7@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> it is not dataflow instead focused on orchestrating REST services but
> >>> you may find it useful datapoint - we created worfklow service that
> uses
> >>> natively JavaScript and JSON to describe what happens during workflow
> >>> execution:
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>
> >>> HTH,
> >>>
> >>> Alek
> >>>
> >>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
> wrote:
> >>>
> >>>> Hi Chris,
> >>>>
> >>>> Great to hear OODT community will be interested in adopting a JSON
> >>>> based workflow language and potentially a web based composer as well.
> >>>> Airavata previously had BPEL support initially through a home grown
> >>>> implementation [1] by Alek Slominski and later through Apache ODE
> [2]. Also
> >>>> a white paper [3] by Alek on this topic is an interesting read.
> >>>>
> >>>> I am of the same opinion that we should adopt something more modern as
> >>>> the challenges from scientific workflows seems to be converging with
> the
> >>>> data flow patterns in business workflows.
> >>>>
> >>>> It will be great if we can all compile a list of potential candidates
> >>>> and hack them through.
> >>>>
> >>>> Suresh
> >>>> [1] -
> >>>>
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> >>>> [2] -
> >>>>
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
> >>>> [3] -
> >>>>
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
> >>>>
> >>>>
> >>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> >>>> chris.a.mattmann@jpl.nasa.gov> wrote:
> >>>>
> >>>> > Hi Guys,
> >>>> >
> >>>> > I've been interested in this too - we don't per have a specific
> >>>> > OODT workflow language, but we specific workflows using XML, and
> >>>> > other configuration (we are also thinking of moving to JSON for
> >>>> > this).
> >>>> >
> >>>> > In the past I've also looked at YAWL and BPEL - both seem complex
> >>>> > to me.
> >>>> >
> >>>> > I wonder at the end of the day if we should adopt something more
> >>>> > modern like PIG or some other data flow type of language (PIG
> >>>> > is really neat).
> >>>> >
> >>>> > Cheers,
> >>>> > Chris
> >>>> >
> >>>> >
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Chris Mattmann, Ph.D.
> >>>> > Chief Architect
> >>>> > Instrument Software and Science Data Systems Section (398)
> >>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>>> > Office: 168-519, Mailstop: 168-527
> >>>> > Email: chris.a.mattmann@nasa.gov
> >>>> > WWW:  http://sunset.usc.edu/~mattmann/
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> > Adjunct Associate Professor, Computer Science Department
> >>>> > University of Southern California, Los Angeles, CA 90089 USA
> >>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > -----Original Message-----
> >>>> > From: Shameera Rathnayaka <sh...@gmail.com>
> >>>> > Reply-To: "architecture@airavata.apache.org"
> >>>> > <ar...@airavata.apache.org>
> >>>> > Date: Thursday, September 18, 2014 8:26 AM
> >>>> > To: "architecture@airavata.apache.org" <
> >>>> architecture@airavata.apache.org>,
> >>>> > dev <de...@airavata.apache.org>
> >>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> Airavata.
> >>>> >
> >>>> >> Hi All,
> >>>> >>
> >>>> >> As we all know Airavata has its own workflow language call XWF.
> When
> >>>> XWF
> >>>> >> was introduced, main focus points are interoperability and
> >>>> convertibility.
> >>>> >> But with years of experience it is convinced that above
> requirements
> >>>> are
> >>>> >> not really useful when we come to real world use cases. And XWF is
> >>>> XML
> >>>> >> based bulky language where we attache WSDLs and Workflow image it
> >>>> self.
> >>>> >> But
> >>>> >> with the recent changes WSDL part is being removed from XWF.
> >>>> >>
> >>>> >> It is worth to evaluate handy Scientific workflow languages in
> >>>> industry
> >>>> >> and
> >>>> >> find out pros and cons, at the end of this evaluation we need to
> >>>> come up
> >>>> >> with idea how we should improve Airavata workflow language, either
> >>>> we can
> >>>> >> improve existing XWF language, totally change to a new language
> >>>> available
> >>>> >> in industry or write a new light weight language. Basic
> requirements
> >>>> that
> >>>> >> we expect from new improvement are, high usability, flexible, light
> >>>> weight
> >>>> >> and real time monitoring support. As you can see above requirements
> >>>> are
> >>>> >> not
> >>>> >> direct comes with workflow languages but we need workflow language
> >>>> which
> >>>> >> help to support above requirements.
> >>>> >>
> >>>> >> After reading few papers and googling, initially i have come up
> with
> >>>> >> following three existing languages,
> >>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> >>>> >> 2. WS-BPEL
> >>>> >> 3. SIDL
> >>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
> >>>> >>
> >>>> >> In my opinion SIDL is more familiar with scientific domain,
> >>>> Radical-SAGA
> >>>> >> also uses slightly modified version of SIDL. Other than above three
> >>>> >> languages we can come up with simple workflow language base on
> >>>> json(or
> >>>> >> yaml) which support all our requirements for some extends.
> >>>> >>
> >>>> >> It would be grate if I can get more input regarding the $Subject
> >>>> form the
> >>>> >> airavata community. You all are more than welcome to provide any
> >>>> type of
> >>>> >> suggestions.
> >>>> >>
> >>>> >> Thanks,
> >>>> >> Shameera.
> >>>> >>
> >>>> >> 
> >>>> >>
> >>>> >> --
> >>>> >> Best Regards,
> >>>> >> Shameera Rathnayaka.
> >>>> >>
> >>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> The best way to predict the future is to invent it - Alan Kay
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards,
> >> Shameera Rathnayaka.
> >>
> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >> Blog : http://shameerarathnayaka.blogspot.com/
> >>
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
> >
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>



-- 
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Nipurn Doshi <ni...@umail.iu.edu>.

Hi,

How about I suggest a third option where I have the current code setup
which has access to thrift classes, airavata interface classes and a
complete framework ( Laravel ) that supports rest apis to generate json
data that we are looking at?

I am suggesting this only because I have done this in previous projects I
have worked on using Laravel.

Although, it's in PHP.
 On 16-Oct-2014 6:47 pm, "Shameera Rathnayaka" <sh...@gmail.com>
wrote:

> Hi Supun,
>
> Considering we are going to provide web base GUI support and  JavaScript
> is a well-known client side language, I would select JavaScript over
> Python. I am not much familiar with Python, so would like to know the
> advantages we get by selecting python here.
>
>
> Thanks,
> Shameera.
>
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
> wrote:
>
>> Hi Shameera,
>>
>> Why you prefer JavaScript over a language like Python?
>>
>> Thanks,
>> Supun..
>>
>> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> shameerainfo@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> First of all thanks everyone for giving valuable inputs. After doing
>>> some background search and talking to different people in the University
>>> who has used different workflow languages, I myself convinced that
>>> introducing an another workflow language is not what actually they need. By
>>> changing exiting workflow language to another will not solve problems. What
>>> they asking is a easy way to construct the workflows. Indirectly what they
>>> asking for a sort of API which they can use to generate the workflows and
>>> run it. Correct me if i am wrong here.
>>>
>>> As most of above replies depict,  if we can get a simple API, as an
>>> example, for a web based application, JavaScript API would be a good
>>> solution, and probably JSON would be a good candidate for language, instead
>>> of XML.
>>>
>>> Airavata community already have started to implement web base GUI. Hence
>>> introducing a JSON base JavaScript API would be great help. WDYT?
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> alekny7@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> it is not dataflow instead focused on orchestrating REST services but
>>>> you may find it useful datapoint - we created worfklow service that uses
>>>> natively JavaScript and JSON to describe what happens during workflow
>>>> execution:
>>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>>
>>>> HTH,
>>>>
>>>> Alek
>>>>
>>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>> Great to hear OODT community will be interested in adopting a JSON
>>>>> based workflow language and potentially a web based composer as well.
>>>>> Airavata previously had BPEL support initially through a home grown
>>>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
>>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>>
>>>>> I am of the same opinion that we should adopt something more modern as
>>>>> the challenges from scientific workflows seems to be converging with the
>>>>> data flow patterns in business workflows.
>>>>>
>>>>> It will be great if we can all compile a list of potential candidates
>>>>> and hack them through.
>>>>>
>>>>> Suresh
>>>>> [1] -
>>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>>> [2] -
>>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>>>> [3] -
>>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>>>
>>>>>
>>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>>
>>>>> > Hi Guys,
>>>>> >
>>>>> > I've been interested in this too - we don't per have a specific
>>>>> > OODT workflow language, but we specific workflows using XML, and
>>>>> > other configuration (we are also thinking of moving to JSON for
>>>>> > this).
>>>>> >
>>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>>> > to me.
>>>>> >
>>>>> > I wonder at the end of the day if we should adopt something more
>>>>> > modern like PIG or some other data flow type of language (PIG
>>>>> > is really neat).
>>>>> >
>>>>> > Cheers,
>>>>> > Chris
>>>>> >
>>>>> >
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Chris Mattmann, Ph.D.
>>>>> > Chief Architect
>>>>> > Instrument Software and Science Data Systems Section (398)
>>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>> > Office: 168-519, Mailstop: 168-527
>>>>> > Email: chris.a.mattmann@nasa.gov
>>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> > Adjunct Associate Professor, Computer Science Department
>>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > -----Original Message-----
>>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>>> > Reply-To: "architecture@airavata.apache.org"
>>>>> > <ar...@airavata.apache.org>
>>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>>> > To: "architecture@airavata.apache.org" <
>>>>> architecture@airavata.apache.org>,
>>>>> > dev <de...@airavata.apache.org>
>>>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>>>> >
>>>>> >> Hi All,
>>>>> >>
>>>>> >> As we all know Airavata has its own workflow language call XWF.
>>>>> When XWF
>>>>> >> was introduced, main focus points are interoperability and
>>>>> convertibility.
>>>>> >> But with years of experience it is convinced that above
>>>>> requirements are
>>>>> >> not really useful when we come to real world use cases. And XWF is
>>>>> XML
>>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>>> self.
>>>>> >> But
>>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>>> >>
>>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>>> industry
>>>>> >> and
>>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>>> come up
>>>>> >> with idea how we should improve Airavata workflow language, either
>>>>> we can
>>>>> >> improve existing XWF language, totally change to a new language
>>>>> available
>>>>> >> in industry or write a new light weight language. Basic
>>>>> requirements that
>>>>> >> we expect from new improvement are, high usability, flexible, light
>>>>> weight
>>>>> >> and real time monitoring support. As you can see above requirements
>>>>> are
>>>>> >> not
>>>>> >> direct comes with workflow languages but we need workflow language
>>>>> which
>>>>> >> help to support above requirements.
>>>>> >>
>>>>> >> After reading few papers and googling, initially i have come up with
>>>>> >> following three existing languages,
>>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>>> >> 2. WS-BPEL
>>>>> >> 3. SIDL
>>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>>> >>
>>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>>> Radical-SAGA
>>>>> >> also uses slightly modified version of SIDL. Other than above three
>>>>> >> languages we can come up with simple workflow language base on
>>>>> json(or
>>>>> >> yaml) which support all our requirements for some extends.
>>>>> >>
>>>>> >> It would be grate if I can get more input regarding the $Subject
>>>>> form the
>>>>> >> airavata community. You all are more than welcome to provide any
>>>>> type of
>>>>> >> suggestions.
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Shameera.
>>>>> >>
>>>>> >> 
>>>>> >>
>>>>> >> --
>>>>> >> Best Regards,
>>>>> >> Shameera Rathnayaka.
>>>>> >>
>>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> The best way to predict the future is to invent it - Alan Kay
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>>
>>> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>
>>
>>
>> --
>> Supun Kamburugamuva
>> Member, Apache Software Foundation; http://www.apache.org
>> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
>> Blog: http://supunk.blogspot.com
>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Shameera Rathnayaka <sh...@gmail.com>.

Hi Supun,

Considering we are going to provide web base GUI support and  JavaScript is
a well-known client side language, I would select JavaScript over Python. I
am not much familiar with Python, so would like to know the advantages we
get by selecting python here.


Thanks,
Shameera.

On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:

> Hi Shameera,
>
> Why you prefer JavaScript over a language like Python?
>
> Thanks,
> Supun..
>
> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi,
>>
>> First of all thanks everyone for giving valuable inputs. After doing some
>> background search and talking to different people in the University who has
>> used different workflow languages, I myself convinced that introducing an
>> another workflow language is not what actually they need. By changing
>> exiting workflow language to another will not solve problems. What they
>> asking is a easy way to construct the workflows. Indirectly what they
>> asking for a sort of API which they can use to generate the workflows and
>> run it. Correct me if i am wrong here.
>>
>> As most of above replies depict,  if we can get a simple API, as an
>> example, for a web based application, JavaScript API would be a good
>> solution, and probably JSON would be a good candidate for language, instead
>> of XML.
>>
>> Airavata community already have started to implement web base GUI. Hence
>> introducing a JSON base JavaScript API would be great help. WDYT?
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> alekny7@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> it is not dataflow instead focused on orchestrating REST services but
>>> you may find it useful datapoint - we created worfklow service that uses
>>> natively JavaScript and JSON to describe what happens during workflow
>>> execution:
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>
>>> HTH,
>>>
>>> Alek
>>>
>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>>>
>>>> Hi Chris,
>>>>
>>>> Great to hear OODT community will be interested in adopting a JSON
>>>> based workflow language and potentially a web based composer as well.
>>>> Airavata previously had BPEL support initially through a home grown
>>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>
>>>> I am of the same opinion that we should adopt something more modern as
>>>> the challenges from scientific workflows seems to be converging with the
>>>> data flow patterns in business workflows.
>>>>
>>>> It will be great if we can all compile a list of potential candidates
>>>> and hack them through.
>>>>
>>>> Suresh
>>>> [1] -
>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>> [2] -
>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>>> [3] -
>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>>
>>>>
>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>
>>>> > Hi Guys,
>>>> >
>>>> > I've been interested in this too - we don't per have a specific
>>>> > OODT workflow language, but we specific workflows using XML, and
>>>> > other configuration (we are also thinking of moving to JSON for
>>>> > this).
>>>> >
>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>> > to me.
>>>> >
>>>> > I wonder at the end of the day if we should adopt something more
>>>> > modern like PIG or some other data flow type of language (PIG
>>>> > is really neat).
>>>> >
>>>> > Cheers,
>>>> > Chris
>>>> >
>>>> >
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Chris Mattmann, Ph.D.
>>>> > Chief Architect
>>>> > Instrument Software and Science Data Systems Section (398)
>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> > Office: 168-519, Mailstop: 168-527
>>>> > Email: chris.a.mattmann@nasa.gov
>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Adjunct Associate Professor, Computer Science Department
>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > -----Original Message-----
>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>> > Reply-To: "architecture@airavata.apache.org"
>>>> > <ar...@airavata.apache.org>
>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>> > To: "architecture@airavata.apache.org" <
>>>> architecture@airavata.apache.org>,
>>>> > dev <de...@airavata.apache.org>
>>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>>> >
>>>> >> Hi All,
>>>> >>
>>>> >> As we all know Airavata has its own workflow language call XWF. When
>>>> XWF
>>>> >> was introduced, main focus points are interoperability and
>>>> convertibility.
>>>> >> But with years of experience it is convinced that above requirements
>>>> are
>>>> >> not really useful when we come to real world use cases. And XWF is
>>>> XML
>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>> self.
>>>> >> But
>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>> >>
>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>> industry
>>>> >> and
>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>> come up
>>>> >> with idea how we should improve Airavata workflow language, either
>>>> we can
>>>> >> improve existing XWF language, totally change to a new language
>>>> available
>>>> >> in industry or write a new light weight language. Basic requirements
>>>> that
>>>> >> we expect from new improvement are, high usability, flexible, light
>>>> weight
>>>> >> and real time monitoring support. As you can see above requirements
>>>> are
>>>> >> not
>>>> >> direct comes with workflow languages but we need workflow language
>>>> which
>>>> >> help to support above requirements.
>>>> >>
>>>> >> After reading few papers and googling, initially i have come up with
>>>> >> following three existing languages,
>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>> >> 2. WS-BPEL
>>>> >> 3. SIDL
>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>> >>
>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>> Radical-SAGA
>>>> >> also uses slightly modified version of SIDL. Other than above three
>>>> >> languages we can come up with simple workflow language base on
>>>> json(or
>>>> >> yaml) which support all our requirements for some extends.
>>>> >>
>>>> >> It would be grate if I can get more input regarding the $Subject
>>>> form the
>>>> >> airavata community. You all are more than welcome to provide any
>>>> type of
>>>> >> suggestions.
>>>> >>
>>>> >> Thanks,
>>>> >> Shameera.
>>>> >>
>>>> >> 
>>>> >>
>>>> >> --
>>>> >> Best Regards,
>>>> >> Shameera Rathnayaka.
>>>> >>
>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>
>>>>
>>>
>>>
>>> --
>>> The best way to predict the future is to invent it - Alan Kay
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>


-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Supun Kamburugamuva <su...@gmail.com>.

Once we had an offline discussion about the Airavata Workflow language
(with Milinda, Saliya and Shameera). In that discussion one thing came out
was why we need to invent a different language when a simple library like
Python will full fill of Airavata requirements.

There are many benefits in using a Python library as the API for
controlling Airavata workflows.

1. It is a library, gives the ultimate control over the execution and it
can be simpler than any domain specific language that we can come with like
XML, JSON etc
2. Most people use python and can learn it easily than any Domain specific
language
3. You can easily create a Python library for Airavata because all the APIs
are thrift based.
4. If you design the constructs correctly you can plug an XBaya.

Any thoughts?

Thanks,
Supun..


On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:

> Hi Shameera,
>
> Why you prefer JavaScript over a language like Python?
>
> Thanks,
> Supun..
>
> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi,
>>
>> First of all thanks everyone for giving valuable inputs. After doing some
>> background search and talking to different people in the University who has
>> used different workflow languages, I myself convinced that introducing an
>> another workflow language is not what actually they need. By changing
>> exiting workflow language to another will not solve problems. What they
>> asking is a easy way to construct the workflows. Indirectly what they
>> asking for a sort of API which they can use to generate the workflows and
>> run it. Correct me if i am wrong here.
>>
>> As most of above replies depict,  if we can get a simple API, as an
>> example, for a web based application, JavaScript API would be a good
>> solution, and probably JSON would be a good candidate for language, instead
>> of XML.
>>
>> Airavata community already have started to implement web base GUI. Hence
>> introducing a JSON base JavaScript API would be great help. WDYT?
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> alekny7@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> it is not dataflow instead focused on orchestrating REST services but
>>> you may find it useful datapoint - we created worfklow service that uses
>>> natively JavaScript and JSON to describe what happens during workflow
>>> execution:
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>
>>> HTH,
>>>
>>> Alek
>>>
>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>>>
>>>> Hi Chris,
>>>>
>>>> Great to hear OODT community will be interested in adopting a JSON
>>>> based workflow language and potentially a web based composer as well.
>>>> Airavata previously had BPEL support initially through a home grown
>>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>
>>>> I am of the same opinion that we should adopt something more modern as
>>>> the challenges from scientific workflows seems to be converging with the
>>>> data flow patterns in business workflows.
>>>>
>>>> It will be great if we can all compile a list of potential candidates
>>>> and hack them through.
>>>>
>>>> Suresh
>>>> [1] -
>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>> [2] -
>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>>> [3] -
>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>>
>>>>
>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>
>>>> > Hi Guys,
>>>> >
>>>> > I've been interested in this too - we don't per have a specific
>>>> > OODT workflow language, but we specific workflows using XML, and
>>>> > other configuration (we are also thinking of moving to JSON for
>>>> > this).
>>>> >
>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>> > to me.
>>>> >
>>>> > I wonder at the end of the day if we should adopt something more
>>>> > modern like PIG or some other data flow type of language (PIG
>>>> > is really neat).
>>>> >
>>>> > Cheers,
>>>> > Chris
>>>> >
>>>> >
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Chris Mattmann, Ph.D.
>>>> > Chief Architect
>>>> > Instrument Software and Science Data Systems Section (398)
>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> > Office: 168-519, Mailstop: 168-527
>>>> > Email: chris.a.mattmann@nasa.gov
>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Adjunct Associate Professor, Computer Science Department
>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > -----Original Message-----
>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>> > Reply-To: "architecture@airavata.apache.org"
>>>> > <ar...@airavata.apache.org>
>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>> > To: "architecture@airavata.apache.org" <
>>>> architecture@airavata.apache.org>,
>>>> > dev <de...@airavata.apache.org>
>>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>>> >
>>>> >> Hi All,
>>>> >>
>>>> >> As we all know Airavata has its own workflow language call XWF. When
>>>> XWF
>>>> >> was introduced, main focus points are interoperability and
>>>> convertibility.
>>>> >> But with years of experience it is convinced that above requirements
>>>> are
>>>> >> not really useful when we come to real world use cases. And XWF is
>>>> XML
>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>> self.
>>>> >> But
>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>> >>
>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>> industry
>>>> >> and
>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>> come up
>>>> >> with idea how we should improve Airavata workflow language, either
>>>> we can
>>>> >> improve existing XWF language, totally change to a new language
>>>> available
>>>> >> in industry or write a new light weight language. Basic requirements
>>>> that
>>>> >> we expect from new improvement are, high usability, flexible, light
>>>> weight
>>>> >> and real time monitoring support. As you can see above requirements
>>>> are
>>>> >> not
>>>> >> direct comes with workflow languages but we need workflow language
>>>> which
>>>> >> help to support above requirements.
>>>> >>
>>>> >> After reading few papers and googling, initially i have come up with
>>>> >> following three existing languages,
>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>> >> 2. WS-BPEL
>>>> >> 3. SIDL
>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>> >>
>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>> Radical-SAGA
>>>> >> also uses slightly modified version of SIDL. Other than above three
>>>> >> languages we can come up with simple workflow language base on
>>>> json(or
>>>> >> yaml) which support all our requirements for some extends.
>>>> >>
>>>> >> It would be grate if I can get more input regarding the $Subject
>>>> form the
>>>> >> airavata community. You all are more than welcome to provide any
>>>> type of
>>>> >> suggestions.
>>>> >>
>>>> >> Thanks,
>>>> >> Shameera.
>>>> >>
>>>> >> 
>>>> >>
>>>> >> --
>>>> >> Best Regards,
>>>> >> Shameera Rathnayaka.
>>>> >>
>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>
>>>>
>>>
>>>
>>> --
>>> The best way to predict the future is to invent it - Alan Kay
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>


-- 
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Shameera Rathnayaka <sh...@gmail.com>.

Hi Supun,

Considering we are going to provide web base GUI support and  JavaScript is
a well-known client side language, I would select JavaScript over Python. I
am not much familiar with Python, so would like to know the advantages we
get by selecting python here.


Thanks,
Shameera.

On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:

> Hi Shameera,
>
> Why you prefer JavaScript over a language like Python?
>
> Thanks,
> Supun..
>
> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi,
>>
>> First of all thanks everyone for giving valuable inputs. After doing some
>> background search and talking to different people in the University who has
>> used different workflow languages, I myself convinced that introducing an
>> another workflow language is not what actually they need. By changing
>> exiting workflow language to another will not solve problems. What they
>> asking is a easy way to construct the workflows. Indirectly what they
>> asking for a sort of API which they can use to generate the workflows and
>> run it. Correct me if i am wrong here.
>>
>> As most of above replies depict,  if we can get a simple API, as an
>> example, for a web based application, JavaScript API would be a good
>> solution, and probably JSON would be a good candidate for language, instead
>> of XML.
>>
>> Airavata community already have started to implement web base GUI. Hence
>> introducing a JSON base JavaScript API would be great help. WDYT?
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> alekny7@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> it is not dataflow instead focused on orchestrating REST services but
>>> you may find it useful datapoint - we created worfklow service that uses
>>> natively JavaScript and JSON to describe what happens during workflow
>>> execution:
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>
>>> HTH,
>>>
>>> Alek
>>>
>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>>>
>>>> Hi Chris,
>>>>
>>>> Great to hear OODT community will be interested in adopting a JSON
>>>> based workflow language and potentially a web based composer as well.
>>>> Airavata previously had BPEL support initially through a home grown
>>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>
>>>> I am of the same opinion that we should adopt something more modern as
>>>> the challenges from scientific workflows seems to be converging with the
>>>> data flow patterns in business workflows.
>>>>
>>>> It will be great if we can all compile a list of potential candidates
>>>> and hack them through.
>>>>
>>>> Suresh
>>>> [1] -
>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>> [2] -
>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>>> [3] -
>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>>
>>>>
>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>
>>>> > Hi Guys,
>>>> >
>>>> > I've been interested in this too - we don't per have a specific
>>>> > OODT workflow language, but we specific workflows using XML, and
>>>> > other configuration (we are also thinking of moving to JSON for
>>>> > this).
>>>> >
>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>> > to me.
>>>> >
>>>> > I wonder at the end of the day if we should adopt something more
>>>> > modern like PIG or some other data flow type of language (PIG
>>>> > is really neat).
>>>> >
>>>> > Cheers,
>>>> > Chris
>>>> >
>>>> >
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Chris Mattmann, Ph.D.
>>>> > Chief Architect
>>>> > Instrument Software and Science Data Systems Section (398)
>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> > Office: 168-519, Mailstop: 168-527
>>>> > Email: chris.a.mattmann@nasa.gov
>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Adjunct Associate Professor, Computer Science Department
>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > -----Original Message-----
>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>> > Reply-To: "architecture@airavata.apache.org"
>>>> > <ar...@airavata.apache.org>
>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>> > To: "architecture@airavata.apache.org" <
>>>> architecture@airavata.apache.org>,
>>>> > dev <de...@airavata.apache.org>
>>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>>> >
>>>> >> Hi All,
>>>> >>
>>>> >> As we all know Airavata has its own workflow language call XWF. When
>>>> XWF
>>>> >> was introduced, main focus points are interoperability and
>>>> convertibility.
>>>> >> But with years of experience it is convinced that above requirements
>>>> are
>>>> >> not really useful when we come to real world use cases. And XWF is
>>>> XML
>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>> self.
>>>> >> But
>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>> >>
>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>> industry
>>>> >> and
>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>> come up
>>>> >> with idea how we should improve Airavata workflow language, either
>>>> we can
>>>> >> improve existing XWF language, totally change to a new language
>>>> available
>>>> >> in industry or write a new light weight language. Basic requirements
>>>> that
>>>> >> we expect from new improvement are, high usability, flexible, light
>>>> weight
>>>> >> and real time monitoring support. As you can see above requirements
>>>> are
>>>> >> not
>>>> >> direct comes with workflow languages but we need workflow language
>>>> which
>>>> >> help to support above requirements.
>>>> >>
>>>> >> After reading few papers and googling, initially i have come up with
>>>> >> following three existing languages,
>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>> >> 2. WS-BPEL
>>>> >> 3. SIDL
>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>> >>
>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>> Radical-SAGA
>>>> >> also uses slightly modified version of SIDL. Other than above three
>>>> >> languages we can come up with simple workflow language base on
>>>> json(or
>>>> >> yaml) which support all our requirements for some extends.
>>>> >>
>>>> >> It would be grate if I can get more input regarding the $Subject
>>>> form the
>>>> >> airavata community. You all are more than welcome to provide any
>>>> type of
>>>> >> suggestions.
>>>> >>
>>>> >> Thanks,
>>>> >> Shameera.
>>>> >>
>>>> >> 
>>>> >>
>>>> >> --
>>>> >> Best Regards,
>>>> >> Shameera Rathnayaka.
>>>> >>
>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>
>>>>
>>>
>>>
>>> --
>>> The best way to predict the future is to invent it - Alan Kay
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>


-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Shameera Rathnayaka <sh...@gmail.com>.

Hi Supun,

Considering we are going to provide web base GUI support and  JavaScript is
a well-known client side language, I would select JavaScript over Python. I
am not much familiar with Python, so would like to know the advantages we
get by selecting python here.


Thanks,
Shameera.

On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:

> Hi Shameera,
>
> Why you prefer JavaScript over a language like Python?
>
> Thanks,
> Supun..
>
> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi,
>>
>> First of all thanks everyone for giving valuable inputs. After doing some
>> background search and talking to different people in the University who has
>> used different workflow languages, I myself convinced that introducing an
>> another workflow language is not what actually they need. By changing
>> exiting workflow language to another will not solve problems. What they
>> asking is a easy way to construct the workflows. Indirectly what they
>> asking for a sort of API which they can use to generate the workflows and
>> run it. Correct me if i am wrong here.
>>
>> As most of above replies depict,  if we can get a simple API, as an
>> example, for a web based application, JavaScript API would be a good
>> solution, and probably JSON would be a good candidate for language, instead
>> of XML.
>>
>> Airavata community already have started to implement web base GUI. Hence
>> introducing a JSON base JavaScript API would be great help. WDYT?
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> alekny7@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> it is not dataflow instead focused on orchestrating REST services but
>>> you may find it useful datapoint - we created worfklow service that uses
>>> natively JavaScript and JSON to describe what happens during workflow
>>> execution:
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>
>>> HTH,
>>>
>>> Alek
>>>
>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>>>
>>>> Hi Chris,
>>>>
>>>> Great to hear OODT community will be interested in adopting a JSON
>>>> based workflow language and potentially a web based composer as well.
>>>> Airavata previously had BPEL support initially through a home grown
>>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>
>>>> I am of the same opinion that we should adopt something more modern as
>>>> the challenges from scientific workflows seems to be converging with the
>>>> data flow patterns in business workflows.
>>>>
>>>> It will be great if we can all compile a list of potential candidates
>>>> and hack them through.
>>>>
>>>> Suresh
>>>> [1] -
>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>> [2] -
>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>>> [3] -
>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>>
>>>>
>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>
>>>> > Hi Guys,
>>>> >
>>>> > I've been interested in this too - we don't per have a specific
>>>> > OODT workflow language, but we specific workflows using XML, and
>>>> > other configuration (we are also thinking of moving to JSON for
>>>> > this).
>>>> >
>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>> > to me.
>>>> >
>>>> > I wonder at the end of the day if we should adopt something more
>>>> > modern like PIG or some other data flow type of language (PIG
>>>> > is really neat).
>>>> >
>>>> > Cheers,
>>>> > Chris
>>>> >
>>>> >
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Chris Mattmann, Ph.D.
>>>> > Chief Architect
>>>> > Instrument Software and Science Data Systems Section (398)
>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> > Office: 168-519, Mailstop: 168-527
>>>> > Email: chris.a.mattmann@nasa.gov
>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Adjunct Associate Professor, Computer Science Department
>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > -----Original Message-----
>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>> > Reply-To: "architecture@airavata.apache.org"
>>>> > <ar...@airavata.apache.org>
>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>> > To: "architecture@airavata.apache.org" <
>>>> architecture@airavata.apache.org>,
>>>> > dev <de...@airavata.apache.org>
>>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>>> >
>>>> >> Hi All,
>>>> >>
>>>> >> As we all know Airavata has its own workflow language call XWF. When
>>>> XWF
>>>> >> was introduced, main focus points are interoperability and
>>>> convertibility.
>>>> >> But with years of experience it is convinced that above requirements
>>>> are
>>>> >> not really useful when we come to real world use cases. And XWF is
>>>> XML
>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>> self.
>>>> >> But
>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>> >>
>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>> industry
>>>> >> and
>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>> come up
>>>> >> with idea how we should improve Airavata workflow language, either
>>>> we can
>>>> >> improve existing XWF language, totally change to a new language
>>>> available
>>>> >> in industry or write a new light weight language. Basic requirements
>>>> that
>>>> >> we expect from new improvement are, high usability, flexible, light
>>>> weight
>>>> >> and real time monitoring support. As you can see above requirements
>>>> are
>>>> >> not
>>>> >> direct comes with workflow languages but we need workflow language
>>>> which
>>>> >> help to support above requirements.
>>>> >>
>>>> >> After reading few papers and googling, initially i have come up with
>>>> >> following three existing languages,
>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>> >> 2. WS-BPEL
>>>> >> 3. SIDL
>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>> >>
>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>> Radical-SAGA
>>>> >> also uses slightly modified version of SIDL. Other than above three
>>>> >> languages we can come up with simple workflow language base on
>>>> json(or
>>>> >> yaml) which support all our requirements for some extends.
>>>> >>
>>>> >> It would be grate if I can get more input regarding the $Subject
>>>> form the
>>>> >> airavata community. You all are more than welcome to provide any
>>>> type of
>>>> >> suggestions.
>>>> >>
>>>> >> Thanks,
>>>> >> Shameera.
>>>> >>
>>>> >> 
>>>> >>
>>>> >> --
>>>> >> Best Regards,
>>>> >> Shameera Rathnayaka.
>>>> >>
>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>
>>>>
>>>
>>>
>>> --
>>> The best way to predict the future is to invent it - Alan Kay
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>


-- 
Best Regards,
Shameera Rathnayaka.

email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Supun Kamburugamuva <su...@gmail.com>.

Once we had an offline discussion about the Airavata Workflow language
(with Milinda, Saliya and Shameera). In that discussion one thing came out
was why we need to invent a different language when a simple library like
Python will full fill of Airavata requirements.

There are many benefits in using a Python library as the API for
controlling Airavata workflows.

1. It is a library, gives the ultimate control over the execution and it
can be simpler than any domain specific language that we can come with like
XML, JSON etc
2. Most people use python and can learn it easily than any Domain specific
language
3. You can easily create a Python library for Airavata because all the APIs
are thrift based.
4. If you design the constructs correctly you can plug an XBaya.

Any thoughts?

Thanks,
Supun..


On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <su...@gmail.com>
wrote:

> Hi Shameera,
>
> Why you prefer JavaScript over a language like Python?
>
> Thanks,
> Supun..
>
> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>> Hi,
>>
>> First of all thanks everyone for giving valuable inputs. After doing some
>> background search and talking to different people in the University who has
>> used different workflow languages, I myself convinced that introducing an
>> another workflow language is not what actually they need. By changing
>> exiting workflow language to another will not solve problems. What they
>> asking is a easy way to construct the workflows. Indirectly what they
>> asking for a sort of API which they can use to generate the workflows and
>> run it. Correct me if i am wrong here.
>>
>> As most of above replies depict,  if we can get a simple API, as an
>> example, for a web based application, JavaScript API would be a good
>> solution, and probably JSON would be a good candidate for language, instead
>> of XML.
>>
>> Airavata community already have started to implement web base GUI. Hence
>> introducing a JSON base JavaScript API would be great help. WDYT?
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> alekny7@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> it is not dataflow instead focused on orchestrating REST services but
>>> you may find it useful datapoint - we created worfklow service that uses
>>> natively JavaScript and JSON to describe what happens during workflow
>>> execution:
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>
>>> HTH,
>>>
>>> Alek
>>>
>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>>>
>>>> Hi Chris,
>>>>
>>>> Great to hear OODT community will be interested in adopting a JSON
>>>> based workflow language and potentially a web based composer as well.
>>>> Airavata previously had BPEL support initially through a home grown
>>>> implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>
>>>> I am of the same opinion that we should adopt something more modern as
>>>> the challenges from scientific workflows seems to be converging with the
>>>> data flow patterns in business workflows.
>>>>
>>>> It will be great if we can all compile a list of potential candidates
>>>> and hack them through.
>>>>
>>>> Suresh
>>>> [1] -
>>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>>> [2] -
>>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>>> [3] -
>>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>>
>>>>
>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>
>>>> > Hi Guys,
>>>> >
>>>> > I've been interested in this too - we don't per have a specific
>>>> > OODT workflow language, but we specific workflows using XML, and
>>>> > other configuration (we are also thinking of moving to JSON for
>>>> > this).
>>>> >
>>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>>> > to me.
>>>> >
>>>> > I wonder at the end of the day if we should adopt something more
>>>> > modern like PIG or some other data flow type of language (PIG
>>>> > is really neat).
>>>> >
>>>> > Cheers,
>>>> > Chris
>>>> >
>>>> >
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Chris Mattmann, Ph.D.
>>>> > Chief Architect
>>>> > Instrument Software and Science Data Systems Section (398)
>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> > Office: 168-519, Mailstop: 168-527
>>>> > Email: chris.a.mattmann@nasa.gov
>>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> > Adjunct Associate Professor, Computer Science Department
>>>> > University of Southern California, Los Angeles, CA 90089 USA
>>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > -----Original Message-----
>>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>>> > Reply-To: "architecture@airavata.apache.org"
>>>> > <ar...@airavata.apache.org>
>>>> > Date: Thursday, September 18, 2014 8:26 AM
>>>> > To: "architecture@airavata.apache.org" <
>>>> architecture@airavata.apache.org>,
>>>> > dev <de...@airavata.apache.org>
>>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>>> >
>>>> >> Hi All,
>>>> >>
>>>> >> As we all know Airavata has its own workflow language call XWF. When
>>>> XWF
>>>> >> was introduced, main focus points are interoperability and
>>>> convertibility.
>>>> >> But with years of experience it is convinced that above requirements
>>>> are
>>>> >> not really useful when we come to real world use cases. And XWF is
>>>> XML
>>>> >> based bulky language where we attache WSDLs and Workflow image it
>>>> self.
>>>> >> But
>>>> >> with the recent changes WSDL part is being removed from XWF.
>>>> >>
>>>> >> It is worth to evaluate handy Scientific workflow languages in
>>>> industry
>>>> >> and
>>>> >> find out pros and cons, at the end of this evaluation we need to
>>>> come up
>>>> >> with idea how we should improve Airavata workflow language, either
>>>> we can
>>>> >> improve existing XWF language, totally change to a new language
>>>> available
>>>> >> in industry or write a new light weight language. Basic requirements
>>>> that
>>>> >> we expect from new improvement are, high usability, flexible, light
>>>> weight
>>>> >> and real time monitoring support. As you can see above requirements
>>>> are
>>>> >> not
>>>> >> direct comes with workflow languages but we need workflow language
>>>> which
>>>> >> help to support above requirements.
>>>> >>
>>>> >> After reading few papers and googling, initially i have come up with
>>>> >> following three existing languages,
>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>>> >> 2. WS-BPEL
>>>> >> 3. SIDL
>>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>>> >>
>>>> >> In my opinion SIDL is more familiar with scientific domain,
>>>> Radical-SAGA
>>>> >> also uses slightly modified version of SIDL. Other than above three
>>>> >> languages we can come up with simple workflow language base on
>>>> json(or
>>>> >> yaml) which support all our requirements for some extends.
>>>> >>
>>>> >> It would be grate if I can get more input regarding the $Subject
>>>> form the
>>>> >> airavata community. You all are more than welcome to provide any
>>>> type of
>>>> >> suggestions.
>>>> >>
>>>> >> Thanks,
>>>> >> Shameera.
>>>> >>
>>>> >> 
>>>> >>
>>>> >> --
>>>> >> Best Regards,
>>>> >> Shameera Rathnayaka.
>>>> >>
>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>>
>>>>
>>>
>>>
>>> --
>>> The best way to predict the future is to invent it - Alan Kay
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>


-- 
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Supun Kamburugamuva <su...@gmail.com>.

Hi Shameera,

Why you prefer JavaScript over a language like Python?

Thanks,
Supun..

On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:

> Hi,
>
> First of all thanks everyone for giving valuable inputs. After doing some
> background search and talking to different people in the University who has
> used different workflow languages, I myself convinced that introducing an
> another workflow language is not what actually they need. By changing
> exiting workflow language to another will not solve problems. What they
> asking is a easy way to construct the workflows. Indirectly what they
> asking for a sort of API which they can use to generate the workflows and
> run it. Correct me if i am wrong here.
>
> As most of above replies depict,  if we can get a simple API, as an
> example, for a web based application, JavaScript API would be a good
> solution, and probably JSON would be a good candidate for language, instead
> of XML.
>
> Airavata community already have started to implement web base GUI. Hence
> introducing a JSON base JavaScript API would be great help. WDYT?
>
> Thanks,
> Shameera.
>
>
> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> alekny7@gmail.com> wrote:
>
>> Hi,
>>
>> it is not dataflow instead focused on orchestrating REST services but you
>> may find it useful datapoint - we created worfklow service that uses
>> natively JavaScript and JSON to describe what happens during workflow
>> execution:
>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>
>> HTH,
>>
>> Alek
>>
>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>>
>>> Hi Chris,
>>>
>>> Great to hear OODT community will be interested in adopting a JSON based
>>> workflow language and potentially a web based composer as well. Airavata
>>> previously had BPEL support initially through a home grown implementation
>>> [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper
>>> [3] by Alek on this topic is an interesting read.
>>>
>>> I am of the same opinion that we should adopt something more modern as
>>> the challenges from scientific workflows seems to be converging with the
>>> data flow patterns in business workflows.
>>>
>>> It will be great if we can all compile a list of potential candidates
>>> and hack them through.
>>>
>>> Suresh
>>> [1] -
>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>> [2] -
>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>> [3] -
>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>
>>>
>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>
>>> > Hi Guys,
>>> >
>>> > I've been interested in this too - we don't per have a specific
>>> > OODT workflow language, but we specific workflows using XML, and
>>> > other configuration (we are also thinking of moving to JSON for
>>> > this).
>>> >
>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>> > to me.
>>> >
>>> > I wonder at the end of the day if we should adopt something more
>>> > modern like PIG or some other data flow type of language (PIG
>>> > is really neat).
>>> >
>>> > Cheers,
>>> > Chris
>>> >
>>> >
>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > Chris Mattmann, Ph.D.
>>> > Chief Architect
>>> > Instrument Software and Science Data Systems Section (398)
>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> > Office: 168-519, Mailstop: 168-527
>>> > Email: chris.a.mattmann@nasa.gov
>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > Adjunct Associate Professor, Computer Science Department
>>> > University of Southern California, Los Angeles, CA 90089 USA
>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>> > Reply-To: "architecture@airavata.apache.org"
>>> > <ar...@airavata.apache.org>
>>> > Date: Thursday, September 18, 2014 8:26 AM
>>> > To: "architecture@airavata.apache.org" <
>>> architecture@airavata.apache.org>,
>>> > dev <de...@airavata.apache.org>
>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>> >
>>> >> Hi All,
>>> >>
>>> >> As we all know Airavata has its own workflow language call XWF. When
>>> XWF
>>> >> was introduced, main focus points are interoperability and
>>> convertibility.
>>> >> But with years of experience it is convinced that above requirements
>>> are
>>> >> not really useful when we come to real world use cases. And XWF is XML
>>> >> based bulky language where we attache WSDLs and Workflow image it
>>> self.
>>> >> But
>>> >> with the recent changes WSDL part is being removed from XWF.
>>> >>
>>> >> It is worth to evaluate handy Scientific workflow languages in
>>> industry
>>> >> and
>>> >> find out pros and cons, at the end of this evaluation we need to come
>>> up
>>> >> with idea how we should improve Airavata workflow language, either we
>>> can
>>> >> improve existing XWF language, totally change to a new language
>>> available
>>> >> in industry or write a new light weight language. Basic requirements
>>> that
>>> >> we expect from new improvement are, high usability, flexible, light
>>> weight
>>> >> and real time monitoring support. As you can see above requirements
>>> are
>>> >> not
>>> >> direct comes with workflow languages but we need workflow language
>>> which
>>> >> help to support above requirements.
>>> >>
>>> >> After reading few papers and googling, initially i have come up with
>>> >> following three existing languages,
>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>> >> 2. WS-BPEL
>>> >> 3. SIDL
>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>> >>
>>> >> In my opinion SIDL is more familiar with scientific domain,
>>> Radical-SAGA
>>> >> also uses slightly modified version of SIDL. Other than above three
>>> >> languages we can come up with simple workflow language base on json(or
>>> >> yaml) which support all our requirements for some extends.
>>> >>
>>> >> It would be grate if I can get more input regarding the $Subject form
>>> the
>>> >> airavata community. You all are more than welcome to provide any type
>>> of
>>> >> suggestions.
>>> >>
>>> >> Thanks,
>>> >> Shameera.
>>> >>
>>> >> 
>>> >>
>>> >> --
>>> >> Best Regards,
>>> >> Shameera Rathnayaka.
>>> >>
>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>>
>>
>>
>> --
>> The best way to predict the future is to invent it - Alan Kay
>>
>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>



-- 
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Supun Kamburugamuva <su...@gmail.com>.

Hi Shameera,

Why you prefer JavaScript over a language like Python?

Thanks,
Supun..

On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:

> Hi,
>
> First of all thanks everyone for giving valuable inputs. After doing some
> background search and talking to different people in the University who has
> used different workflow languages, I myself convinced that introducing an
> another workflow language is not what actually they need. By changing
> exiting workflow language to another will not solve problems. What they
> asking is a easy way to construct the workflows. Indirectly what they
> asking for a sort of API which they can use to generate the workflows and
> run it. Correct me if i am wrong here.
>
> As most of above replies depict,  if we can get a simple API, as an
> example, for a web based application, JavaScript API would be a good
> solution, and probably JSON would be a good candidate for language, instead
> of XML.
>
> Airavata community already have started to implement web base GUI. Hence
> introducing a JSON base JavaScript API would be great help. WDYT?
>
> Thanks,
> Shameera.
>
>
> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> alekny7@gmail.com> wrote:
>
>> Hi,
>>
>> it is not dataflow instead focused on orchestrating REST services but you
>> may find it useful datapoint - we created worfklow service that uses
>> natively JavaScript and JSON to describe what happens during workflow
>> execution:
>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>
>> HTH,
>>
>> Alek
>>
>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>>
>>> Hi Chris,
>>>
>>> Great to hear OODT community will be interested in adopting a JSON based
>>> workflow language and potentially a web based composer as well. Airavata
>>> previously had BPEL support initially through a home grown implementation
>>> [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper
>>> [3] by Alek on this topic is an interesting read.
>>>
>>> I am of the same opinion that we should adopt something more modern as
>>> the challenges from scientific workflows seems to be converging with the
>>> data flow patterns in business workflows.
>>>
>>> It will be great if we can all compile a list of potential candidates
>>> and hack them through.
>>>
>>> Suresh
>>> [1] -
>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>> [2] -
>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>> [3] -
>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>
>>>
>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>
>>> > Hi Guys,
>>> >
>>> > I've been interested in this too - we don't per have a specific
>>> > OODT workflow language, but we specific workflows using XML, and
>>> > other configuration (we are also thinking of moving to JSON for
>>> > this).
>>> >
>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>> > to me.
>>> >
>>> > I wonder at the end of the day if we should adopt something more
>>> > modern like PIG or some other data flow type of language (PIG
>>> > is really neat).
>>> >
>>> > Cheers,
>>> > Chris
>>> >
>>> >
>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > Chris Mattmann, Ph.D.
>>> > Chief Architect
>>> > Instrument Software and Science Data Systems Section (398)
>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> > Office: 168-519, Mailstop: 168-527
>>> > Email: chris.a.mattmann@nasa.gov
>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > Adjunct Associate Professor, Computer Science Department
>>> > University of Southern California, Los Angeles, CA 90089 USA
>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>> > Reply-To: "architecture@airavata.apache.org"
>>> > <ar...@airavata.apache.org>
>>> > Date: Thursday, September 18, 2014 8:26 AM
>>> > To: "architecture@airavata.apache.org" <
>>> architecture@airavata.apache.org>,
>>> > dev <de...@airavata.apache.org>
>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>> >
>>> >> Hi All,
>>> >>
>>> >> As we all know Airavata has its own workflow language call XWF. When
>>> XWF
>>> >> was introduced, main focus points are interoperability and
>>> convertibility.
>>> >> But with years of experience it is convinced that above requirements
>>> are
>>> >> not really useful when we come to real world use cases. And XWF is XML
>>> >> based bulky language where we attache WSDLs and Workflow image it
>>> self.
>>> >> But
>>> >> with the recent changes WSDL part is being removed from XWF.
>>> >>
>>> >> It is worth to evaluate handy Scientific workflow languages in
>>> industry
>>> >> and
>>> >> find out pros and cons, at the end of this evaluation we need to come
>>> up
>>> >> with idea how we should improve Airavata workflow language, either we
>>> can
>>> >> improve existing XWF language, totally change to a new language
>>> available
>>> >> in industry or write a new light weight language. Basic requirements
>>> that
>>> >> we expect from new improvement are, high usability, flexible, light
>>> weight
>>> >> and real time monitoring support. As you can see above requirements
>>> are
>>> >> not
>>> >> direct comes with workflow languages but we need workflow language
>>> which
>>> >> help to support above requirements.
>>> >>
>>> >> After reading few papers and googling, initially i have come up with
>>> >> following three existing languages,
>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>> >> 2. WS-BPEL
>>> >> 3. SIDL
>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>> >>
>>> >> In my opinion SIDL is more familiar with scientific domain,
>>> Radical-SAGA
>>> >> also uses slightly modified version of SIDL. Other than above three
>>> >> languages we can come up with simple workflow language base on json(or
>>> >> yaml) which support all our requirements for some extends.
>>> >>
>>> >> It would be grate if I can get more input regarding the $Subject form
>>> the
>>> >> airavata community. You all are more than welcome to provide any type
>>> of
>>> >> suggestions.
>>> >>
>>> >> Thanks,
>>> >> Shameera.
>>> >>
>>> >> 
>>> >>
>>> >> --
>>> >> Best Regards,
>>> >> Shameera Rathnayaka.
>>> >>
>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>>
>>
>>
>> --
>> The best way to predict the future is to invent it - Alan Kay
>>
>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>



-- 
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

Posted by Supun Kamburugamuva <su...@gmail.com>.

Hi Shameera,

Why you prefer JavaScript over a language like Python?

Thanks,
Supun..

On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:

> Hi,
>
> First of all thanks everyone for giving valuable inputs. After doing some
> background search and talking to different people in the University who has
> used different workflow languages, I myself convinced that introducing an
> another workflow language is not what actually they need. By changing
> exiting workflow language to another will not solve problems. What they
> asking is a easy way to construct the workflows. Indirectly what they
> asking for a sort of API which they can use to generate the workflows and
> run it. Correct me if i am wrong here.
>
> As most of above replies depict,  if we can get a simple API, as an
> example, for a web based application, JavaScript API would be a good
> solution, and probably JSON would be a good candidate for language, instead
> of XML.
>
> Airavata community already have started to implement web base GUI. Hence
> introducing a JSON base JavaScript API would be great help. WDYT?
>
> Thanks,
> Shameera.
>
>
> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> alekny7@gmail.com> wrote:
>
>> Hi,
>>
>> it is not dataflow instead focused on orchestrating REST services but you
>> may find it useful datapoint - we created worfklow service that uses
>> natively JavaScript and JSON to describe what happens during workflow
>> execution:
>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>
>> HTH,
>>
>> Alek
>>
>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <sm...@apache.org> wrote:
>>
>>> Hi Chris,
>>>
>>> Great to hear OODT community will be interested in adopting a JSON based
>>> workflow language and potentially a web based composer as well. Airavata
>>> previously had BPEL support initially through a home grown implementation
>>> [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper
>>> [3] by Alek on this topic is an interesting read.
>>>
>>> I am of the same opinion that we should adopt something more modern as
>>> the challenges from scientific workflows seems to be converging with the
>>> data flow patterns in business workflows.
>>>
>>> It will be great if we can all compile a list of potential candidates
>>> and hack them through.
>>>
>>> Suresh
>>> [1] -
>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>> [2] -
>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>> [3] -
>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>
>>>
>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>>
>>> > Hi Guys,
>>> >
>>> > I've been interested in this too - we don't per have a specific
>>> > OODT workflow language, but we specific workflows using XML, and
>>> > other configuration (we are also thinking of moving to JSON for
>>> > this).
>>> >
>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>> > to me.
>>> >
>>> > I wonder at the end of the day if we should adopt something more
>>> > modern like PIG or some other data flow type of language (PIG
>>> > is really neat).
>>> >
>>> > Cheers,
>>> > Chris
>>> >
>>> >
>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > Chris Mattmann, Ph.D.
>>> > Chief Architect
>>> > Instrument Software and Science Data Systems Section (398)
>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> > Office: 168-519, Mailstop: 168-527
>>> > Email: chris.a.mattmann@nasa.gov
>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> > Adjunct Associate Professor, Computer Science Department
>>> > University of Southern California, Los Angeles, CA 90089 USA
>>> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: Shameera Rathnayaka <sh...@gmail.com>
>>> > Reply-To: "architecture@airavata.apache.org"
>>> > <ar...@airavata.apache.org>
>>> > Date: Thursday, September 18, 2014 8:26 AM
>>> > To: "architecture@airavata.apache.org" <
>>> architecture@airavata.apache.org>,
>>> > dev <de...@airavata.apache.org>
>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>> >
>>> >> Hi All,
>>> >>
>>> >> As we all know Airavata has its own workflow language call XWF. When
>>> XWF
>>> >> was introduced, main focus points are interoperability and
>>> convertibility.
>>> >> But with years of experience it is convinced that above requirements
>>> are
>>> >> not really useful when we come to real world use cases. And XWF is XML
>>> >> based bulky language where we attache WSDLs and Workflow image it
>>> self.
>>> >> But
>>> >> with the recent changes WSDL part is being removed from XWF.
>>> >>
>>> >> It is worth to evaluate handy Scientific workflow languages in
>>> industry
>>> >> and
>>> >> find out pros and cons, at the end of this evaluation we need to come
>>> up
>>> >> with idea how we should improve Airavata workflow language, either we
>>> can
>>> >> improve existing XWF language, totally change to a new language
>>> available
>>> >> in industry or write a new light weight language. Basic requirements
>>> that
>>> >> we expect from new improvement are, high usability, flexible, light
>>> weight
>>> >> and real time monitoring support. As you can see above requirements
>>> are
>>> >> not
>>> >> direct comes with workflow languages but we need workflow language
>>> which
>>> >> help to support above requirements.
>>> >>
>>> >> After reading few papers and googling, initially i have come up with
>>> >> following three existing languages,
>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>>> >> 2. WS-BPEL
>>> >> 3. SIDL
>>> >> <http://computation.llnl.gov/casc/components/index.html#page=home>
>>> >>
>>> >> In my opinion SIDL is more familiar with scientific domain,
>>> Radical-SAGA
>>> >> also uses slightly modified version of SIDL. Other than above three
>>> >> languages we can come up with simple workflow language base on json(or
>>> >> yaml) which support all our requirements for some extends.
>>> >>
>>> >> It would be grate if I can get more input regarding the $Subject form
>>> the
>>> >> airavata community. You all are more than welcome to provide any type
>>> of
>>> >> suggestions.
>>> >>
>>> >> Thanks,
>>> >> Shameera.
>>> >>
>>> >> 
>>> >>
>>> >> --
>>> >> Best Regards,
>>> >> Shameera Rathnayaka.
>>> >>
>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>>>
>>>
>>
>>
>> --
>> The best way to predict the future is to invent it - Alan Kay
>>
>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>



-- 
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun06@gmail.com;  Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com