You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airavata.apache.org by sm...@apache.org on 2011/10/04 19:48:37 UTC
svn commit: r1178878 - in /incubator/airavata/site/trunk:
content/airavata/about/ content/airavata/architecture/ templates/
Author: smarru
Date: Tue Oct 4 17:48:36 2011
New Revision: 1178878
URL: http://svn.apache.org/viewvc?rev=1178878&view=rev
Log:
adding architecture docs
Added:
incubator/airavata/site/trunk/content/airavata/architecture/airavata-registry.png (with props)
incubator/airavata/site/trunk/content/airavata/architecture/airavata.png (with props)
incubator/airavata/site/trunk/content/airavata/architecture/messenger-overview.png (with props)
incubator/airavata/site/trunk/content/airavata/architecture/overview.mdtext
incubator/airavata/site/trunk/content/airavata/architecture/registry.mdtext
incubator/airavata/site/trunk/content/airavata/architecture/workflow-suite.png (with props)
incubator/airavata/site/trunk/content/airavata/architecture/workflow.mdtext
incubator/airavata/site/trunk/content/airavata/architecture/ws-messenger.mdtext
Modified:
incubator/airavata/site/trunk/content/airavata/about/history.mdtext
incubator/airavata/site/trunk/templates/sidenav.mdtext
Modified: incubator/airavata/site/trunk/content/airavata/about/history.mdtext
URL: http://svn.apache.org/viewvc/incubator/airavata/site/trunk/content/airavata/about/history.mdtext?rev=1178878&r1=1178877&r2=1178878&view=diff
==============================================================================
--- incubator/airavata/site/trunk/content/airavata/about/history.mdtext (original)
+++ incubator/airavata/site/trunk/content/airavata/about/history.mdtext Tue Oct 4 17:48:36 2011
@@ -28,7 +28,7 @@ Cyberinfrastructure systems to enable fa
LEAD goals required to create a dynamically adaptive, on-demand, grid-enabled workflow system
supporting long running applications and on-demand computing. LEAD has subsequently produced close
to 450 research publications combined in all disciplines. The resulting software framework was built
-on the concepts of service oriented architectures powering the LEAD Gateway Portal~\cite{leadportal}.
+on the concepts of service oriented architectures powering the LEAD Gateway Portal.
## Open Gateway Computing Environments
Added: incubator/airavata/site/trunk/content/airavata/architecture/airavata-registry.png
URL: http://svn.apache.org/viewvc/incubator/airavata/site/trunk/content/airavata/architecture/airavata-registry.png?rev=1178878&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/airavata/site/trunk/content/airavata/architecture/airavata-registry.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/airavata/site/trunk/content/airavata/architecture/airavata.png
URL: http://svn.apache.org/viewvc/incubator/airavata/site/trunk/content/airavata/architecture/airavata.png?rev=1178878&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/airavata/site/trunk/content/airavata/architecture/airavata.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/airavata/site/trunk/content/airavata/architecture/messenger-overview.png
URL: http://svn.apache.org/viewvc/incubator/airavata/site/trunk/content/airavata/architecture/messenger-overview.png?rev=1178878&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/airavata/site/trunk/content/airavata/architecture/messenger-overview.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/airavata/site/trunk/content/airavata/architecture/overview.mdtext
URL: http://svn.apache.org/viewvc/incubator/airavata/site/trunk/content/airavata/architecture/overview.mdtext?rev=1178878&view=auto
==============================================================================
--- incubator/airavata/site/trunk/content/airavata/architecture/overview.mdtext (added)
+++ incubator/airavata/site/trunk/content/airavata/architecture/overview.mdtext Tue Oct 4 17:48:36 2011
@@ -0,0 +1,63 @@
+Title: Airavata Gateway Framework
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Apache Airavata is a software toolkit with features to compose, manage, execute, and monitor small
+to large scale applications and workflows on computational resources ranging from local clusters to
+national grids and computing clouds.
+
+Airavata builds on general concepts of service oriented computing, distributed messaging, and
+workflow composition and orchestration. The services primarly communicate using SOAP messages.
+In addition to the basic Web services specifications like SOAP and WSDL, the Web services also
+support WS-Addressing and WS-Messaging. WS-Addressing is used to provide an asynchronous
+communication mechanism for the services to communicate. All the communication channels are secured
+using https as the transport layer.
+
+Airavata features Dynamic Service Binding where in a proxy service accepts an input message intended
+to a target application Web service instance invoked by a workflow engine. The proxy service binds
+an appropriate running Web service instance by looking up in XRegistry. When no Web service
+instances are available, a new instance of the GFac service is created.
+
+Airavata's primary goal is to support long running applications and workflows on distributed computational resources.
+The architecture is designed to be modular, componentized
+software framework as illustrated in the following Figure. The goal of the Airavata framework is minimalist
+architectural design (i.e., a thin layer), a conceptually simple to understand architecture; and easier to install,
+maintain and use. Its salient feature includes using components by themselves or as an integrated solution.
+
+
+
+## Airavata Features
+* desktop tools and browser-based web interface components for managing applications, workflows and generated data.
+* sophisticated server-side tools for registering and managing scientific applications on computational resources.
+* graphical user interfaces to construct, execute, control, manage and reuse of scientific workflows.
+* interfacing and interoperability with with various external (third party) data, workflow and provenance management tools.
+
+## Airavata Components
+* XBaya Workflow Suite - includes a GUI for workflow composition and monitoring. The workflows can be
+interpreted at each step providing dynamic interactive capabilities. The composed workflow can be
+exported to various workflow languages like BPEL, SCUFL, Condor DAG, Jython and Java.
+The defacto workflow enacting engine used is Airavata Workflow Engine. In the future, it will be Apache ODE.
+* GFac - an application wrapper service that can be used to wrap command line-driven science
+applications and make them into robust, network- accessible services. This component is build on
+Axis2 web service stack.
+and constructed workflows.
+* WS-Messenger - a publish-subscribe based message broker implemented on top of Apache Axis2 web
+services stack. It implements the WS-Eventing and WS-Notifications specifications and incorporates
+a message box component that facilities communications with clients behind firewalls and overcomes
+network glitches. As an example, the workflow execution engine sends out events in various stages,
+including start, end, failure, successful invocation, of its workflow execution.
+* Registry-API: A thick client registry API for Airavata to put and get documents. Current JCR implementation is supported by Jack-Rabbit.
\ No newline at end of file
Added: incubator/airavata/site/trunk/content/airavata/architecture/registry.mdtext
URL: http://svn.apache.org/viewvc/incubator/airavata/site/trunk/content/airavata/architecture/registry.mdtext?rev=1178878&view=auto
==============================================================================
--- incubator/airavata/site/trunk/content/airavata/architecture/registry.mdtext (added)
+++ incubator/airavata/site/trunk/content/airavata/architecture/registry.mdtext Tue Oct 4 17:48:36 2011
@@ -0,0 +1,44 @@
+Title: Information and Data Registry
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Airavata provides capabilities for an integrated workflow and data management system. The information and data has to be persisted into a registry
+service. To adhere to the philosophies of Airavata for layering over existing stable third party components, Airavata usage is consolidated into a
+thick-client API. This API allows the pluggability of the infrastructure and avoids locks in into any specific registry architectures.
+This follwing figure summerizes the use of the registry.
+
+
+
+## JCR Specification:
+The Content Repository for Java Technology specification, developed under the Java Community Process JSR-170,
+aims to standardize Java API to repositiries. The specification provides a unified API under the javax.jcr namespace that allows you to
+access any specification-compliant repository implementation in a vendor-neutral manner. A major advantage of JSR-170 is that it is not
+tied to any particular underlying architecture. The back-end data storage for a JSR-170 implementation, for instance, may be a filesystem,
+a WebDAV repository, an XML-backed system, or even an SQL database. Furthermore, the export and import facilities of JSR-170 allow an integrator
+to switch seamlessly between content back ends and JCR implementations. Finally, the JCR provides a straightforward interface that can be layered
+on top of a wide variety of existing content repositories, while simultaneously standardizing complex functionality such as versioning,
+access control, and searching. There are open source implementations of JSR-170 such as Apache Jackrabbit, Alfresco and WSO2 Registry.
+
+##Airavata Registry API
+When implementing registry functionality to store data and retrieve data we did not want to have locking for any custom API's to content
+repository implementations. Airavata tried to integrate with existing repositories without having to re-invent the wheel.
+JCR-2.0 specification and the JSR-170 implementation is plugged in to Apache Airavata system. The Airavata specific API layers over
+the generic JCR API and is integrated with Apache Jackrabbit implementation. The API is modularized as a common component reused by XBaya and
+GFac components to store and retrieve data. In addition to the service information, the registry is used to catalog workflow consumed inputs
+and generated output data. The registry is also used to advertise and retrieve persistent service information. The use of the registry for
+GFac service registration allows for late service binding and supporting large number of applications provided by the inherent load balance
+capabilities.
\ No newline at end of file
Added: incubator/airavata/site/trunk/content/airavata/architecture/workflow-suite.png
URL: http://svn.apache.org/viewvc/incubator/airavata/site/trunk/content/airavata/architecture/workflow-suite.png?rev=1178878&view=auto
==============================================================================
Binary file - no diff available.
Propchange: incubator/airavata/site/trunk/content/airavata/architecture/workflow-suite.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: incubator/airavata/site/trunk/content/airavata/architecture/workflow.mdtext
URL: http://svn.apache.org/viewvc/incubator/airavata/site/trunk/content/airavata/architecture/workflow.mdtext?rev=1178878&view=auto
==============================================================================
--- incubator/airavata/site/trunk/content/airavata/architecture/workflow.mdtext (added)
+++ incubator/airavata/site/trunk/content/airavata/architecture/workflow.mdtext Tue Oct 4 17:48:36 2011
@@ -0,0 +1,76 @@
+Title: Airavata Workflow Suite
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Airavata XBaya workflow system provides a programming model that allows the scientist to program experiments using
+service-oriented architecture that would abstract the complexities of the underlying middleware.
+Airavata XBaya provides interfaces for composition, execution and monitoring of the workflows as illustrated in the following Figure.
+Airavata XBaya consist of convenient GUI interface for workflow composition, workflow enactment
+engine/interface for workflow enactment and workflow monitoring module.
+
+
+
+## Workflow Composition
+Airavata XBaya by design decouples composition and monitoring from the orchestration of the workflow although it does provide an embedded workflow
+enactment engine integrated with the workbench. As a scientific workflow suite XBaya is often expected to run long running computations thus it
+often delegates the workflow execution to a persistent orchestration engine and XBaya workbench can monitor the progress of the workflow asynchronously. The workbench provides a convenient drag and drop GUI for SOA based service composition along with other functionalities like service discovery, registry lookup and workflow experiment management.
+
+## Workflow orchestration
+XBaya provides a unique pluggable architecture for selecting the orchestration engine. When a user composes a workflow using XBaya workbench
+it builds an abstract Directed Acyclic Graph (DAG) which is independent of any workflow runtime. There are pluggable compiler modules that are
+capable of producing workflow execution scripts for target runtimes. The above Figure illustrated how the DAG can compiled into different runtimes.
+Currently XBaya supports BPEL 2.0 and Apache ODE; SCUFL and Taverna; DAGMAN and Pegasus; Jython scripts and XBaya Interpreter Service.
+
+Each of these workflow runtimes have their own strengths, for example, Apache ODE is a robust SOA based workflow orchestration engine well suited
+for long running applications. Pegasus workflow system is well suited for parametric sweeps whereas XBaya Interpreter engine is strong in dynamic
+and user interactive workflow execution. It is also capable of generating a Jython script that can be executed in any Jython runtime independent
+of any of the workflow infrastructure.
+
+## Workflow interpreter
+XBaya provides a inbuilt workflow enactment engine that provides extremely dynamic and interactive workflow execution.
+As the name suggests the workflow interpreter provides an interpreted workflow execution framework rather than the compiled workflow execution
+environments. In this context the interpreted workflow means at that execution framework executed tasks in the XBaya DAG as and when the tasks
+become runnable. Once the user composes the workflow and launches the workflow the workflow interpreter will start executing the workflow DAG.
+Topological sort of the DAG provides execution ordering of the tasks which allow the identification of the initial ready task set and incrementally
+adding new tasks to the ready list as previous tasks finish execution. What is unique about this type of execution model is it allows the user
+to pause the execution of the workflow as necessary and update the DAGâs execution states or even the DAG itself and resume execution and the
+changes are immediately picked up by the workflow interpreter.
+
+Workflow interpreter can be used as an embedded workflow enactment engine within XBaya GUI or as an interpreter service that may run as a persistent
+service. The functionality provided by the workflow interpreter allows fine gain control over the workflow execution which translates into following
+functionalities.
+* Static workflow interactions
+* Dynamic change workflow inputs, workflow rerun.
+* Stop, pause, resume execution of the workflow. Pause workflow at a particular (debug) point
+* Dynamic change in point of execution, workflow smart rerun.
+* Fault handling and exception models
+* Dynamic workflow interactions
+* Reconfiguration of workflow activity
+* Dynamic addition of activities to the DAG.
+* Dynamic remove or replace of activity to the DAG.
+
+## Workflow monitoring:
+XBaya workbench allows the user to monitor the progress of the workflow in real-time. Monitoring includes the services which are currently executing,
+those that are done. It also provides state of the job submissions to the batch job queues and status of the data transfer. Workflow monitoring in
+XBaya workbench work in two modes; synchronous mode and asynchronous mode. The synchronous mode only applies when XBaya interpreter runs embedded in
+the workbench. This allows the execution engine to be aware of which task are ready, running, finished and waiting and thus accordingly update the
+DAG execution state. But when the workflow execution is delegated to a persistant service like Apache ODE serve or XBaya interpreter service,
+it is necessary to have asynchronous monitoring. This is achieved using the WS-Eventing based notification messages published by different components
+of the workflow system. If a user launches a workflow instance a unique topic will be associated with that instance and all the workflow components
+like workflow engine, Factory service will publish status notification about the workflow to the event bus using the topic assigned to the workflow
+instance. This asynchronous monitoring is truly asynchronous where the user can launch the workflow, start monitoring, loose internet connectivity and
+once reconnected the XBaya workbench will bring back the monitoring to the current execution state of the workflow.
Added: incubator/airavata/site/trunk/content/airavata/architecture/ws-messenger.mdtext
URL: http://svn.apache.org/viewvc/incubator/airavata/site/trunk/content/airavata/architecture/ws-messenger.mdtext?rev=1178878&view=auto
==============================================================================
--- incubator/airavata/site/trunk/content/airavata/architecture/ws-messenger.mdtext (added)
+++ incubator/airavata/site/trunk/content/airavata/architecture/ws-messenger.mdtext Tue Oct 4 17:48:36 2011
@@ -0,0 +1,80 @@
+Title: WS-Messenger System
+Notice: Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+ .
+ http://www.apache.org/licenses/LICENSE-2.0
+ .
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Airavata WS-Messenger is a Web Services-based Messaging System for Service-Oriented Computing and is
+an implementation of [WS-Notification][wsn] and [WS-Eventing][wse] specifications. The message format
+can be automatically converted between the two specifications. The subscription request type
+(Ws-Notification or WS-Eventing) for a listener determines the message format that the notification
+consumer will receive. If WS-Notification subscription request is received by the broker, it will
+send WS-Notification messages to the listener. Similarly, if WS-Eventing subscription is received
+by the broker, the broker will produce WS-Eventing messages for that listener. The publisher can
+publish messages in either format. The consumers will recieve messages in the format they subscribed
+irrespective of the format the message is published.
+
+The WS-Messenger guarantees scalable, reliable and efficient message delivery. WS-Messenger contains
+sub-components of Message broker, Message box and WS-Messenger client libraries.
+WS-Messenger acts like the nervous system within Airavata infrastructure but can very well be integrated
+into any WS-Eventing/WS-Notification standard compliant infrastructures. The Messenger is deployed as a
+Axis2 service providing inherent support for different transports including HTTP 1.0, HTTP 1.1 and JMS.
+Custom transport protocols can be plugged in using the Axis2 framework's transport plug-ins.
+WS-Messenger is integrated with Mysql and Apache Derby database implementations.
+The embedded Derby database is deployed within the same Java process. The system also supports persistent
+subscriptions and notifications which will tolerate failures and restarts of the application server.
+
+The following figure illustrates the high level overview of the WS-Messenger
+
+
+## WS-Messenger Modules
+
+1. Message Broker: The broker handles filtering/delivering messages to even sinks and subscription management.
+WS-Messenger support filtering messages based on XPath expression and topic expression. Message broker is implemented on
+Web services specifications and provide the mediation between WS-Eventing Specification and WS-Notification Specifications.
+WS-Eventing and WS-Notification specifications are two competing specifications defining the interfaces for Web services-based
+publish/subscribe systems. The Web Services interfaces and the notification message formats defined in these two
+specifications are fundamentally different. WS-Messenger provides a mediation approach facilitating interoperability with supported services.
+WS-Messenger can automatically convert the notification message formats to makes sure that the message âon the wireâ can be understood by the
+event consumers. For example, a WS-Notification consumer can receive notification messages published by a WS-Eventing publisher and vice versa.
+The event consumer type is determined by the subscription request message type. Here we assume that the subscribers and the event
+consumers use the same specification. This is a valid assumption for most cases since the subscribers needs to know the location of the event consumers.
+The message delivering schema is implemented by consumer blacklisting to minimize the overhead of message delivering failures.
+Message delivering performance has been implemented with a parallel message delivering sub system. The blocking I/O overhead is reduced
+configuring the NIO transport. WS-Messenger uses the NIO transport provided by Apache Synapse project to enable non-blocking I/O in Axis2.
+3. Message Box: WS-Messenger decouples the consumer tolerating network failures and firewalls.
+The Message box acts like an intermediate notification storage between broker and the event sink. Users can simply create a message box and
+provide the message box URL as the event sink URL during the subscription. When messages are publishing to message broker notifications will
+send to the message box and it stores the incoming messages in to database or keep them in memory based on the WS-Messenger configuration.
+Message box Service has operations to pull the messages from the message box so whenever actual event sink is ready to pull the messages out
+from message box they simply have to invoke the message Pulling API and take the notifications. When the message box is no longer needed,
+the subscriber can send another web service request to destroy the message box.This is a useful feature in a scenario where event source cannot
+send notifications directly to even sink running behind the firewall. This deployment is a very common firewall configuration between DMZ and green zone
+in an enterprise deployment.provide a web service interface for subscribers behind the firewalls.
+
+## Airavata Usage of the WS-Messenger
+Within Airavata, the WS-Messenger is used for user and system communications between GFac, XBaya and Workflow Intepreter.
+XBaya in monitoring mode subscribes to a pre-specified topic to which Workflow Engine and GFac are publishing status notifications.
+XBaya is instrumented to react to the messages to illustrate workflow progress by means of color coding of tasks being executed.
+
+The following steps illustrates the usage in detail:
+* XBaya creates a message Box, subscribes to it and then invokes a workflow, with the created message box EPR passed in context.
+* Then XBaya sends a subscription request to WS-Messenger with a particular topic and message box URL received as the event sink.
+* Then workflow notifiers in XBaya send the notifications using the same topic to the message broker.
+* Individual application services created by GFac are instrumented to publish detailed status and send progress to the EPR received in workflow execution context.
+* XBaya translates the messages into graphical colors and input and output extractions.
+
+ [wsn]: http://www.oasis-open.org/committees/wsn/
+ [wse]: http://www.w3.org/Submission/WS-Eventing/
Modified: incubator/airavata/site/trunk/templates/sidenav.mdtext
URL: http://svn.apache.org/viewvc/incubator/airavata/site/trunk/templates/sidenav.mdtext?rev=1178878&r1=1178877&r2=1178878&view=diff
==============================================================================
--- incubator/airavata/site/trunk/templates/sidenav.mdtext (original)
+++ incubator/airavata/site/trunk/templates/sidenav.mdtext Tue Oct 4 17:48:36 2011
@@ -8,9 +8,10 @@
# Architecture
- [Airavata Overview](/airavata/architecture/overview.html)
- - [WS Messenger](/airavata/architecture/ws-messenger.html)
- - [XBaya](/airavata/architecture/workflow/xbaya.html)
- [GFac](/airavata/architecture/gfac/gfac.html)
+ - [Workflow Suite](/airavata/architecture/workflow.html)
+ - [WS Messenger](/airavata/architecture/ws-messenger.html)
+ - [Registry API](/airavata/architecture/registry.html)
# Community