You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@taverna.apache.org by st...@apache.org on 2016/05/04 17:00:41 UTC
[5/5] incubator-taverna-engine git commit: README - document the
modules
README - document the modules
Project: http://git-wip-us.apache.org/repos/asf/incubator-taverna-engine/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-taverna-engine/commit/896917e8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-taverna-engine/tree/896917e8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-taverna-engine/diff/896917e8
Branch: refs/heads/master
Commit: 896917e8425dd45470ffe6cc5891bbc71adfa2e3
Parents: 35c29d1
Author: Stian Soiland-Reyes <st...@apache.org>
Authored: Wed May 4 18:00:06 2016 +0100
Committer: Stian Soiland-Reyes <st...@apache.org>
Committed: Wed May 4 18:00:06 2016 +0100
----------------------------------------------------------------------
README.md | 158 ++++++++++++++++++++++++++++++++++++++++++++++-----------
1 file changed, 128 insertions(+), 30 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-taverna-engine/blob/896917e8/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index a798fe2..f54abab 100644
--- a/README.md
+++ b/README.md
@@ -104,7 +104,7 @@ This will build each module and run their tests.
## Skipping tests
-To skip the tests (these can be timeconsuming), use:
+To skip the tests (these can be time-consuming), use:
mvn clean install -DskipTests
@@ -128,43 +128,137 @@ should only need to depend on the `-api` modules, while the `-impl` are added by
packaging of the
[taverna-commandline-product](https://github.com/apache/incubator-taverna-commandline/tree/master/taverna-commandline-product).
-* [taverna-activity-archetype](taverna-activity-archetype/) Apache Taverna Activity archetype
-* [taverna-activity-test-utils](taverna-activity-test-utils/) Apache Taverna Activity test utils
* [taverna-capability-api](taverna-capability-api/) Apache Taverna Platform Capability API
* [taverna-capability-impl](taverna-capability-impl/) Apache Taverna Platform Capability impl
* [taverna-credential-manager-api](taverna-credential-manager-api/) Apache Taverna Credential Manager API
* [taverna-credential-manager-impl](taverna-credential-manager-impl/) Apache Taverna Credential Manager impl
* [taverna-database-configuration-api](taverna-database-configuration-api/) Apache Taverna Database Configuration API
* [taverna-database-configuration-impl](taverna-database-configuration-impl/) Apache Taverna Database Configuration impl
-* [taverna-dataflow-activity](taverna-dataflow-activity/) Apache Taverna Dataflow Activity
* [taverna-execution-api](taverna-execution-api/) Apache Taverna Platform Execution Service API
-* [taverna-execution-hadoop](taverna-execution-hadoop/) Apache Taverna Hadoop Workflow Execution Service
* [taverna-execution-impl](taverna-execution-impl/) Apache Taverna Platform Execution Service impl
* [taverna-execution-local](taverna-execution-local/) Apache Taverna Platform Local Execution Service
-* [taverna-execution-remote](taverna-execution-remote/) Apache Taverna Platform Remote Execution Service
-* [taverna-observer](taverna-observer/) Apache Taverna Observer pattern
-* [taverna-prov](taverna-prov/) Apache Taverna PROV support
-* [taverna-provenanceconnector](taverna-provenanceconnector/) Apache Taverna Provenance Connector
-* [taverna-prov-owl-bindings](taverna-prov-owl-bindings/) Apache Taverna PROV OWL bindings
* [taverna-reference-api](taverna-reference-api/) Apache Taverna Reference Manager API
* [taverna-reference-impl](taverna-reference-impl/) Apache Taverna Reference Manager impl
-* [taverna-reference-testhelpers](taverna-reference-testhelpers/) Apache Taverna Reference Test Helpers
* [taverna-reference-types](taverna-reference-types/) Apache Taverna Engine Reference Types
* [taverna-report-api](taverna-report-api/) Apache Taverna Platform Report Service
* [taverna-run-api](taverna-run-api/) Apache Taverna Platform Run Service API
* [taverna-run-impl](taverna-run-impl/) Apache Taverna Platform Run Service impl
* [taverna-services-api](taverna-services-api/) Apache Taverna Services API
* [taverna-services-impl](taverna-services-impl/) Apache Taverna Platform Services impl
-* [taverna-stringconstant-activity](taverna-stringconstant-activity/) Apache Taverna StringConstant Activity
* [taverna-workflowmodel-api](taverna-workflowmodel-api/) Apache Taverna Workflow Model API
-* [taverna-workflowmodel-extensions](taverna-workflowmodel-extensions/) Apache Taverna Workflow Model Extension Points
* [taverna-workflowmodel-impl](taverna-workflowmodel-impl/) Apache Taverna Workflow Model impl
+* [taverna-workflowmodel-extensions](taverna-workflowmodel-extensions/) Apache Taverna Workflow Model Extension Points
-See the [taverna-engine javadoc](http://taverna.incubator.apache.org/javadoc/taverna-engine/)
-for details. In brief - assuming a workflow run by the
-[Apache Taverna Command Line](https://github.com/apache/incubator-taverna-commandline/):
+These modules include utilities used by the above, or for test purposes:
+
+* [taverna-observer](taverna-observer/) Apache Taverna Observer pattern
+* [taverna-activity-test-utils](taverna-activity-test-utils/) Apache Taverna Activity test utils
+* [taverna-reference-testhelpers](taverna-reference-testhelpers/) Apache Taverna Reference Test Helpers
+
+These modules include structural workflow activities:
+
+* [taverna-dataflow-activity](taverna-dataflow-activity/) Apache Taverna Dataflow Activity
+* [taverna-stringconstant-activity](taverna-stringconstant-activity/) Apache Taverna StringConstant Activity
+
+_See the separate release of
+[taverna-common-activities](taverna.incubator.apache.org/download/common-activities/)
+for activities that invoke Beanshell, WSDL, REST and command line tool services._
+
+
+These modules are experimental:
+
+* [taverna-execution-remote](taverna-execution-remote/) Apache Taverna Platform Remote Execution Service
+* [taverna-execution-hadoop](taverna-execution-hadoop/) Apache Taverna Hadoop Workflow Execution Service
+
+These modules are not yet fully updated to Taverna 3 (contributions welcome!):
+
+* [taverna-activity-archetype](taverna-activity-archetype/) Apache Taverna Activity archetype
+* [taverna-prov](taverna-prov/) Apache Taverna PROV support ()
+* [taverna-provenanceconnector](taverna-provenanceconnector/) Apache Taverna Provenance Connector
+* [taverna-prov-owl-bindings](taverna-prov-owl-bindings/) Apache Taverna PROV OWL bindings
+
+
+See the [taverna-engine JavaDoc](http://taverna.incubator.apache.org/javadoc/taverna-engine/)
+for more details.
+
+## Which module does what?
+
+In brief - assuming a workflow run by the
+[Apache Taverna Command Line](https://github.com/apache/incubator-taverna-commandline/),
+this is how a workflow is executed and which Taverna Engine modules are involved.
+
+There are two layers of Taverna Engine:
+
+* Apache Taverna Platform provides an outer API for creating and managing workflow runs, potentially remotely.
+* Apache Taverna Core (aka `workflowmodel`, formerly `t2.core`) is the orchestrating part of the Taverna Engine. It's [Activity](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/workflowmodel/processor/activity/Activity.html) is implemented by multiple plugins depending on the activity type.
+
+### Preparations
+
+* Load a SCUFL2 [WorkflowBundle](https://taverna.incubator.apache.org/javadoc/taverna-language/org/apache/taverna/scufl2/api/container/WorkflowBundle.html)
+ from a `.wfbundle` or `.t2flow` file
+ using [WorkflowBundleIO](https://taverna.incubator.apache.org/javadoc/taverna-language/org/apache/taverna/scufl2/api/io/WorkflowBundleIO.html)
+ from [Taverna Language](http://taverna.incubator.apache.org/download/language/)
+ + (Optional) choose a [Workflow](https://taverna.incubator.apache.org/javadoc/taverna-language/org/apache/taverna/scufl2/api/container/WorkflowBundle.html#getWorkflows--) to bind
+ workflows (usually the
+ [main workflow](https://taverna.incubator.apache.org/javadoc/taverna-language/org/apache/taverna/scufl2/api/container/WorkflowBundle.html#getMainWorkflow--))
+ + (Optional) choose a [Profile](https://taverna.incubator.apache.org/javadoc/taverna-language/org/apache/taverna/scufl2/api/container/WorkflowBundle.html#getProfiles--) to bind
+ workflow implementations (there's normally only one profile)
+
+* Create or load a [Bundle](https://taverna.incubator.apache.org/javadoc/taverna-language/org/apache/taverna/robundle/Bundle.html)
+ with the expected [inputs](https://taverna.incubator.apache.org/javadoc/taverna-language/org/apache/taverna/scufl2/api/core/Workflow.html#getInputPorts--) of the
+ [main workflow](https://taverna.incubator.apache.org/javadoc/taverna-language/org/apache/taverna/scufl2/api/container/WorkflowBundle.html#getMainWorkflow--) using
+ [DataBundles](https://taverna.incubator.apache.org/javadoc/taverna-language/org/apache/taverna/databundle/DataBundles.html)
+from [Taverna Language](http://taverna.incubator.apache.org/download/language/)
+
+### Running a workflow
+
+* You must be running within OSGi to support multiple Taverna activities
+ and their dependencies, e.g. by using the
+ [OsgiLauncher](https://taverna.incubator.apache.org/javadoc/taverna-osgi/org/apache/taverna/osgilauncher/OsgiLauncher.html) from [Apache Taverna OSGi](http://taverna.incubator.apache.org/download/osgi/).
+ See the [TavernaStarter example](https://github.com/taverna-extras/taverna-starter/blob/master/src/main/java/org/apache/taverna/osgistarter/TavernaStarter.java)
+ for details.
+* (Optional) populate service credentials (username/password)
+ to be used by the workflow into the
+ [CredentialManager](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/security/credentialmanager/CredentialManager.html) from
+ [taverna-credential-manager-api](taverna-credential-manager-api/).
+* Use the [RunService](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/run/api/RunService.html) from the [taverna-run-api](taverna-run-api) (discovered as a [Spring service](#spring-services)).
+* Use [runService.getExecutionEnvironments()](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/run/api/RunService.html#getExecutionEnvironments-org.apache.taverna.scufl2.api.container.WorkflowBundle-) to find an executer for your workflow bundle or profile.
+ + The RunService selects an appropriate
+ [ExecutionService](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/execution/api/ExecutionService.html)
+ service ([taverna-execution-api](taverna-execution-api)), checking the [Profile](https://taverna.incubator.apache.org/javadoc/taverna-language/org/apache/taverna/scufl2/api/container/WorkflowBundle.html#getProfiles--) against the installed [activity types](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/execution/api/ExecutionEnvironment.html#getActivityTypes--) from [common-activities](https://github.com/apache/incubator-taverna-common-activities/) and plugins; discovered by the [taverna-capability-api](taverna-capability-api)
+ + In the Taverna Command Line there is typically only one match, a [LocalExecutionEnvironment](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/execution/impl/local/LocalExecutionEnvironment.html), representing the [LocalExecutionService](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/execution/impl/local/LocalExecutionService.html) from [taverna-execution-local](taverna-execution-local)
+ + The experimental module [taverna-execution-remote](taverna-execution-remote) aims to execute a workflow on a remote [Taverna Server](http://github.com/apache/incubator-taverna-server/) - e.g. for running workflows remotely from Taverna Workbench
+ + The prototype module [taverna-execution-hadoop](taverna-execution-hadoop) is an attempt to execute compatible Taverna workflows as an [Apache Hadoop](http://hadoop.apache.org/) map-reduce workflow. (This code needs to be updated for newer Hadoop releases)
+* [Create an execution job](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/execution/api/ExecutionService.html#createExecution-org.apache.taverna.platform.execution.api.ExecutionEnvironment-org.apache.taverna.scufl2.api.container.WorkflowBundle-org.apache.taverna.scufl2.api.core.Workflow-org.apache.taverna.scufl2.api.profiles.Profile-org.apache.taverna.robundle.Bundle-), and provide the `ExecutionEnvironment`, the `WorkflowBundle`, its `Workflow`, `Profile` and data `Bundle` with inputs. A workflow run id `String` is returned for subsequent calls.
+ + Typically this creates a [LocalExecution](taverna-execution-local/src/main/java/org/apache/taverna/platform/execution/impl/local/LocalExecution.java#L105) instance ([taverna-execution-local](taverna-execution-local)), which converts the
+ abstract SCUFL2 [Workflow](https://taverna.incubator.apache.org/javadoc/taverna-language/org/apache/taverna/scufl2/api/core/Workflow.html) to an executable
+ [Dataflow](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/workflowmodel/Dataflow.html) ([taverna-workflowmodel-api](taverna-workflowmodel-api)) which has corresponding [Processor](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/workflowmodel/Processor.html)s and
+ [Activity](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/workflowmodel/processor/activity/Activity.html) implementations from the installed plugins.
+ + A [WorkflowInstanceFacade](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/facade/WorkflowInstanceFacade.html) is prepared - the engine's top-level representation of a workflow run
+ + A [LocalExecutionManager](https://github.com/apache/incubator-taverna-engine/blob/master/taverna-execution-local/src/main/java/org/apache/taverna/platform/execution/impl/local/LocalExecutionMonitor.java) is added as a bridge between the Taverna Core engine and the Platform
+
+* Start the run using [runService.start()](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/run/api/RunService.html#start-java.lang.String-)
+ * Input values from the data [Bundle](https://taverna.incubator.apache.org/javadoc/taverna-language/org/apache/taverna/robundle/Bundle.html) is
+ [converted](https://github.com/apache/incubator-taverna-engine/blob/master/taverna-execution-local/src/main/java/org/apache/taverna/platform/execution/impl/local/T2ReferenceConverter.java#L34) to be registered with a
+[ReferenceService](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/reference/ReferenceService.html) ([taverna-reference-api](taverna-reference-api)) - as Taverna workflows pass [T2Reference](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/reference/T2Reference.html)s.
+ + Values are represented as either a [InlineStringReference](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/reference/impl/external/object/InlineStringReference.html) or [InlineByteArrayReference](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/reference/impl/external/object/InlineByteArrayReference.html), or could be external [HttpReference](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/reference/impl/external/http/HttpReference.html)s or [FileReference](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/reference/impl/external/file/FileReference.html)s ([taverna-reference-types](taverna-reference-types)). These types are recognized by the `Activity` implementations, allowing pass-by-reference
+ where supported, with conversion to strings and bytes if needed.
+ + Lists, values and references are stored [in memory](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/reference/impl/InMemoryReferenceSetDao.html).
+ + This was previously backed by [Hibernate](https://github.com/apache/incubator-taverna-engine/blob/old/core-pre-incubator-20141219/reference-impl/src/main/java/net/sf/taverna/t2/reference/impl/HibernateReferenceSetDao.java) stored in [Apache Derby](http://db.apache.org/derby/)
+
+
+* Monitor the [state of the run](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/run/api/RunService.html#getState-java.lang.String-) or
+[get](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/run/api/RunService.html#getWorkflowReport-java.lang.String-) the [WorkflowReport](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/report/WorkflowReport.html) for detailed progress ([taverna-report-api](taverna-report-api)), such as
+the [number of jobs completed](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/report/ProcessorReport.html#getJobsCompleted--) for a given [Processor](https://taverna.incubator.apache.org/javadoc/taverna-language/org/apache/taverna/scufl2/api/core/Processor.html)
+* Once the workflow is [COMPLETED](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/report/State.html#COMPLETED) (and even while it is [RUNNING](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/report/State.html#RUNNING)) you can
+retrieve the [bundle](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/run/api/RunService.html#getDataBundle-java.lang.String-) that contains the output values and intermediate values.
+ + To relate intermediate values, you will need to inspect either the
+ [WorkflowReport](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/report/WorkflowReport.html),
+ [ProcessorReport](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/report/ProcessorReport.html) or [ActivityReport](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/report/ActivityReport.html)s
+ to [get the invocations](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/report/StatusReport.html#getInvocations--), which provide the [inputs](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/report/Invocation.html#getInputs--)
+ and [outputs](https://taverna.incubator.apache.org/javadoc/taverna-engine/org/apache/taverna/platform/report/Invocation.html#getOutputs--) as
+ [Path](http://docs.oracle.com/javase/8/docs/api/java/nio/file/Path.html)s within the bundle.
-*
## Spring services
@@ -175,21 +269,25 @@ your `META-INF/spring/update-context-osgi.xml`:
```xml
- <beans:beans xmlns="http://www.springframework.org/schema/osgi" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xmlns:beans="http://www.springframework.org/schema/beans"
- xsi:schemaLocation="http://www.springframework.org/schema/beans
- http://www.springframework.org/schema/beans/spring-beans.xsd
- http://www.springframework.org/schema/osgi
- http://www.springframework.org/schema/osgi/spring-osgi.xsd">
- <reference id="applicationConfiguration" interface="org.apache.taverna.configuration.app.ApplicationConfiguration" />
- <reference id="proxyConfiguration" interface="org.apache.taverna.configuration.proxy.HttpProxyConfiguration" />
- <reference id="configurationManager" interface="org.apache.taverna.configuration.ConfigurationManager" />
- <reference id="downloadManager" interface="org.apache.taverna.download.DownloadManager" />
- <reference id="pluginManager" interface="org.apache.taverna.plugin.PluginManager" />
- <reference id="updateManager" interface="org.apache.taverna.update.UpdatenManager" />
- </beans:beans>
+ <beans:beans xmlns="http://www.springframework.org/schema/osgi" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xmlns:beans="http://www.springframework.org/schema/beans"
+ xsi:schemaLocation="
+ http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
+ http://www.springframework.org/schema/osgi http://www.springframework.org/schema/osgi/spring-osgi.xsd">
+
+ <reference id="executionService" interface="org.apache.taverna.platform.execution.api.ExecutionService"/>
+ <reference id="runService" interface="org.apache.taverna.platform.run.api.RunService"/>
+ <reference id="credentialManager" interface="org.apache.taverna.security.credentialmanager.CredentialManager" />
+ <reference id="databaseConfiguration" interface="org.apache.taverna.configuration.database.DatabaseConfiguration" />
+ <reference id="databaseManager" interface="org.apache.taverna.configuration.database.DatabaseManager" />
+
+ </beans:beans>
```
+You might want to combine these with the
+[Taverna OSGi services](https://github.com/apache/incubator-taverna-osgi/#spring-services).
+
+
# Export restrictions
This distribution includes cryptographic software.