You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@oozie.apache.org by an...@apache.org on 2018/09/14 14:07:07 UTC
[01/11] oozie git commit: OOZIE-2734 [docs] Switch from TWiki to
Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Repository: oozie
Updated Branches:
refs/heads/master d5c4f3b7b -> 4e5b3cb59
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/index.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/index.twiki b/docs/src/site/twiki/index.twiki
deleted file mode 100644
index 3003fa9..0000000
--- a/docs/src/site/twiki/index.twiki
+++ /dev/null
@@ -1,111 +0,0 @@
-<noautolink>
-
----+!! Oozie, Workflow Engine for Apache Hadoop
-
-Oozie v3 is a server based _Bundle Engine_ that provides a higher-level oozie abstraction that will batch a set of coordinator applications. The user will be able to start/stop/suspend/resume/rerun a set coordinator jobs in the bundle level resulting a better and easy operational control.
-
-Oozie v2 is a server based _Coordinator Engine_ specialized in running workflows based on time and data triggers.
-It can continuously run workflows based on time (e.g. run it every hour), and data availability
-(e.g. wait for my input data to exist before running my workflow).
-
-Oozie v1 is a server based _Workflow Engine_ specialized in running workflow jobs with actions that
-execute Hadoop Map/Reduce and Pig jobs.
-
-%TOC%
-
----++ Distribution Contents
-
-Oozie distribution consists of a single 'tar.gz' file containing:
-
- * Readme, license, notice & [[./release-log.txt][Release log]] files.
- * Oozie server: =oozie-server= directory.
- * Scripts: =bin/= directory, client and server scripts.
- * Binaries: =lib/= directory, client JAR files.
- * Configuration: =conf/= server configuration directory.
- * Archives:
- * =oozie-client-*.tar.gz=: Client tools.
- * =oozie.war=: Oozie WAR file.
- * =docs.zip=: Documentation.
- * =oozie-examples-*.tar.gz=: Examples.
- * =oozie-sharelib-*.tar.gz=: Share libraries (with Streaming, Pig JARs).
-
----++ Quick Start
-
-Enough reading already? Follow the steps in [[DG_QuickStart][Oozie Quick Start]] to get Oozie up and running.
-
----++ Developer Documentation
-
- * [[DG_Overview][Overview]]
- * [[DG_QuickStart][Oozie Quick Start]]
- * [[DG_Examples][Running the Examples]]
- * [[WorkflowFunctionalSpec][Workflow Functional Specification]]
- * [[CoordinatorFunctionalSpec][Coordinator Functional Specification]]
- * [[BundleFunctionalSpec][Bundle Functional Specification]]
- * [[https://docs.oracle.com/javaee/7/tutorial/jsf-el.htm][EL Expression Language Quick Reference]]
- * [[DG_CommandLineTool][Command Line Tool]]
- * [[DG_WorkflowReRun][Workflow Re-runs Explained]]
- * [[DG_HCatalogIntegration][HCatalog Integration Explained]]
-
- * [[./client/apidocs/index.html][Oozie Client Javadocs]]
- * [[./core/apidocs/index.html][Oozie Core Javadocs]]
- * [[WebServicesAPI][Oozie Web Services API]]
- * [[DG_ActionAuthentication][Action Authentication]]
-
----+++ Action Extensions
-
- * [[DG_EmailActionExtension][Email Action]]
- * [[DG_ShellActionExtension][Shell Action]]
- * [[DG_HiveActionExtension][Hive Action]]
- * [[DG_Hive2ActionExtension][Hive 2 Action]]
- * [[DG_SqoopActionExtension][Sqoop Action]]
- * [[DG_SshActionExtension][Ssh Action]]
- * [[DG_DistCpActionExtension][DistCp Action]]
- * [[DG_SparkActionExtension][Spark Action]]
- * [[DG_CustomActionExecutor][Writing a Custom Action Executor]]
-
----+++ Job Status and SLA Monitoring
-
- * [[DG_JMSNotifications][JMS Notifications for Job and SLA]]
- * [[DG_SLAMonitoring][Configuring and Monitoring SLA]]
-
----++ Administrator Documentation
-
- * [[AG_Install][Oozie Install]]
- * [[AG_OozieLogging][Oozie Logging]]
- * [[AG_HadoopConfiguration][Hadoop Configuration]]
- * [[AG_ActionConfiguration][Action Configuration]]
- * [[AG_Monitoring][Oozie Monitoring]]
- * [[DG_CommandLineTool][Command Line Tool]]
- * [[AG_OozieUpgrade][Oozie Upgrade]]
-
-#LicenseInfo
----++ Licensing Information
-
-Oozie is distributed under [[http://www.apache.org/licenses/LICENSE-2.0][Apache License 2.0]].
-
-For details on the license of the dependent components, refer to the
-[[./dependencies.html#Licenses][Dependencies Report, Licenses section]].
-
-Oozie bundles an embedded Jetty 9.x.
-
-Some of the components in the dependencies report don't mention their license in the published POM. They are:
-
- * JDOM: [[http://www.jdom.org/docs/faq.html#a0030][JDOM License]] (Apache style).
- * Oro: [[http://www.apache.org/licenses/LICENSE-2.0][Apache License 2.0]].
-
-Oozie uses a modified version of the Apache Doxia core and twiki plugins to generate Oozie documentation.
-
----++ Engineering Documentation
-
- * [[ENG_Building][Building Oozie]]
- * [[./dependencies.html][Dependencies Report]]
-
----++ MiniOozie Documentation
-
- * [[ENG_MiniOozie][Testing User Oozie Applications Using MiniOozie]]
-
----++ Oozie User Authentication Documentation
-
- * [[ENG_Custom_Authentication][Create Custom Oozie Authentication]]
-
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 423d19d..d6440e4 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1729,9 +1729,34 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-site-plugin</artifactId>
- <!-- Don't change version as doc generation may fail -->
- <!-- (using custom doxia for twiki pages generation) -->
- <version>2.0-beta-6</version>
+ <version>3.7</version>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.maven.doxia</groupId>
+ <artifactId>doxia-site-renderer</artifactId>
+ <version>1.8.1</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.maven.doxia</groupId>
+ <artifactId>doxia-sink-api</artifactId>
+ <version>1.8</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.velocity</groupId>
+ <artifactId>velocity</artifactId>
+ <version>1.5</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.maven.doxia</groupId>
+ <artifactId>doxia-module-markdown</artifactId>
+ <version>1.8</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.maven.doxia</groupId>
+ <artifactId>doxia-core</artifactId>
+ <version>1.7</version>
+ </dependency>
+ </dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
@@ -2011,7 +2036,7 @@
<exclude>work.log</exclude>
<exclude>SecurityAuth.audit</exclude>
<exclude>.gitignore</exclude>
- <exclude>docs/src/site/twiki/*.twiki</exclude>
+ <exclude>docs/src/site/markdown/*.md</exclude>
<exclude>examples/src/main/data/*</exclude>
<exclude>core/src/test/resources/test-ioutils.txt</exclude>
<exclude>core/src/test/resources/oldWorkFlowApp.serialized</exclude>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/release-log.txt
----------------------------------------------------------------------
diff --git a/release-log.txt b/release-log.txt
index b835951..bf6c1c7 100644
--- a/release-log.txt
+++ b/release-log.txt
@@ -1,5 +1,6 @@
-- Oozie 5.1.0 release (trunk - unreleased)
+OOZIE-2734 [docs] Switch from TWiki to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
OOZIE-3298 [MapReduce action] External ID is not filled properly and failing MR job is treated as SUCCEEDED (andras.piros via pbacsko, asasvari, gezapeti)
OOZIE-3317 amend [build] Fix false positive precommit reports (kmarton via andras.piros)
OOZIE-3160 PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting (pbacsko)
[05/11] oozie git commit: OOZIE-2734 [docs] Switch from TWiki to
Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_CoordinatorRerun.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_CoordinatorRerun.twiki b/docs/src/site/twiki/DG_CoordinatorRerun.twiki
index fbb1376..f535d16 100644
--- a/docs/src/site/twiki/DG_CoordinatorRerun.twiki
+++ b/docs/src/site/twiki/DG_CoordinatorRerun.twiki
@@ -1,12 +1,12 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
----+!! Coordinator Rerun
+[::Go back to Oozie Documentation Index::](index.html)
-%TOC%
+# Coordinator Rerun
----++ Pre-Conditions
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Pre-Conditions
* Rerun coordinator action must be in TIMEDOUT/SUCCEEDED/KILLED/FAILED.
* Coordinator actions cannot be rerun if the coordinator job is in the PREP or IGNORED state.
@@ -16,14 +16,15 @@
* Coordinator Rerun will only use the original configs from first run.
* Coordinator Rerun will not re-read the coordinator.xml in hdfs.
----++ Rerun Arguments
+## Rerun Arguments
+
- <verbatim>
+```
$oozie job -rerun <coord_Job_id> [-nocleanup] [-refresh] [-failed] [-config <arg>]
[-action 1, 3-4, 7-40] (-action or -date is required to rerun.)
[-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z]
(if neither -action nor -date is given, the exception will be thrown.)
-</verbatim>
+```
* Either -action or -date should be given.
* If -action and -date both are given, an error will be thrown.
@@ -37,7 +38,7 @@ $oozie job -rerun <coord_Job_id> [-nocleanup] [-refresh] [-failed] [-config <arg
* If -failed is set, re-runs the failed workflow actions of the coordinator actions.
* -config can be used to supply properties to workflow by job configuration file '.xml' or '.properties'.
----++ Rerun coordinator actions
+## Rerun coordinator actions
* Rerun terminated (timeout, succeeded, killed, failed) coordinator actions.
* By default, Oozie will delete the 'output-event' directories before changing actions' status and materializing actions.
@@ -47,6 +48,6 @@ $oozie job -rerun <coord_Job_id> [-nocleanup] [-refresh] [-failed] [-config <arg
within that range. If the existing actions are action #5....#40, which map to Jan 15 to Feb 15, then only those actions will run.
* The rerun action_id and nominal_time of the actions which are eligible to rerun will be returned.
-[[index][::Go back to Oozie Documentation Index::]]
+[::Go back to Oozie Documentation Index::](index.html)
+
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_CustomActionExecutor.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_CustomActionExecutor.twiki b/docs/src/site/twiki/DG_CustomActionExecutor.twiki
index 7831484..5768b27 100644
--- a/docs/src/site/twiki/DG_CustomActionExecutor.twiki
+++ b/docs/src/site/twiki/DG_CustomActionExecutor.twiki
@@ -1,78 +1,83 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
----+!! Custom Action Nodes
+[::Go back to Oozie Documentation Index::](index.html)
-%TOC%
+# Custom Action Nodes
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Introduction
+Oozie can be extended to support additional action types by writing a custom [Action Node](WorkflowFunctionalSpec.html#ActionNodes). Action Nodes can be synchronous or asynchronous.
----++ Introduction
-Oozie can be extended to support additional action types by writing a custom [[WorkflowFunctionalSpec#ActionNodes][Action Node]]. Action Nodes can be synchronous or asynchronous.
* Synchronous Node - Sync nodes are executed inline by Oozie, which waits for completion of these nodes before proceeding. Hence, these nodes should almost never be used and are meant for lightweight tasks like FileSystem move, mkdir, delete.
* Asynchronous Nodes - Oozie starts asynchronous nodes, and then monitors the action being executed for completion. This is done via a callback from the action or Oozie polling for the action status.
----++ Writing a custom Action Node
-Action Executors are configured in the oozie configuration file oozie-site.xml. These executors are loaded during Oozie startup. [[DG_CustomActionExecutor#Deploying_a_custom_Action_Executor][Deploying a Custom Action Executor]].
+## Writing a custom Action Node
+Action Executors are configured in the oozie configuration file oozie-site.xml. These executors are loaded during Oozie startup. [Deploying a Custom Action Executor](DG_CustomActionExecutor.html#Deploying_a_custom_Action_Executor).
-Action Executors MUST extend the =ActionExecutor= class and override the required methods.
+Action Executors MUST extend the `ActionExecutor` class and override the required methods.
Most methods take as argument the Execution Context and the actual Action object with various configuration properties resolved.
----+++ ActionExecutor.Context
+### ActionExecutor.Context
The Execution context gives Action Nodes access to configuration properties, methods to set the state of the action, methods to set variables which are to be made available later in the execution path.
-*The following methods from the ActionExecutor interface should be implemented.*
----+++ Constructor
+**The following methods from the ActionExecutor interface should be implemented.**
+### Constructor
A no argument constructor should be implemented, which calls super(ACTION_TYPE). ACTION_TYPE is the name of the action which will be used in the workflow xml, and is used by Oozie to instantiate the correct type of Executor.
----+++ initActionType()
+### initActionType()
This method is called once during initialization of the Action Executor during Oozie startup. Any common initialization code for the Action Node should go here.
As an example, setting up of error handling for the Custom Action should be done here.
This method must call super.initActionType() as it's first statement.
----+++ start(ActionExecutor.Context context, Action action)
+### start(ActionExecutor.Context context, Action action)
The action start up happens here.
+
* Async Actions - The action should be started and context.setStartData(externalId, trackerUri, consoleUrl) must be set. A check can be made for whether the action has completed, in which case context.setExecutionData(externalStatus, actionData) must be called.
* Sync Actions - The action should be started and should complete execution. context.setExecutionData(externalStatus, actionData) must be called.
----+++ check(ActionExecutor.Context context, Action action)
-<code>check(...)</code> is used by Oozie to poll for the status of the action. This method should interact with the action started previously, and update the status. If the action has completed, context.setExecutionData(externalStatus, actionData) must be called. Otherwise, the status can be updated using context.setExternalStatus(externalStatus).
+### check(ActionExecutor.Context context, Action action)
+`check(...)` is used by Oozie to poll for the status of the action. This method should interact with the action started previously, and update the status. If the action has completed, context.setExecutionData(externalStatus, actionData) must be called. Otherwise, the status can be updated using context.setExternalStatus(externalStatus).
For sync actions, this method will not be called, and should throw an UnsupportedOperationException().
----+++ kill(ActionExecutor.Context context, Action action)
-<code>kill(...)</code> is called when there is an attempt to kill the running job or action. No workflow transition is made after this.
+### kill(ActionExecutor.Context context, Action action)
+`kill(...)` is called when there is an attempt to kill the running job or action. No workflow transition is made after this.
The implementation for a custom action should interact with and kill the running action, and take care of any cleanup which may be required. context.setEndData(status, signalValue) should be called with both values set to Action.Status.KILLED.
----+++ end(ActionExecutor.Context context, Action action)
-<code>end(...)</end> is used for any cleanup or processing which may need to be done after completion of the action. After any processing, context.setEndData(status, signalValue) should be called to complete execution of the action and trigger the next workflow transition. signalValue can be Action.Status.OK or Action.Status.ERROR.
----+++ Registering Errors
+### end(ActionExecutor.Context context, Action action)
+`end(...)` is used for any cleanup or processing which may need to be done after completion of the action. After any processing, context.setEndData(status, signalValue) should be called to complete execution of the action and trigger the next workflow transition. signalValue can be Action.Status.OK or Action.Status.ERROR.
+### Registering Errors
Oozie actions can generate different types of Errors.
+
* TRANSIENT - will be retried
* NON TRANSIENT - the job will be suspended and can be resumed later by human intervention, after fixing whatever problem caused this error.
* ERROR - causes the error transition to be taken.
* FAILED - the action and the job are set to FAILED state. No transitions are taken.
registerError(exceptionClassName, errorType, errorMessage) can be used to register possible exceptions while executing the action, along with their type and error message. This will normally be done during initialization of the Action Executor.
----++ Deploying a custom Action Executor
+## Deploying a custom Action Executor
Action Nodes can be registered in the oozie configuration file oozie-site.xml, by changing the property 'oozie.service.ActionService.executor.ext.classes'. For multiple Executors, the class name should be separated by commas.
-<verbatim> <property>
+
+``` <property>
<name>oozie.service.ActionService.executor.ext.classes</name>
<value>
org.apache.oozie.wf.action.decision.CustomActionExecutor,
Custom_Action_Executr_2.class
</value>
- </property></verbatim>
+ </property>
+```
Any configuration properties to be made available to this class should also be added to oozie-site.xml. The convention to be followed for naming these properties is 'oozie.action.[ActionName].property.name'
The XML schema (XSD) for the new Actions should be added to oozie-site.xml, under the property 'oozie.service.WorkflowSchemaService.ext.schemas'. A comma separated list for multiple Action schemas.
The XML schema (XSD) for the new action should be also added to Fluent Job API. Please refer to
-[[DG_FluentJobAPI#AE.C_Appendix_C_How_To_Extend][Fluent Job API :: How To Extend]] for details.
+[Fluent Job API :: How To Extend](DG_FluentJobAPI.html#AE.C_Appendix_C_How_To_Extend) for details.
The executor class should be placed along with the oozie webapp in the correct path. Once Oozie is restarted, the custom action node can be used in workflows.
-[[index][::Go back to Oozie Documentation Index::]]
+[::Go back to Oozie Documentation Index::](index.html)
+
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_DistCpActionExtension.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_DistCpActionExtension.twiki b/docs/src/site/twiki/DG_DistCpActionExtension.twiki
index 8bab3da..13c2a0a 100644
--- a/docs/src/site/twiki/DG_DistCpActionExtension.twiki
+++ b/docs/src/site/twiki/DG_DistCpActionExtension.twiki
@@ -1,27 +1,28 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
+
+[::Go back to Oozie Documentation Index::](index.html)
-----
----+!! Oozie DistCp Action Extension
+# Oozie DistCp Action Extension
-%TOC%
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
----++ DistCp Action
+## DistCp Action
-The =DistCp= action uses Hadoop distributed copy to copy files from one cluster to another or within the same cluster.
+The `DistCp` action uses Hadoop distributed copy to copy files from one cluster to another or within the same cluster.
-*IMPORTANT:* The DistCp action may not work properly with all configurations (secure, insecure) in all versions
+**IMPORTANT:** The DistCp action may not work properly with all configurations (secure, insecure) in all versions
of Hadoop. For example, distcp between two secure clusters is tested and works well. Same is true with two insecure
clusters. In cases where a secure and insecure clusters are involved, distcp will not work.
Both Hadoop clusters have to be configured with proxyuser for the Oozie process as explained
-[[DG_QuickStart#HadoopProxyUser][here]] on the Quick Start page.
+[here](DG_QuickStart.html#HadoopProxyUser) on the Quick Start page.
+
+**Syntax:**
-*Syntax:*
-<verbatim>
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<action name="distcp-example">
@@ -36,29 +37,31 @@ Both Hadoop clusters have to be configured with proxyuser for the Oozie process
</action>
...
</workflow-app>
-</verbatim>
+```
-The first =arg= indicates the input and the second =arg= indicates the output. In the above example, the input is on =namenode1=
-and the output is on =namenode2=.
+The first `arg` indicates the input and the second `arg` indicates the output. In the above example, the input is on `namenode1`
+and the output is on `namenode2`.
-*IMPORTANT:* If using the DistCp action between 2 secure clusters, the following property must be added to the =configuration= of
+**IMPORTANT:** If using the DistCp action between 2 secure clusters, the following property must be added to the `configuration` of
the action:
-<verbatim>
+
+```
<property>
<name>oozie.launcher.mapreduce.job.hdfs-servers</name>
<value>${nameNode1},${nameNode2}</value>
</property>
-</verbatim>
+```
-The =DistCp= action is also commonly used to copy files within the same cluster. Cases where copying files within
+The `DistCp` action is also commonly used to copy files within the same cluster. Cases where copying files within
a directory to another directory or directories to target directory is supported. Example below will illustrate a
-copy within a cluster, notice the source and target =nameNode= is the same and use of =*= syntax is supported to
-represent only child files or directories within a source directory. For the sake of the example, =jobTracker= and =resourceManager=
+copy within a cluster, notice the source and target `nameNode` is the same and use of `*` syntax is supported to
+represent only child files or directories within a source directory. For the sake of the example, `jobTracker` and `resourceManager`
are synonymous.
-*Syntax:*
+**Syntax:**
+
-<verbatim>
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<action name="copy-example">
@@ -73,14 +76,15 @@ are synonymous.
</action>
...
</workflow-app>
-</verbatim>
+```
----++ Appendix, DistCp XML-Schema
+## Appendix, DistCp XML-Schema
----+++ AE.A Appendix A, DistCp XML-Schema
+### AE.A Appendix A, DistCp XML-Schema
----++++ DistCp Action Schema Version 1.0
-<verbatim>
+#### DistCp Action Schema Version 1.0
+
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:distcp="uri:oozie:distcp-action:1.0" elementFormDefault="qualified"
targetNamespace="uri:oozie:distcp-action:1.0">
@@ -105,10 +109,11 @@ are synonymous.
</xs:complexType>
.
</xs:schema>
-</verbatim>
+```
+
+#### DistCp Action Schema Version 0.2
----++++ DistCp Action Schema Version 0.2
-<verbatim>
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:distcp="uri:oozie:distcp-action:0.2" elementFormDefault="qualified"
targetNamespace="uri:oozie:distcp-action:0.2">
@@ -156,10 +161,11 @@ are synonymous.
</xs:complexType>
.
</xs:schema>
-</verbatim>
+```
----++++ DistCp Action Schema Version 0.1
-<verbatim>
+#### DistCp Action Schema Version 0.1
+
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:distcp="uri:oozie:distcp-action:0.1" elementFormDefault="qualified"
targetNamespace="uri:oozie:distcp-action:0.1">
@@ -207,8 +213,8 @@ are synonymous.
</xs:complexType>
.
</xs:schema>
-</verbatim>
+```
+
+[::Go back to Oozie Documentation Index::](index.html)
-[[index][::Go back to Oozie Documentation Index::]]
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_EmailActionExtension.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_EmailActionExtension.twiki b/docs/src/site/twiki/DG_EmailActionExtension.twiki
index 4de290c..1afcbb4 100644
--- a/docs/src/site/twiki/DG_EmailActionExtension.twiki
+++ b/docs/src/site/twiki/DG_EmailActionExtension.twiki
@@ -1,28 +1,29 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
+
+[::Go back to Oozie Documentation Index::](index.html)
-----
----+!! Oozie Email Action Extension
+# Oozie Email Action Extension
-%TOC%
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-#EmailAction
----++++ 3.2.4 Email action
+<a name="EmailAction"></a>
+## 3.2.4 Email action
-The =email= action allows sending emails in Oozie from a workflow application. An email action must provide =to=
-addresses, =cc= addresses (optional), =bcc= addresses (optional), a =subject= and a =body=.
+The `email` action allows sending emails in Oozie from a workflow application. An email action must provide `to`
+addresses, `cc` addresses (optional), `bcc` addresses (optional), a `subject` and a `body`.
Multiple recipients of an email can be provided as comma separated addresses.
The email action is executed synchronously, and the workflow job will wait until the specified
emails are sent before continuing to the next action.
-All values specified in the =email= action can be parameterized (templatized) using EL expressions.
+All values specified in the `email` action can be parameterized (templatized) using EL expressions.
+
+**Syntax:**
-*Syntax:*
-<verbatim>
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.1">
...
<action name="[NODE-NAME]">
@@ -40,37 +41,39 @@ All values specified in the =email= action can be parameterized (templatized) us
</action>
...
</workflow-app>
-</verbatim>
+```
-The =to= and =cc= and =bcc= commands are used to specify recipients who should get the mail. Multiple email recipients
-can be provided using comma-separated values. Providing a =to= command is necessary, while the =cc= or =bcc= may
+The `to` and `cc` and `bcc` commands are used to specify recipients who should get the mail. Multiple email recipients
+can be provided using comma-separated values. Providing a `to` command is necessary, while the `cc` or `bcc` may
optionally be used along.
-The =subject= and =body= commands are used to specify subject and body of the mail.
+The `subject` and `body` commands are used to specify subject and body of the mail.
From uri:oozie:email-action:0.2 one can also specify mail content type as <content_type>text/html</content_type>.
"text/plain" is default.
-The =attachment= is used to attach a file(s) on HDFS to the mail. Multiple attachment can be provided using comma-separated values.
+The `attachment` is used to attach a file(s) on HDFS to the mail. Multiple attachment can be provided using comma-separated values.
Non fully qualified path is considered as a file on default HDFS. A local file cannot be attached.
-*Configuration*
+**Configuration**
-The =email= action requires some SMTP server configuration to be present (in oozie-site.xml). The following are the values
+The `email` action requires some SMTP server configuration to be present (in oozie-site.xml). The following are the values
it looks for:
- * =oozie.email.smtp.host= - The host where the email action may find the SMTP server (localhost by default).
- * =oozie.email.smtp.port= - The port to connect to for the SMTP server (25 by default).
- * =oozie.email.from.address= - The from address to be used for mailing all emails (oozie@localhost by default).
- * =oozie.email.smtp.auth= - Boolean property that toggles if authentication is to be done or not. (false by default).
- * =oozie.email.smtp.starttls.enable= - Boolean property that toggles if use TLS communication or not. (false by default).
- * =oozie.email.smtp.username= - If authentication is enabled, the username to login as (empty by default).
- * =oozie.email.smtp.password= - If authentication is enabled, the username's password (empty by default).
- * =oozie.email.attachment.enabled= - Boolean property that toggles if configured attachments are to be placed into the emails.
+
+ * `oozie.email.smtp.host` - The host where the email action may find the SMTP server (localhost by default).
+ * `oozie.email.smtp.port` - The port to connect to for the SMTP server (25 by default).
+ * `oozie.email.from.address` - The from address to be used for mailing all emails (oozie@localhost by default).
+ * `oozie.email.smtp.auth` - Boolean property that toggles if authentication is to be done or not. (false by default).
+ * `oozie.email.smtp.starttls.enable` - Boolean property that toggles if use TLS communication or not. (false by default).
+ * `oozie.email.smtp.username` - If authentication is enabled, the username to login as (empty by default).
+ * `oozie.email.smtp.password` - If authentication is enabled, the username's password (empty by default).
+ * `oozie.email.attachment.enabled` - Boolean property that toggles if configured attachments are to be placed into the emails.
(false by default).
- * =oozie.email.smtp.socket.timeout.ms= - The timeout to apply over all SMTP server socket operations (10000ms by default).
+ * `oozie.email.smtp.socket.timeout.ms` - The timeout to apply over all SMTP server socket operations (10000ms by default).
+
+**Example:**
-*Example:*
-<verbatim>
+```
<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1">
...
<action name="an-email">
@@ -86,14 +89,15 @@ it looks for:
</action>
...
</workflow-app>
-</verbatim>
+```
In the above example, an email is sent to 'bob', 'the.other.bob', 'will' (cc), yet.another.bob (bcc)
with the subject and body both containing the workflow ID after substitution.
----+++ AE.A Appendix A, Email XML-Schema
+## AE.A Appendix A, Email XML-Schema
-<verbatim>
+
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:email="uri:oozie:email-action:0.2" elementFormDefault="qualified"
targetNamespace="uri:oozie:email-action:0.2">
@@ -112,11 +116,12 @@ with the subject and body both containing the workflow ID after substitution.
</xs:sequence>
</xs:complexType>
</xs:schema>
-</verbatim>
+```
+
+**GMail example to oozie-site.xml**
-*GMail example to oozie-site.xml*
-<verbatim>
+```
oozie.email.smtp.host=smtp.gmail.com
oozie.email.smtp.port=587
oozie.email.from.address=<some email address>
@@ -124,8 +129,8 @@ oozie.email.smtp.auth=true
oozie.email.smtp.starttls.enable=true
oozie.email.smtp.username=<Gmail Id>
oozie.email.smtp.password=<Gmail Pass>
-</verbatim>
+```
+
+[::Go back to Oozie Documentation Index::](index.html)
-[[index][::Go back to Oozie Documentation Index::]]
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_Examples.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_Examples.twiki b/docs/src/site/twiki/DG_Examples.twiki
index 5323a17..ff33506 100644
--- a/docs/src/site/twiki/DG_Examples.twiki
+++ b/docs/src/site/twiki/DG_Examples.twiki
@@ -1,58 +1,61 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
----+!! Oozie Examples
+[::Go back to Oozie Documentation Index::](index.html)
-%TOC%
+# Oozie Examples
----++ Command Line Examples
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
----+++ Setting Up the Examples
+## Command Line Examples
-Oozie examples are bundled within the Oozie distribution in the =oozie-examples.tar.gz= file.
+### Setting Up the Examples
-Expanding this file will create an =examples/= directory in the local file system.
+Oozie examples are bundled within the Oozie distribution in the `oozie-examples.tar.gz` file.
-The =examples/= directory must be copied to the user HOME directory in HDFS:
+Expanding this file will create an `examples/` directory in the local file system.
-<verbatim>
+The `examples/` directory must be copied to the user HOME directory in HDFS:
+
+
+```
$ hadoop fs -put examples examples
-</verbatim>
+```
-*NOTE:* If an examples directory already exists in HDFS, it must be deleted before copying it again. Otherwise files may not be
+**NOTE:** If an examples directory already exists in HDFS, it must be deleted before copying it again. Otherwise files may not be
copied.
----+++ Running the Examples
+### Running the Examples
-For the Streaming and Pig example, the [[DG_QuickStart#OozieShareLib][Oozie Share Library]] must be installed in HDFS.
+For the Streaming and Pig example, the [Oozie Share Library](DG_QuickStart.html#OozieShareLib) must be installed in HDFS.
-Add Oozie =bin/= to the environment PATH.
+Add Oozie `bin/` to the environment PATH.
-The examples assume the ResourceManager is =localhost:8032= and the NameNode is =hdfs://localhost:8020=. If the actual
+The examples assume the ResourceManager is `localhost:8032` and the NameNode is `hdfs://localhost:8020`. If the actual
values are different, the job properties files in the examples directory must be edited to the correct values.
The example applications are under the examples/app directory, one directory per example. The directory contains the
-application XML file (workflow, or workflow and coordinator), the =job.properties= file to submit the job and any JAR
+application XML file (workflow, or workflow and coordinator), the `job.properties` file to submit the job and any JAR
files the example may need.
-The inputs for all examples are in the =examples/input-data/= directory.
+The inputs for all examples are in the `examples/input-data/` directory.
-The examples create output under the =examples/output-data/${EXAMPLE_NAME}= directory.
+The examples create output under the `examples/output-data/${EXAMPLE_NAME}` directory.
-*Note*: The =job.properties= file needs to be a local file during submissions, and not a HDFS path.
+**Note**: The `job.properties` file needs to be a local file during submissions, and not a HDFS path.
-*How to run an example application:*
+**How to run an example application:**
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run
.
job: 14-20090525161321-oozie-tucu
-</verbatim>
+```
Check the workflow job status:
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -info 14-20090525161321-oozie-tucu
.
.----------------------------------------------------------------------------------------------------------------------------------------------------------------
@@ -71,28 +74,30 @@ Action Name Type Status Transition External Id
.----------------------------------------------------------------------------------------------------------------------------------------------------------------
mr-node map-reduce OK end job_200904281535_0254 SUCCEEDED - 2009-05-26 05:01 +0000 2009-05-26 05:01 +0000
.----------------------------------------------------------------------------------------------------------------------------------------------------------------
-</verbatim>
+```
-To check the workflow job status via the Oozie web console, with a browser go to =http://localhost:11000/oozie=.
+To check the workflow job status via the Oozie web console, with a browser go to `http://localhost:11000/oozie`.
-To avoid having to provide the =-oozie= option with the Oozie URL with every =oozie= command, set =OOZIE_URL= env
+To avoid having to provide the `-oozie` option with the Oozie URL with every `oozie` command, set `OOZIE_URL` env
variable to the Oozie URL in the shell environment. For example:
-<verbatim>
+
+```
$ export OOZIE_URL="http://localhost:11000/oozie"
$
$ oozie job -info 14-20090525161321-oozie-tucu
-</verbatim>
+```
----++ Java API Example
+## Java API Example
-Oozie provides a =[[./apidocs/org/org/apache/oozie/client/package-summary.html][Java Client API]] that simplifies
+Oozie provides a [Java Client API](./apidocs/org/org/apache/oozie/client/package-summary.html) that simplifies
integrating Oozie with Java applications. This Java Client API is a convenience API to interact with Oozie Web-Services
API.
The following code snippet shows how to submit an Oozie job using the Java Client API.
-<verbatim>
+
+```
import org.apache.oozie.client.OozieClient;
import org.apache.oozie.client.WorkflowJob;
.
@@ -127,20 +132,21 @@ import java.util.Properties;
System.out.println("Workflow job completed ...");
System.out.println(wf.getJobInfo(jobId));
...
-</verbatim>
+```
----++ Local Oozie Example
+## Local Oozie Example
-Oozie provides an embedded Oozie implementation, =[[./apidocs/org/apache/oozie/local/LocalOozie.html][LocalOozie]]=,
+Oozie provides an embedded Oozie implementation, [LocalOozie](./apidocs/org/apache/oozie/local/LocalOozie.html) ,
which is useful for development, debugging and testing of workflow applications within the convenience of an IDE.
-The code snippet below shows the usage of the =LocalOozie= class. All the interaction with Oozie is done using Oozie
- =OozieClient= Java API, as shown in the previous section.
+The code snippet below shows the usage of the `LocalOozie` class. All the interaction with Oozie is done using Oozie
+ `OozieClient` Java API, as shown in the previous section.
-The examples bundled with Oozie include the complete and running class, =LocalOozieExample= from where this snippet was
+The examples bundled with Oozie include the complete and running class, `LocalOozieExample` from where this snippet was
taken.
-<verbatim>
+
+```
import org.apache.oozie.local.LocalOozie;
import org.apache.oozie.client.OozieClient;
import org.apache.oozie.client.WorkflowJob;
@@ -181,18 +187,18 @@ import java.util.Properties;
// stop local Oozie
LocalOozie.stop();
...
-</verbatim>
+```
+
+Also asynchronous actions like FS action can be used / tested using `LocalOozie` / `OozieClient` API. Please see the module
+`oozie-mini` for details like `fs-decision.xml` workflow example.
-Also asynchronous actions like FS action can be used / tested using =LocalOozie= / =OozieClient= API. Please see the module
-=oozie-mini= for details like =fs-decision.xml= workflow example.
+## Fluent Job API Examples
----++ Fluent Job API Examples
+There are some elaborate examples how to use the [Fluent Job API](DG_FluentJobAPI.html), under `examples/fluentjob/`. There are two
+simple examples covered under [Fluent Job API :: A Simple Example](DG_FluentJobAPI.html#A_Simple_Example) and
+[Fluent Job API :: A More Verbose Example](DG_FluentJobAPI.html#A_More_Verbose_Example).
-There are some elaborate examples how to use the [[DG_FluentJobAPI][Fluent Job API]], under =examples/fluentjob/=. There are two
-simple examples covered under [[DG_FluentJobAPI#A_Simple_Example][Fluent Job API :: A Simple Example]] and
-[[DG_FluentJobAPI#A_More_Verbose_Example][Fluent Job API :: A More Verbose Example]].
+[::Go back to Oozie Documentation Index::](index.html)
-[[index][::Go back to Oozie Documentation Index::]]
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_FluentJobAPI.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_FluentJobAPI.twiki b/docs/src/site/twiki/DG_FluentJobAPI.twiki
index c8b764b..bd36517 100644
--- a/docs/src/site/twiki/DG_FluentJobAPI.twiki
+++ b/docs/src/site/twiki/DG_FluentJobAPI.twiki
@@ -1,27 +1,27 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
----+!! Fluent Job API
+[::Go back to Oozie Documentation Index::](index.html)
-%TOC%
+# Fluent Job API
----++ Introduction
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Introduction
Oozie is a mature workflow scheduler system. XML is the standard way of defining workflow, coordinator, or bundle jobs. For users
who prefer an alternative, the Fluent Job API provides a Java interface instead.
----+++ Motivation
+### Motivation
Prior to Oozie 5.1.0, the following ways were available to submit a workflow, coordinator, or bundle job: through Oozie CLI or via
HTTP submit a generic workflow, coordinator, or bundle job, or submit a Pig, Hive, Sqoop, or MapReduce workflow job.
As the generic way goes, the user has to have uploaded a workflow, coordinator, or bundle XML and all necessary dependencies like
-scripts, JAR or ZIP files, to HDFS beforehand, as well as have a =job.properties= file at command line and / or provide any
+scripts, JAR or ZIP files, to HDFS beforehand, as well as have a `job.properties` file at command line and / or provide any
missing parameters as part of the command.
As the specific Pig, Hive, or Sqoop ways go, the user can provide all necessary parameters as part of the command issued. A
- =workflow.xml= file will be generated with all the necessary details and stored to HDFS so that Oozie can grab it. Note that
+ `workflow.xml` file will be generated with all the necessary details and stored to HDFS so that Oozie can grab it. Note that
dependencies have to be uploaded to HDFS beforehand as well.
There are some usability problems by using the XML job definition. XML is not an ideal way to express dependencies and a directed
@@ -40,7 +40,7 @@ fork / join pairs automatically.
Either way, there were no programmatic ways to define workflow jobs. That doesn't mean users could not generate XML themselves -
actually this is something HUE's Oozie UI also tries to target.
----+++ Goals
+### Goals
Fluent Job API aims to solve following from the user's perspective. It provides a Java API instead of declarative XML to define
workflows. It defines dependencies across actions as opposed to defining a control flow. This is how data engineers and data
@@ -54,7 +54,7 @@ workflow rendered as XML, as well as coexist XML based and Fluent Job API based
time all workflow action types. When XSDs change, as few manual steps are necessary as possible both on API internal and public
side.
----+++ Non-goals
+### Non-goals
The following points are not targeted for the initial release of Fluent Job API with Oozie 5.1.0. It doesn't provide API in any
language other than Java. It doesn't provide a REPL. It doesn't allow for dynamic action instantiation depending on e.g. conditional
@@ -71,7 +71,7 @@ for user-supplied custom actions / XSDs.
Most of the non-goals may be targeted as enhancements of the Fluent Job API for future Oozie releases.
----+++ Approach
+### Approach
When using the Fluent Job API, the following points are different from the XML jobs definition. Instead of control flow (successor)
definition, the user can define dependencies (parents of an action).
@@ -82,32 +82,33 @@ Control flow and necessary boilerplate are generated automatically by keeping us
new dependencies to keep Oozie workflow format of nested fork / join pairs. Note that not every dependency DAG can be expressed in
the Oozie workflow format. When this is not possible, user is notified at build time.
----++ How To Use
+## How To Use
----+++ A Simple Example
+### A Simple Example
The simplest thing to create using the Oozie Fluent Job API is a workflow consisting of only one action. Let's see how it goes, step
by step.
-First, put the project =org.apache.oozie:oozie-fluent-job-api= to the build path. In case of a Maven managed build, create a new
-Maven project and declare a Maven dependency to =org.apache.oozie:oozie-fluent-job-api=.
+First, put the project `org.apache.oozie:oozie-fluent-job-api` to the build path. In case of a Maven managed build, create a new
+Maven project and declare a Maven dependency to `org.apache.oozie:oozie-fluent-job-api`.
-Then, create a class that =implements WorkflowFactory= and implement the method =WorkflowFactory#create()=. inside that method,
-create a =ShellAction= using =ShellActionBuilder=, fill in some attributes then create a =Workflow= using =WorkflowBuilder= using
-the =ShellAction= just built. Return the =Workflow=.
+Then, create a class that `implements WorkflowFactory` and implement the method `WorkflowFactory#create()`. inside that method,
+create a `ShellAction` using `ShellActionBuilder`, fill in some attributes then create a `Workflow` using `WorkflowBuilder` using
+the `ShellAction` just built. Return the `Workflow`.
-Compile a Fluent Job API jar that has the =Main-Class= attribute set to the =WorkflowFactory= subclass just created,
-e.g. =shell-workflow.jar=.
+Compile a Fluent Job API jar that has the `Main-Class` attribute set to the `WorkflowFactory` subclass just created,
+e.g. `shell-workflow.jar`.
-Moving on, [[DG_CommandLineTool#Checking_a_workflow_definition_generated_by_a_Fluent_Job_API_jar_file][check via command line]] that
+Moving on, [check via command line](DG_CommandLineTool.html#Checking_a_workflow_definition_generated_by_a_Fluent_Job_API_jar_file) that
the compiled API JAR file is valid.
As a finishing touch,
-[[DG_CommandLineTool#Running_a_workflow_definition_generated_by_a_Fluent_Job_API_jar_file][run via command line]] the Fluent Job API
+[run via command line](DG_CommandLineTool.html#Running_a_workflow_definition_generated_by_a_Fluent_Job_API_jar_file) the Fluent Job API
workflow.
-*For reference, a simplistic API JAR example consisting of a =Workflow= having only one =ShellAction=:*
-<verbatim>
+**For reference, a simplistic API JAR example consisting of a `Workflow` having only one `ShellAction`:**
+
+```
public class MyFirstWorkflowFactory implements WorkflowFactory {
.
@Override
@@ -129,10 +130,11 @@ public class MyFirstWorkflowFactory implements WorkflowFactory {
return shellWorkflow;
}
}
-</verbatim>
+```
+
+**After check, the generated workflow XML looks like this:**
-*After check, the generated workflow XML looks like this:*
-<verbatim>
+```
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<workflow:workflow-app xmlns:workflow="uri:oozie:workflow:1.0" xmlns:shell="uri:oozie:shell-action:1.0" name="shell-workflow">
.
@@ -163,21 +165,22 @@ public class MyFirstWorkflowFactory implements WorkflowFactory {
<workflow:end name="end"/>
.
</workflow:workflow-app>
-</verbatim>
+```
----+++ A More Verbose Example
+### A More Verbose Example
-*Error handling*
+**Error handling**
-If you would like to provide some error handling in case of action failure, you should add an =ErrorHandler= to the =Node=
-representing the action. The error handler action will be added as the ="error-transition"= of the original action in the generated
-Oozie workflow XML. Both the ="ok-transition"= and the ="error-transition"= of the error handler action itself will lead to an
+If you would like to provide some error handling in case of action failure, you should add an `ErrorHandler` to the `Node`
+representing the action. The error handler action will be added as the `"error-transition"` of the original action in the generated
+Oozie workflow XML. Both the `"ok-transition"` and the `"error-transition"` of the error handler action itself will lead to an
autogenerated kill node.
-*Here you find an example consisting of a =Workflow= having three =ShellAction=s, an error handler =EmailAction=, and one =decision=
-to sort out which way to go:*
-<verbatim>
+**Here you find an example consisting of a `Workflow` having three `ShellAction`s, an error handler `EmailAction`, and one `decision`
+to sort out which way to go:**
+
+```
public class MySecondWorkflowFactory implements WorkflowFactory {
.
@Override
@@ -218,10 +221,11 @@ public class MySecondWorkflowFactory implements WorkflowFactory {
return workflow;
}
}
-</verbatim>
+```
+
+**After check, the generated workflow XML looks like this:**
-*After check, the generated workflow XML looks like this:*
-<verbatim>
+```
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<workflow:workflow-app ... name="shell-example">
.
@@ -305,72 +309,76 @@ public class MySecondWorkflowFactory implements WorkflowFactory {
<workflow:end name="end"/>
.
</workflow:workflow-app>
-</verbatim>
+```
----+++ Runtime Limitations
+### Runtime Limitations
Even if Fluent Job API tries to abstract away the task of assembly job descriptor XML files, there are some runtime
-limitations apart from the [[DG_FluentJobAPI#Non-goals][non-goals section]]. All such limitations are based on the current
+limitations apart from the [non-goals section](DG_FluentJobAPI.html#Non-goals). All such limitations are based on the current
implementations and subject to further improvements and fixes.
-There is only one =kill= possibility in every =workflow=. That is, there can be defined only one =action= to be executed just before
-any other =action= turns to be =kill=ed. Furthermore, =kill= goes to =end= directly. That means, there cannot be defined an
-intricate network of =kill= nodes, cascading sometimes to other =action= nodes, avoiding going to =end= in the first place.
+There is only one `kill` possibility in every `workflow`. That is, there can be defined only one `action` to be executed just before
+any other `action` turns to be `kill`ed. Furthermore, `kill` goes to `end` directly. That means, there cannot be defined an
+intricate network of `kill` nodes, cascading sometimes to other `action` nodes, avoiding going to `end` in the first place.
-There are places where =decision= node generation fails, throwing an =Exception=. The problem is that during the transformation,
-Fluent Job API reaches a state where there is a =fork= that transitions to two =decision= nodes, which in turn split into two paths
-each. One of the paths from the first =decision= joins a path from the other =decision=, but the remaining conditional paths never
-meet. Therefore, not all paths originating from the =fork= converge to the same =join=.
+There are places where `decision` node generation fails, throwing an `Exception`. The problem is that during the transformation,
+Fluent Job API reaches a state where there is a `fork` that transitions to two `decision` nodes, which in turn split into two paths
+each. One of the paths from the first `decision` joins a path from the other `decision`, but the remaining conditional paths never
+meet. Therefore, not all paths originating from the `fork` converge to the same `join`.
----++ Appendixes
+## Appendixes
----+++ AE.A Appendix A, API JAR format
+### AE.A Appendix A, API JAR format
-It's kept simple - all the necessary Java class files that are needed are packed into a JAR file, that has a =META-INF/MANIFEST.MF=
-with a single entry having the =Main-Class= attribute set to the fully qualified name of the entry class, the one that
-=implements WorkflowFactory=:
-<verbatim>
+It's kept simple - all the necessary Java class files that are needed are packed into a JAR file, that has a `META-INF/MANIFEST.MF`
+with a single entry having the `Main-Class` attribute set to the fully qualified name of the entry class, the one that
+`implements WorkflowFactory`:
+
+```
Main-Class: org.apache.oozie.jobs.api.factory.MyFirstWorkflowFactory
-</verbatim>
+```
+
+**An example of the command line assembly of such an API JAR:**
-*An example of the command line assembly of such an API JAR:*
-<verbatim>
+```
jar cfe simple-workflow.jar org.apache.oozie.fluentjob.api.factory.MyFirstWorkflowFactory \
-C /Users/forsage/Workspace/oozie/fluent-job/fluent-job-api/target/classes \
org/apache/oozie/jobs/api/factory/MyFirstWorkflowFactory.class
-</verbatim>
+```
+
+### AE.B Appendix B, Some Useful Builder classes
----+++ AE.B Appendix B, Some Useful Builder classes
+For a complete list of `Builder` classes, please have a look at `oozie-fluent-job-api` artifact's following packages:
-For a complete list of =Builder= classes, please have a look at =oozie-fluent-job-api= artifact's following packages:
- * =org.apache.oozie.fluentjob.api.action= - =ActionBuilder= classes
- * =org.apache.oozie.fluentjob.api.factory= - the single entry point, =WorkflowFactory= is here
- * =org.apache.oozie.fluentjob.api.workflow= - workflow related =Builder= classes
+ * `org.apache.oozie.fluentjob.api.action` - `ActionBuilder` classes
+ * `org.apache.oozie.fluentjob.api.factory` - the single entry point, `WorkflowFactory` is here
+ * `org.apache.oozie.fluentjob.api.workflow` - workflow related `Builder` classes
-On examples how to use these please see =oozie-examples= artifact's =org.apache.oozie.example.fluentjob= package.
+On examples how to use these please see `oozie-examples` artifact's `org.apache.oozie.example.fluentjob` package.
----+++ AE.C Appendix C, How To Extend
+### AE.C Appendix C, How To Extend
Sometimes there are new XSD versions of an existing custom or core workflow action, sometimes it's a new custom workflow action that
gets introduced. In any case, Fluent Job API needs to keep up with the changes.
Here are the steps needed:
- * in =fluent-job-api/pom.xml= extend or modify =jaxb2-maven-plugin= section =sources= by a new =source=
- * in =fluent-job-api/src/main/xjb/bindings.xml= extend by a new or modify an existing =jaxb:bindings=
- * in =fluent-job-api=, =org.apache.oozie.fluentjob.api.mapping= package, introduce a new or modify an existing =DozerConverter=
- * in =dozer_config.xml=, introduce a new or modify an existing =converter= inside =custom-converters=
- * in =fluent-job-api=, =org.apache.oozie.fluentjob.api.action=, introduce a new =Action= and a new =Builder=
+
+ * in `fluent-job-api/pom.xml` extend or modify `jaxb2-maven-plugin` section `sources` by a new `source`
+ * in `fluent-job-api/src/main/xjb/bindings.xml` extend by a new or modify an existing `jaxb:bindings`
+ * in `fluent-job-api`, `org.apache.oozie.fluentjob.api.mapping` package, introduce a new or modify an existing `DozerConverter`
+ * in `dozer_config.xml`, introduce a new or modify an existing `converter` inside `custom-converters`
+ * in `fluent-job-api`, `org.apache.oozie.fluentjob.api.action`, introduce a new `Action` and a new `Builder`
* write new / modify existing relevant unit and integration tests
----+++ AE.D Appendix D, API compatibility guarantees
+### AE.D Appendix D, API compatibility guarantees
-Fluent Job API is available beginning version 5.1.0. It's marked =@InterfaceAudience.Private= (intended for use in Oozie itself) and
-=@InterfaceStability.Unstable= (no stability guarantees are provided across any level of release granularity) to indicate that for
+Fluent Job API is available beginning version 5.1.0. It's marked `@InterfaceAudience.Private` (intended for use in Oozie itself) and
+`@InterfaceStability.Unstable` (no stability guarantees are provided across any level of release granularity) to indicate that for
the next few minor releases it's bound to change a lot.
-Beginning from around 5.4.0 planning the next phase, =@InterfaceStability.Evolving= (compatibility breaking only between minors),
-and a few minor releases later, =@InterfaceAudience.Public= (safe to use outside of Oozie).
+Beginning from around 5.4.0 planning the next phase, `@InterfaceStability.Evolving` (compatibility breaking only between minors),
+and a few minor releases later, `@InterfaceAudience.Public` (safe to use outside of Oozie).
+
+[::Go back to Oozie Documentation Index::](index.html)
-[[index][::Go back to Oozie Documentation Index::]]
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_HCatalogIntegration.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_HCatalogIntegration.twiki b/docs/src/site/twiki/DG_HCatalogIntegration.twiki
index d3107b4..5c592e8 100644
--- a/docs/src/site/twiki/DG_HCatalogIntegration.twiki
+++ b/docs/src/site/twiki/DG_HCatalogIntegration.twiki
@@ -1,32 +1,32 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
----+!! HCatalog Integration (Since Oozie 4.x)
+[::Go back to Oozie Documentation Index::](index.html)
-%TOC%
+# HCatalog Integration (Since Oozie 4.x)
----++ HCatalog Overview
- HCatalog is a table and storage management layer for Hadoop that enables users with different data processing
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## HCatalog Overview
+HCatalog is a table and storage management layer for Hadoop that enables users with different data processing
tools - Pig, MapReduce, and Hive - to more easily read and write data on the grid. HCatalog's table abstraction presents
users with a relational view of data in the Hadoop distributed file system (HDFS).
- Read [[http://incubator.apache.org/hcatalog/docs/r0.5.0/index.html][HCatalog Documentation]] to know more about HCatalog.
+Read [HCatalog Documentation](http://incubator.apache.org/hcatalog/docs/r0.5.0/index.html) to know more about HCatalog.
Working with HCatalog using pig is detailed in
-[[http://incubator.apache.org/hcatalog/docs/r0.5.0/loadstore.html][HCatLoader and HCatStorer]].
+[HCatLoader and HCatStorer](http://incubator.apache.org/hcatalog/docs/r0.5.0/loadstore.html).
Working with HCatalog using MapReduce directly is detailed in
-[[http://incubator.apache.org/hcatalog/docs/r0.5.0/inputoutput.html][HCatInputFormat and HCatOutputFormat]].
+[HCatInputFormat and HCatOutputFormat](http://incubator.apache.org/hcatalog/docs/r0.5.0/inputoutput.html).
----+++ HCatalog notifications
+### HCatalog notifications
HCatalog provides notifications through a JMS provider like ActiveMQ when a new partition is added to a table in the
database. This allows applications to consume those events and schedule the work that depends on them. In case of Oozie,
the notifications are used to determine the availability of HCatalog partitions defined as data dependencies in the
Coordinator and trigger workflows.
-Read [[http://incubator.apache.org/hcatalog/docs/r0.5.0/notification.html][HCatalog Notification]] to know more about
+Read [HCatalog Notification](http://incubator.apache.org/hcatalog/docs/r0.5.0/notification.html) to know more about
notifications in HCatalog.
----++ Oozie HCatalog Integration
+## Oozie HCatalog Integration
Oozie's Coordinators so far have been supporting HDFS directories as a input data dependency. When a HDFS URI
template is specified as a dataset and input events are defined in Coordinator for the dataset, Oozie performs data
availability checks by polling the HDFS directory URIs resolved based on the nominal time. When all the data
@@ -49,14 +49,14 @@ coordinator action was materialized and to deal with missed notifications due to
fallback polling is usually lower than the constant polling. Defaults are 10 minutes and 1 minute respectively.
----+++ Oozie Server Configuration
- Refer to [[AG_Install#HCatalog_Configuration][HCatalog Configuration]] section of [[AG_Install][Oozie Install]]
+### Oozie Server Configuration
+ Refer to [HCatalog Configuration](AG_Install.html#HCatalog_Configuration) section of [Oozie Install](AG_Install.html)
documentation for the Oozie server side configuration required to support HCatalog table partitions as a data dependency.
----+++ HCatalog URI Format
+### HCatalog URI Format
Oozie supports specifying HCatalog partitions as a data dependency through a URI notation. The HCatalog partition URI is
-used to identify a set of table partitions: hcat://bar:8020/logsDB/logsTable/dt=20090415;region=US.
+used to identify a set of table partitions: `hcat://bar:8020/logsDB/logsTable/dt=20090415;region=US`
The format to specify a HCatalog table URI is:
@@ -67,14 +67,15 @@ The format to specify a HCatalog table partition URI is:
hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value];...
For example,
-<verbatim>
+
+```
<dataset name="logs" frequency="${coord:days(1)}"
initial-instance="2009-02-15T08:15Z" timezone="America/Los_Angeles">
<uri-template>
hcat://myhcatmetastore:9080/database1/table1/datestamp=${YEAR}${MONTH}${DAY}${HOUR};region=USA
</uri-template>
</dataset>
-</verbatim>
+```
Post Oozie-4.3.0 release, Oozie also supports the multiple HCatalog servers in the URI. Each of the server needs to be
separated by single comma (,).
@@ -84,59 +85,63 @@ The format to specify a HCatalog table partition URI with multiple HCatalog serv
hcat://[metastore_server]:[port],[metastore_server]:[port]/[database_name]/[table_name]/[partkey1]=[value];[partkey2]=[value];...
For example,
-<verbatim>
+
+```
<dataset name="logs" frequency="${coord:days(1)}"
initial-instance="2009-02-15T08:15Z" timezone="America/Los_Angeles">
<uri-template>
hcat://myhcatmetastore:9080,myhcatmetastore:9080/database1/table1/datestamp=${YEAR}${MONTH}${DAY}${HOUR};region=USA
</uri-template>
</dataset>
-</verbatim>
+```
The regex for parsing the multiple HCatalog URI is exposed via oozie-site.xml, So Users can modify if there is any
-requirement. Key for the regex is: =oozie.hcat.uri.regex.pattern=
+requirement. Key for the regex is: `oozie.hcat.uri.regex.pattern`
For example, following has multiple HCatalog URI with multiple HCatalog servers. To understand this, Oozie will split them into
two HCatalog URIs. For splitting the URIs, above mentioned regex is used.
-hcat://hostname1:1000,hcat://hostname2:2000/mydb/clicks/datastamp=12;region=us,scheme://hostname3:3000,scheme://hostname4:4000,scheme://hostname5:5000/db/table/p1=12;p2=us
+`hcat://hostname1:1000,hcat://hostname2:2000/mydb/clicks/datastamp=12;region=us,scheme://hostname3:3000,scheme://hostname4:4000,scheme://hostname5:5000/db/table/p1=12;p2=us`
After split: (This is internal Oozie mechanism)
-hcat://hostname1:1000,hcat://hostname2:2000/mydb/clicks/datastamp=12;region=us
+`hcat://hostname1:1000,hcat://hostname2:2000/mydb/clicks/datastamp=12;region=us`
-scheme://hostname3:3000,scheme://hostname4:4000,scheme://hostname5:5000/db/table/p1=12;p2=us
+`scheme://hostname3:3000,scheme://hostname4:4000,scheme://hostname5:5000/db/table/p1=12;p2=us`
-#HCatalogLibraries
----+++ HCatalog Libraries
+<a name="HCatalogLibraries"></a>
+### HCatalog Libraries
-A workflow action interacting with HCatalog requires the following jars in the classpath:
+A workflow action interacting with HCatalog requires the following jars in the classpath:
hcatalog-core.jar, hcatalog-pig-adapter.jar, webhcat-java-client.jar, hive-common.jar, hive-exec.jar,
hive-metastore.jar, hive-serde.jar and libfb303.jar.
hive-site.xml which has the configuration to talk to the HCatalog server also needs to be in the classpath. The correct
version of HCatalog and hive jars should be placed in classpath based on the version of HCatalog installed on the cluster.
The jars can be added to the classpath of the action using one of the below ways.
- * You can place the jars and hive-site.xml in the system shared library. The shared library for a pig, hive or java action can be overridden to include hcatalog shared libraries along with the action's shared library. Refer to [[WorkflowFunctionalSpec.html#a17_HDFS_Share_Libraries_for_Workflow_Applications_since_Oozie_2.3][Shared Libraries]] for more information. The oozie-sharelib-[version].tar.gz in the oozie distribution bundles the required HCatalog jars in a hcatalog sharelib. If using a different version of HCatalog than the one bundled in the sharelib, copy the required HCatalog jars from such version into the sharelib.
+
+ * You can place the jars and hive-site.xml in the system shared library. The shared library for a pig, hive or java action can be overridden to include hcatalog shared libraries along with the action's shared library. Refer to [Shared Libraries](WorkflowFunctionalSpec.html#a17_HDFS_Share_Libraries_for_Workflow_Applications_since_Oozie_2.3) for more information. The oozie-sharelib-[version].tar.gz in the oozie distribution bundles the required HCatalog jars in a hcatalog sharelib. If using a different version of HCatalog than the one bundled in the sharelib, copy the required HCatalog jars from such version into the sharelib.
* You can place the jars and hive-site.xml in the workflow application lib/ path.
- * You can specify the location of the jar files in =archive= tag and the hive-site.xml in =file= tag in the corresponding pig, hive or java action.
+ * You can specify the location of the jar files in `archive` tag and the hive-site.xml in `file` tag in the corresponding pig, hive or java action.
----+++ Coordinator
+### Coordinator
+
+Refer to [Coordinator Functional Specification](CoordinatorFunctionalSpec.html) for more information about
-Refer to [[CoordinatorFunctionalSpec][Coordinator Functional Specification]] for more information about
* how to specify HCatalog partitions as a data dependency using input dataset events
* how to specify HCatalog partitions as output dataset events
* the various EL functions available to work with HCatalog dataset events and how to use them to access HCatalog partitions in pig, hive or java actions in a workflow.
----+++ Workflow
-Refer to [[WorkflowFunctionalSpec][Workflow Functional Specification]] for more information about
+### Workflow
+Refer to [Workflow Functional Specification](WorkflowFunctionalSpec.html) for more information about
+
* how to drop HCatalog table/partitions in the prepare block of a action
* the HCatalog EL functions available to use in workflows
-Refer to [[DG_ActionAuthentication][Action Authentication]] for more information about
+Refer to [Action Authentication](DG_ActionAuthentication.html) for more information about
+
* how to access a secure HCatalog from any action (e.g. hive, pig, etc) in a workflow
----+++ Known Issues
+### Known Issues
* When rerunning a coordinator action without specifying -nocleanup option if the 'output-event' are hdfs directories, then they are deleted. But if the 'output-event' is a hcatalog partition, currently the partition is not dropped.
-</noautolink>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_Hive2ActionExtension.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_Hive2ActionExtension.twiki b/docs/src/site/twiki/DG_Hive2ActionExtension.twiki
index efbe56d..d81ed02 100644
--- a/docs/src/site/twiki/DG_Hive2ActionExtension.twiki
+++ b/docs/src/site/twiki/DG_Hive2ActionExtension.twiki
@@ -1,41 +1,42 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
+
+[::Go back to Oozie Documentation Index::](index.html)
-----
----+!! Oozie Hive 2 Action Extension
+# Oozie Hive 2 Action Extension
-%TOC%
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
----++ Hive 2 Action
+## Hive 2 Action
-The =hive2= action runs Beeline to connect to Hive Server 2.
+The `hive2` action runs Beeline to connect to Hive Server 2.
The workflow job will wait until the Hive Server 2 job completes before
continuing to the next action.
-To run the Hive Server 2 job, you have to configure the =hive2= action with the =resource-manager=, =name-node=, =jdbc-url=,
- =password= elements, and either Hive's =script= or =query= element, as well as the necessary parameters and configuration.
+To run the Hive Server 2 job, you have to configure the `hive2` action with the `resource-manager`, `name-node`, `jdbc-url`,
+ `password` elements, and either Hive's `script` or `query` element, as well as the necessary parameters and configuration.
-A =hive2= action can be configured to create or delete HDFS directories
+A `hive2` action can be configured to create or delete HDFS directories
before starting the Hive Server 2 job.
Oozie EL expressions can be used in the inline configuration. Property
-values specified in the =configuration= element override values specified
-in the =job-xml= file.
+values specified in the `configuration` element override values specified
+in the `job-xml` file.
-As with Hadoop =map-reduce= jobs, it is possible to add files and
+As with Hadoop `map-reduce` jobs, it is possible to add files and
archives in order to make them available to Beeline. Refer to the
-[WorkflowFunctionalSpec#FilesArchives][Adding Files and Archives for the Job]
+[Adding Files and Archives for the Job](WorkflowFunctionalSpec.html#FilesArchives)
section for more information about this feature.
Oozie Hive 2 action supports Hive scripts with parameter variables, their
-syntax is =${VARIABLES}=.
+syntax is `${VARIABLES}`.
+
+**Syntax:**
-*Syntax:*
-<verbatim>
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<action name="[NODE-NAME]">
@@ -75,44 +76,45 @@ syntax is =${VARIABLES}=.
</action>
...
</workflow-app>
-</verbatim>
+```
-The =prepare= element, if present, indicates a list of paths to delete
-or create before starting the job. Specified paths must start with =hdfs://HOST:PORT=.
+The `prepare` element, if present, indicates a list of paths to delete
+or create before starting the job. Specified paths must start with `hdfs://HOST:PORT`.
-The =job-xml= element, if present, specifies a file containing configuration
-for Beeline. Multiple =job-xml= elements are allowed in order to specify multiple =job.xml= files.
+The `job-xml` element, if present, specifies a file containing configuration
+for Beeline. Multiple `job-xml` elements are allowed in order to specify multiple `job.xml` files.
-The =configuration= element, if present, contains configuration
+The `configuration` element, if present, contains configuration
properties that are passed to the Beeline job.
-The =jdbc-url= element must contain the JDBC URL for the Hive Server 2. Beeline will use this to know where to connect to.
+The `jdbc-url` element must contain the JDBC URL for the Hive Server 2. Beeline will use this to know where to connect to.
-The =password= element must contain the password of the current user. However, the =password= is only used if Hive Server 2 is
+The `password` element must contain the password of the current user. However, the `password` is only used if Hive Server 2 is
backed by something requiring a password (e.g. LDAP); non-secured Hive Server 2 or Kerberized Hive Server 2 don't require a password
-so in those cases the =password= is ignored and can be omitted from the action XML. It is up to the user to ensure that a password
+so in those cases the `password` is ignored and can be omitted from the action XML. It is up to the user to ensure that a password
is specified when required.
-The =script= element must contain the path of the Hive script to
+The `script` element must contain the path of the Hive script to
execute. The Hive script can be templatized with variables of the form
-=${VARIABLE}=. The values of these variables can then be specified
-using the =params= element.
+`${VARIABLE}`. The values of these variables can then be specified
+using the `params` element.
-The =query= element available from uri:oozie:hive2-action:0.2, can be used instead of the =script= element. It allows for embedding
-queries within the =worklfow.xml= directly. Similar to the =script= element, it also allows for the templatization of variables
-in the form =${VARIABLE}=.
+The `query` element available from uri:oozie:hive2-action:0.2, can be used instead of the `script` element. It allows for embedding
+queries within the `worklfow.xml` directly. Similar to the `script` element, it also allows for the templatization of variables
+in the form `${VARIABLE}`.
-The =params= element, if present, contains parameters to be passed to
+The `params` element, if present, contains parameters to be passed to
the Hive script.
-The =argument= element, if present, contains arguments to be passed as-is to Beeline.
+The `argument` element, if present, contains arguments to be passed as-is to Beeline.
All the above elements can be parameterized (templatized) using EL
expressions.
-*Example:*
+**Example:**
-<verbatim>
+
+```
<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
...
<action name="my-hive2-action">
@@ -139,22 +141,21 @@ expressions.
</action>
...
</workflow-app>
-</verbatim>
+```
----+++ Security
+### Security
-As mentioned above, =password= is only used in cases where Hive Server 2 is backed by something requiring a password (e.g. LDAP).
+As mentioned above, `password` is only used in cases where Hive Server 2 is backed by something requiring a password (e.g. LDAP).
Non-secured Hive Server 2 and Kerberized Hive Server 2 don't require a password so in these cases it can be omitted.
-See [[DG_UnifiedCredentialsModule][here]] for more information on the configuration for using the Hive Server 2 Action
-with a Kerberized Hive Server 2.
----++ Appendix, Hive 2 XML-Schema
+## Appendix, Hive 2 XML-Schema
+
+### AE.A Appendix A, Hive 2 XML-Schema
----+++ AE.A Appendix A, Hive 2 XML-Schema
+#### Hive 2 Action Schema Version 1.0
----++++ Hive 2 Action Schema Version 1.0
-<verbatim>
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:hive2="uri:oozie:hive2-action:1.0" elementFormDefault="qualified"
targetNamespace="uri:oozie:hive2-action:1.0">
@@ -188,10 +189,11 @@ with a Kerberized Hive Server 2.
</xs:complexType>
.
</xs:schema>
-</verbatim>
+```
----++++ Hive 2 Action Schema Version 0.2
-<verbatim>
+#### Hive 2 Action Schema Version 0.2
+
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:hive2="uri:oozie:hive2-action:0.2" elementFormDefault="qualified"
targetNamespace="uri:oozie:hive2-action:0.2">
@@ -248,10 +250,11 @@ with a Kerberized Hive Server 2.
</xs:complexType>
.
</xs:schema>
-</verbatim>
+```
+
+#### Hive 2 Action Schema Version 0.1
----++++ Hive 2 Action Schema Version 0.1
-<verbatim>
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:hive2="uri:oozie:hive2-action:0.1" elementFormDefault="qualified"
targetNamespace="uri:oozie:hive2-action:0.1">
@@ -305,8 +308,8 @@ with a Kerberized Hive Server 2.
</xs:complexType>
.
</xs:schema>
-</verbatim>
+```
+
+[::Go back to Oozie Documentation Index::](index.html)
-[[index][::Go back to Oozie Documentation Index::]]
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_HiveActionExtension.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_HiveActionExtension.twiki b/docs/src/site/twiki/DG_HiveActionExtension.twiki
index aaa74fa..99a73c6 100644
--- a/docs/src/site/twiki/DG_HiveActionExtension.twiki
+++ b/docs/src/site/twiki/DG_HiveActionExtension.twiki
@@ -1,48 +1,49 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
+
+[::Go back to Oozie Documentation Index::](index.html)
-----
----+!! Oozie Hive Action Extension
+# Oozie Hive Action Extension
-%TOC%
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-#HiveAction
----++ Hive Action
+<a name="HiveAction"></a>
+## Hive Action
-The =hive= action runs a Hive job.
+The `hive` action runs a Hive job.
The workflow job will wait until the Hive job completes before
continuing to the next action.
-To run the Hive job, you have to configure the =hive= action with the =resource-manager=, =name-node= and Hive =script=
-(or Hive =query=) elements as well as the necessary parameters and configuration.
+To run the Hive job, you have to configure the `hive` action with the `resource-manager`, `name-node` and Hive `script`
+(or Hive `query`) elements as well as the necessary parameters and configuration.
-A =hive= action can be configured to create or delete HDFS directories
+A `hive` action can be configured to create or delete HDFS directories
before starting the Hive job.
-Hive configuration can be specified with a file, using the =job-xml=
-element, and inline, using the =configuration= elements.
+Hive configuration can be specified with a file, using the `job-xml`
+element, and inline, using the `configuration` elements.
Oozie EL expressions can be used in the inline configuration. Property
-values specified in the =configuration= element override values specified
-in the =job-xml= file.
+values specified in the `configuration` element override values specified
+in the `job-xml` file.
-Note that YARN =yarn.resourcemanager.address= (=resource-manager=) and HDFS =fs.default.name= (=name-node=) properties
+Note that YARN `yarn.resourcemanager.address` (`resource-manager`) and HDFS `fs.default.name` (`name-node`) properties
must not be present in the inline configuration.
-As with Hadoop =map-reduce= jobs, it is possible to add files and
+As with Hadoop `map-reduce` jobs, it is possible to add files and
archives in order to make them available to the Hive job. Refer to the
[WorkflowFunctionalSpec#FilesArchives][Adding Files and Archives for the Job]
section for more information about this feature.
Oozie Hive action supports Hive scripts with parameter variables, their
-syntax is =${VARIABLES}=.
+syntax is `${VARIABLES}`.
+
+**Syntax:**
-*Syntax:*
-<verbatim>
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<action name="[NODE-NAME]">
@@ -77,37 +78,38 @@ syntax is =${VARIABLES}=.
</action>
...
</workflow-app>
-</verbatim>
+```
-The =prepare= element, if present, indicates a list of paths to delete
-or create before starting the job. Specified paths must start with =hdfs://HOST:PORT=.
+The `prepare` element, if present, indicates a list of paths to delete
+or create before starting the job. Specified paths must start with `hdfs://HOST:PORT`.
-The =job-xml= element, if present, specifies a file containing configuration
-for the Hive job. As of schema 0.3, multiple =job-xml= elements are allowed in order to
-specify multiple =job.xml= files.
+The `job-xml` element, if present, specifies a file containing configuration
+for the Hive job. As of schema 0.3, multiple `job-xml` elements are allowed in order to
+specify multiple `job.xml` files.
-The =configuration= element, if present, contains configuration
+The `configuration` element, if present, contains configuration
properties that are passed to the Hive job.
-The =script= element must contain the path of the Hive script to
+The `script` element must contain the path of the Hive script to
execute. The Hive script can be templatized with variables of the form
-=${VARIABLE}=. The values of these variables can then be specified
-using the =params= element.
+`${VARIABLE}`. The values of these variables can then be specified
+using the `params` element.
-The =query= element available from uri:oozie:hive-action:0.6, can be used instead of the
-=script= element. It allows for embedding queries within the =worklfow.xml= directly.
-Similar to the =script= element, it also allows for the templatization of variables in the
-form =${VARIABLE}=.
+The `query` element available from uri:oozie:hive-action:0.6, can be used instead of the
+`script` element. It allows for embedding queries within the `worklfow.xml` directly.
+Similar to the `script` element, it also allows for the templatization of variables in the
+form `${VARIABLE}`.
-The =params= element, if present, contains parameters to be passed to
+The `params` element, if present, contains parameters to be passed to
the Hive script.
All the above elements can be parameterized (templatized) using EL
expressions.
-*Example:*
+**Example:**
+
-<verbatim>
+```
<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
...
<action name="myfirsthivejob">
@@ -132,14 +134,14 @@ expressions.
</action>
...
</workflow-app>
-</verbatim>
+```
----+++ Hive Default and Site Configuration Files
+### Hive Default and Site Configuration Files
-Hive (as of Hive 0.8) ignores a =hive-default.xml= file. As a result, Oozie (as of Oozie 3.4) ignores the =oozie.hive.defaults=
+Hive (as of Hive 0.8) ignores a `hive-default.xml` file. As a result, Oozie (as of Oozie 3.4) ignores the `oozie.hive.defaults`
property that was previously required by earlier versions of Oozie for the Hive action.
----+++ Hive Action Logging
+### Hive Action Logging
Hive action logs are redirected to the Oozie Launcher map-reduce job task STDOUT/STDERR that runs Hive.
@@ -147,14 +149,15 @@ From Oozie web-console, from the Hive action pop up using the 'Console URL' link
to navigate to the Oozie Launcher map-reduce job task logs via the Hadoop job-tracker web-console.
The logging level of the Hive action can set in the Hive action configuration using the
-property =oozie.hive.log.level=. The default value is =INFO=.
+property `oozie.hive.log.level`. The default value is `INFO`.
----++ Appendix, Hive XML-Schema
+## Appendix, Hive XML-Schema
----+++ AE.A Appendix A, Hive XML-Schema
+### AE.A Appendix A, Hive XML-Schema
----++++ Hive Action Schema Version 1.0
-<verbatim>
+#### Hive Action Schema Version 1.0
+
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:hive="uri:oozie:hive-action:1.0"
elementFormDefault="qualified"
@@ -187,10 +190,11 @@ property =oozie.hive.log.level=. The default value is =INFO=.
</xs:complexType>
.
</xs:schema>
-</verbatim>
+```
+
+#### Hive Action Schema Version 0.6
----++++ Hive Action Schema Version 0.6
-<verbatim>
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:hive="uri:oozie:hive-action:0.6" elementFormDefault="qualified"
targetNamespace="uri:oozie:hive-action:0.6">
@@ -245,9 +249,10 @@ property =oozie.hive.log.level=. The default value is =INFO=.
</xs:complexType>
.
</xs:schema>
-</verbatim>
----++++ Hive Action Schema Version 0.5
-<verbatim>
+```
+#### Hive Action Schema Version 0.5
+
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:hive="uri:oozie:hive-action:0.5" elementFormDefault="qualified"
targetNamespace="uri:oozie:hive-action:0.5">
@@ -299,10 +304,11 @@ property =oozie.hive.log.level=. The default value is =INFO=.
</xs:complexType>
.
</xs:schema>
-</verbatim>
+```
+
+#### Hive Action Schema Version 0.4
----++++ Hive Action Schema Version 0.4
-<verbatim>
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:hive="uri:oozie:hive-action:0.4" elementFormDefault="qualified"
targetNamespace="uri:oozie:hive-action:0.4">
@@ -353,10 +359,11 @@ property =oozie.hive.log.level=. The default value is =INFO=.
</xs:complexType>
.
</xs:schema>
-</verbatim>
+```
+
+#### Hive Action Schema Version 0.3
----++++ Hive Action Schema Version 0.3
-<verbatim>
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:hive="uri:oozie:hive-action:0.3" elementFormDefault="qualified"
targetNamespace="uri:oozie:hive-action:0.3">
@@ -407,10 +414,11 @@ property =oozie.hive.log.level=. The default value is =INFO=.
</xs:complexType>
.
</xs:schema>
-</verbatim>
+```
----++++ Hive Action Schema Version 0.2
-<verbatim>
+#### Hive Action Schema Version 0.2
+
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:hive="uri:oozie:hive-action:0.2" elementFormDefault="qualified"
targetNamespace="uri:oozie:hive-action:0.2">
@@ -461,8 +469,8 @@ property =oozie.hive.log.level=. The default value is =INFO=.
</xs:complexType>
.
</xs:schema>
-</verbatim>
+```
+
+[::Go back to Oozie Documentation Index::](index.html)
-[[index][::Go back to Oozie Documentation Index::]]
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_JMSNotifications.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_JMSNotifications.twiki b/docs/src/site/twiki/DG_JMSNotifications.twiki
index a4b0f0d..e8f8a76 100644
--- a/docs/src/site/twiki/DG_JMSNotifications.twiki
+++ b/docs/src/site/twiki/DG_JMSNotifications.twiki
@@ -1,26 +1,26 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
----+!! JMS Notifications
+[::Go back to Oozie Documentation Index::](index.html)
-%TOC%
+# JMS Notifications
----++ Overview
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Overview
Since Oozie 4.0, Oozie supports publishing notifications to a JMS Provider for job status changes and SLA met and miss
events. This provides an alternative to polling Oozie for Job or SLA related information and getting events as they
happen without any delay. Clients can be written to consume these notifications and integrate with different monitoring
and alerting systems.
----++ Oozie Server Configuration
-Refer to [[AG_Install#Notifications_Configuration][Notifications Configuration]] section of [[AG_Install][Oozie Install]]
+## Oozie Server Configuration
+Refer to [Notifications Configuration](AG_Install.html#Notifications_Configuration) section of [Oozie Install](AG_Install.html)
documentation for the Oozie server side configuration required to support publishing notifications to a JMS Provider.
The JNDI properties for the JMS provider, the topics to publish to and the notification types to publish (Job and/or SLA)
need to be configured.
----++ Consuming Notifications
+## Consuming Notifications
----+++ Notification types
+### Notification types
Job and SLA notifications are published to the configured JMS Provider on the configured topics.
Job status change notifications include job start, success, failure, suspended, etc. Currently only workflow job and
@@ -28,26 +28,28 @@ coordinator action status change notifications are published.
SLA notifications include START_MET, END_MET, DURATION_MET, START_MISS, END_MISS, DURATION_MISS events and are published
for a workflow job, workflow action or coordinator action for which SLA information is configured in the job xml. Refer
-to [[DG_SLAMonitoring#Configuring_SLA_in_Applications][SLA Configuration]] for information on configuring SLA for a workflow or
+to [SLA Configuration](DG_SLAMonitoring.html#Configuring_SLA_in_Applications) for information on configuring SLA for a workflow or
coordinator.
----+++ JMS Topic
+### JMS Topic
Consumers interested in notification on events will require to know the JNDI properties to connect to the JMS provider.
They will also need to know the JMS topic on which notifications for a particular job are published.
Oozie Client provides the following APIs :
-<verbatim>
+
+```
public JMSConnectionInfo getJMSConnectionInfo()
public String getJMSTopicName(String jobId)
-</verbatim>
+```
The JMSConnectionInfo exposes 3 methods:
-<verbatim>
+
+```
Properties getJNDIProperties();
String getTopicPattern(AppType appType);
String getTopicPrefix();
-</verbatim>
+```
The topic is obtained by concatenating topic prefix and the substituted value for topic pattern. The topic pattern
can be a constant value like workflow or coordinator which the administrator has configured or a variable (either ${username}
@@ -59,13 +61,14 @@ The getJMSTopicName API can be used if the job id is already known and will give
notifications for that job are published.
----+++ JMS Message Format
-JMS messages published are =javax.jms.TextMessage=. The body contains JSON and the header contains multiple properties
+### JMS Message Format
+JMS messages published are `javax.jms.TextMessage`. The body contains JSON and the header contains multiple properties
that can be used as selectors. The header properties are not repeated in the body of the message to keep the messages
small.
<b>Message Header:</b> <br/>
The different header properties are:
+
* msgType - Value can be JOB or SLA.
* user - The user who submitted the job.
* appName - Application name of the job.
@@ -86,77 +89,88 @@ FAILURE = When the Workflow Job or Coordinator Action is in terminal state other
<b>Message Body for Job Notifications:</b> <br/>
Sample JSON response for different job and sla events as below.
-<verbatim>
+
+```
Workflow Job in RUNNING state:
{"status":"RUNNING","id":"0000042-130618221729631-oozie-oozi-W","startTime":1342915200000}
-</verbatim>
+```
-<verbatim>
+
+```
Workflow Job in FAILED state:
{"status":"FAILED","errorCode":"EL_ERROR","errorMessage":"variable [dummyvalue] cannot be resolved",
"id":"0000042-130618221729631-oozie-oozi-W","startTime":1342915200000,"endTime":1366672183543}
-</verbatim>
+```
+
-<verbatim>
+```
Workflow Job in SUCCEEDED state:
{"status":"SUCCEEDED","id":"0000039-130618221729631-oozie-oozi-W","startTime":1342915200000,
"parentId":"0000025-130618221729631-oozie-oozi-C@1","endTime":1366676224154}
-</verbatim>
+```
-<verbatim>
+
+```
Workflow Job in SUSPENDED state:
{"status":"SUSPENDED","id":"0000039-130618221729631-oozie-oozi-W","startTime":1342915200000,
"parentId":"0000025-130618221729631-oozie-oozi-C@1"}
-</verbatim>
+```
+
-<verbatim>
+```
Coordinator Action in WAITING state:
{"status":"WAITING","nominalTime":1310342400000,"missingDependency":"hdfs://gsbl90107.blue.com:8020/user/john/dir1/file1",
"id":"0000025-130618221729631-oozie-oozi-C@1","startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C"}
-</verbatim>
+```
-<verbatim>
+
+```
Coordinator Action in RUNNING state:
{"status":"RUNNING","nominalTime":1310342400000,"id":"0000025-130618221729631-oozie-oozi-C@1",
"startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C"}
-</verbatim>
+```
+
-<verbatim>
+```
Coordinator Action in SUCCEEDED state:
{"status":"SUCCEEDED","nominalTime":1310342400000,"id":"0000025-130618221729631-oozie-oozi-C@1",
"startTime":1342915200000,"parentId":"0000025-130618221729631-oozie-oozi-C","endTime":1366677082799}
-</verbatim>
+```
+
-<verbatim>
+```
Coordinator Action in FAILED state:
{"status":"FAILED","errorCode":"E0101","errorMessage":"dummyError","nominalTime":1310342400000,
"id":"0000025-130618221729631-oozie-oozi-C@1","startTime":1342915200000,
"parentId":"0000025-130618221729631-oozie-oozi-C","endTime":1366677140818}
-</verbatim>
+```
<b>Message Body for SLA Notifications:</b> <br/>
-<verbatim>
+
+```
Workflow Job in sla END_MISS state:
{"id":"0000000-000000000000001-oozie-wrkf-C@1","parentId":"0000000-000000000000001-oozie-wrkf-C",
"expectedStartTime":1356998400000,"notificationMessage":"notification of start miss","actualStartTime":1357002000000,
"expectedDuration":-1, "actualDuration":3600,"expectedEndTime":1356998400000,"actualEndTime":1357002000000}
-</verbatim>
+```
----+++ JMS Client
+### JMS Client
Oozie provides a helper class JMSMessagingUtils for consumers to deserialize the JMS messages back to Java objects.
The below method getEventMessage() expects a sub type of EventMessage.
There are different implementations of EventMessage - WorkflowJobMessage, CoordinatorActionMessage and SLAMessage.
-<verbatim>
+
+```
<T extends EventMessage> T JMSMessagingUtils.getEventMessage(Message jmsMessage)
-</verbatim>
----++++ Example
+```
+#### Example
Below is a sample code to consume notifications.
First, create the Oozie client and retrieve the JNDI properties to make a connection to the JMS server.
-<verbatim>
+
+```
OozieClient oc = new OozieClient("http://localhost:11000/oozie");
JMSConnectionInfo jmsInfo = oc.getJMSConnectionInfo();
Properties jndiProperties = jmsInfo.getJNDIProperties();
@@ -181,13 +195,14 @@ First, create the Oozie client and retrieve the JNDI properties to make a connec
MessageConsumer consumer = session.createConsumer(topic);
consumer.setMessageListener(this);
connection.start();
-</verbatim>
+```
-To start receiving messages, the JMS [[http://docs.oracle.com/javaee/6/api/javax/jms/MessageListener.html][MessageListener]]
+To start receiving messages, the JMS [MessageListener](http://docs.oracle.com/javaee/6/api/javax/jms/MessageListener.html)
interface needs to be implemented. Also, its onMessage() method needs to be implemented.
This method will be called whenever a message is available on the JMS bus.
-<verbatim>
+
+```
public void onMessage(Message message) {
if (message.getStringProperty(JMSHeaderConstants.MESSAGE_TYPE).equals(MessageType.SLA.name())){
SLAMessage slaMessage = JMSMessagingUtils.getEventMessage(message);
@@ -198,22 +213,24 @@ This method will be called whenever a message is available on the JMS bus.
// Further processing
}
}
-</verbatim>
+```
----++++ Applying Selectors
+#### Applying Selectors
Below is a sample ActiveMQ text message header properties section.
-<verbatim>
+
+```
ActiveMQTextMessage
{properties = {appName = map-reduce-wf, msgType=JOB, appType=WORKFLOW_JOB, user=john, msgFormat=json, eventStatus=STARTED} ...}
-</verbatim>
+```
On the header properties, consumers can apply JMS selectors to filter messages from JMS provider.
-They are listed at [[../docs/client/apidocs/org/apache/oozie/client/event/jms/JMSHeaderConstants.html][JMSHeaderConstants]]
+They are listed at [JMSHeaderConstants](../docs/client/apidocs/org/apache/oozie/client/event/jms/JMSHeaderConstants.html)
Sample use of selector to filter events related to Job which have failed and has a particular app-name
-<verbatim>
+
+```
String selector=JMSHeaderConstants.EVENT_STATUS + "='FAILURE' AND " + JMSHeaderConstants.APP_NAME + "='app-name'";
MessageConsumer consumer = session.createConsumer(topic, selector);
-</verbatim>
+```
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_Overview.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_Overview.twiki b/docs/src/site/twiki/DG_Overview.twiki
index 3ec94a2..6a2b9d2 100644
--- a/docs/src/site/twiki/DG_Overview.twiki
+++ b/docs/src/site/twiki/DG_Overview.twiki
@@ -1,8 +1,8 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
----+ Oozie Workflow Overview
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Oozie Workflow Overview
Oozie is a server based _Workflow Engine_ specialized in running workflow jobs with actions that run Hadoop Map/Reduce
and Pig jobs.
@@ -14,7 +14,7 @@ a control dependency DAG (Directed Acyclic Graph). "control dependency" from one
action can't run until the first action has completed.
Oozie workflows definitions are written in hPDL (a XML Process Definition Language similar to
-[[http://www.jboss.org/jbossjbpm/][JBOSS JBPM]] jPDL).
+[JBOSS JBPM](http://www.jboss.org/jbossjbpm/) jPDL).
Oozie workflow actions start jobs in remote systems (i.e. Hadoop, Pig). Upon action completion, the remote systems
callback Oozie to notify the action completion, at this point Oozie proceeds to the next action in the workflow.
@@ -25,26 +25,27 @@ by default to the user code.
Oozie workflows contain control flow nodes and action nodes.
-Control flow nodes define the beginning and the end of a workflow ( =start=, =end= and =fail= nodes) and provide a
-mechanism to control the workflow execution path ( =decision=, =fork= and =join= nodes).
+Control flow nodes define the beginning and the end of a workflow ( `start`, `end` and `fail` nodes) and provide a
+mechanism to control the workflow execution path ( `decision`, `fork` and `join` nodes).
Action nodes are the mechanism by which a workflow triggers the execution of a computation/processing task. Oozie
provides support for different types of actions: Hadoop map-reduce, Hadoop file system, Pig, SSH, HTTP, eMail and
Oozie sub-workflow. Oozie can be extended to support additional type of actions.
-Oozie workflows can be parameterized (using variables like =${inputDir}= within the workflow definition). When
+Oozie workflows can be parameterized (using variables like `${inputDir}` within the workflow definition). When
submitting a workflow job values for the parameters must be provided. If properly parameterized (i.e. using different
output directories) several identical workflow jobs can concurrently.
----++ WordCount Workflow Example
+## WordCount Workflow Example
+
+**Workflow Diagram:**
-*Workflow Diagram:*
+<img src="./DG_Overview.png"/>
-<img src="%ATTACHURLPATH%/DG_Overview.png"/>
+**hPDL Workflow Definition:**
-*hPDL Workflow Definition:*
-<verbatim>
+```
<workflow-app name='wordcount-wf' xmlns="uri:oozie:workflow:0.1">
<start to='wordcount'/>
<action name='wordcount'>
@@ -78,8 +79,8 @@ output directories) several identical workflow jobs can concurrently.
</kill/>
<end name='end'/>
</workflow-app>
-</verbatim>
+```
+
+[::Go back to Oozie Documentation Index::](index.html)
-[[index][::Go back to Oozie Documentation Index::]]
-</noautolink>
[08/11] oozie git commit: OOZIE-2734 [docs] Switch from TWiki to
Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki b/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
index cd416d4..d31d1aa 100644
--- a/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
+++ b/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
@@ -1,74 +1,89 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
+
+[::Go back to Oozie Documentation Index::](index.html)
-----
----+!! Oozie Coordinator Specification
+# Oozie Coordinator Specification
The goal of this document is to define a coordinator engine system specialized in submitting workflows based on time and data triggers.
-%TOC%
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Changelog
+
+**03/JUL/2013**
+
+ * Appendix A, Added new coordinator schema 0.4, sla schema 0.2 and changed schemas ordering to newest first
----++ Changelog
----+++!! 03/JUL/2013
+**07/JAN/2013**
- * #Appendix A, Added new coordinator schema 0.4, sla schema 0.2 and changed schemas ordering to newest first
----+++!! 07/JAN/2013
+ * 6.8 Added section on new EL functions for datasets defined with HCatalog
- * #6.8 Added section on new EL functions for datasets defined with HCatalog
----+++!! 26/JUL/2012
+**26/JUL/2012**
- * #Appendix A, updated XML schema 0.4 to include =parameters= element
- * #6.5 Updated to mention about =parameters= element as of schema 0.4
----+++!! 23/NOV/2011:
+ * Appendix A, updated XML schema 0.4 to include `parameters` element
+ * 6.5 Updated to mention about `parameters` element as of schema 0.4
+
+**23/NOV/2011:**
* Update execution order typo
----+++!! 05/MAY/2011:
+
+**05/MAY/2011:**
* Update coordinator schema 0.2
----+++!! 09/MAR/2011:
+
+**09/MAR/2011:**
* Update coordinator status
----+++!! 02/DEC/2010:
+
+**02/DEC/2010:**
* Update coordinator done-flag
----+++!! 26/AUG/2010:
+
+**26/AUG/2010:**
* Update coordinator rerun
----+++!! 09/JUN/2010:
+
+**09/JUN/2010:**
* Clean up unsupported functions
----+++!! 02/JUN/2010:
+
+**02/JUN/2010:**
* Update all EL functions in CoordFunctionalSpec with "coord:" prefix
----+++!! 02/OCT/2009:
+
+**02/OCT/2009:**
* Added Appendix A, Oozie Coordinator XML-Schema
* Change #5.3., Datasets definition supports 'include' element
----+++!! 29/SEP/2009:
- * Change #4.4.1, added =${coord:endOfDays(int n)}= EL function
- * Change #4.4.2, added =${coord:endOfMonths(int n)}= EL function
----+++!! 11/SEP/2009:
+**29/SEP/2009:**
+
+ * Change #4.4.1, added `${coord:endOfDays(int n)}` EL function
+ * Change #4.4.2, added `${coord:endOfMonths(int n)}` EL function
+
+**11/SEP/2009:**
- * Change #6.6.4. =${coord:tzOffset()}= EL function now returns offset in minutes. Added more explanation on behavior
+ * Change #6.6.4. `${coord:tzOffset()}` EL function now returns offset in minutes. Added more explanation on behavior
* Removed 'oozie' URL from action workflow invocation, per arch review feedback coord&wf run on the same instance
----+++!! 07/SEP/2009:
+
+**07/SEP/2009:**
* Full rewrite of sections #4 and #7
* Added sections #6.1.7, #6.6.2, #6.6.3 & #6.6.4
* Rewording through the spec definitions
* Updated all examples and syntax to latest changes
----+++!! 03/SEP/2009:
+
+**03/SEP/2009:**
* Change #2. Definitions. Some rewording in the definitions
- * Change #6.6.4. Replaced =${coord:next(int n)}= with =${coord:version(int n)}= EL Function
+ * Change #6.6.4. Replaced `${coord:next(int n)}` with `${coord:version(int n)}` EL Function
* Added #6.6.5. Dataset Instance Resolution for Instances Before the Initial Instance
----++ 1. Coordinator Overview
+## 1. Coordinator Overview
-Users typically run map-reduce, hadoop-streaming, hdfs and/or Pig jobs on the grid. Multiple of these jobs can be combined to form a workflow job. [[https://issues.apache.org/jira/browse/HADOOP-5303][Oozie, Hadoop Workflow System]] defines a workflow system that runs such jobs.
+Users typically run map-reduce, hadoop-streaming, hdfs and/or Pig jobs on the grid. Multiple of these jobs can be combined to form a workflow job. [Oozie, Hadoop Workflow System](https://issues.apache.org/jira/browse/HADOOP-5303) defines a workflow system that runs such jobs.
Commonly, workflow jobs are run based on regular time intervals and/or data availability. And, in some cases, they can be triggered by an external event.
@@ -76,54 +91,54 @@ Expressing the condition(s) that trigger a workflow job can be modeled as a pred
It is also necessary to connect workflow jobs that run regularly, but at different time intervals. The outputs of multiple subsequent runs of a workflow become the input to the next workflow. For example, the outputs of last 4 runs of a workflow that runs every 15 minutes become the input of another workflow that runs every 60 minutes. Chaining together these workflows result it is referred as a data application pipeline.
-The Oozie *Coordinator* system allows the user to define and execute recurrent and interdependent workflow jobs (data application pipelines).
+The Oozie **Coordinator** system allows the user to define and execute recurrent and interdependent workflow jobs (data application pipelines).
Real world data application pipelines have to account for reprocessing, late processing, catchup, partial processing, monitoring, notification and SLAs.
This document defines the functional specification for the Oozie Coordinator system.
----++ 2. Definitions
+## 2. Definitions
-*Actual time:* The actual time indicates the time when something actually happens.
+**Actual time:** The actual time indicates the time when something actually happens.
-*Nominal time:* The nominal time specifies the time when something should happen. In theory the nominal time and the actual time should match, however, in practice due to delays the actual time may occur later than the nominal time.
+**Nominal time:** The nominal time specifies the time when something should happen. In theory the nominal time and the actual time should match, however, in practice due to delays the actual time may occur later than the nominal time.
-*Dataset:* Collection of data referred to by a logical name. A dataset normally has several instances of data and each
+**Dataset:** Collection of data referred to by a logical name. A dataset normally has several instances of data and each
one of them can be referred individually. Each dataset instance is represented by a unique set of URIs.
-*Synchronous Dataset:* Synchronous datasets instances are generated at fixed time intervals and there is a dataset
+**Synchronous Dataset:** Synchronous datasets instances are generated at fixed time intervals and there is a dataset
instance associated with each time interval. Synchronous dataset instances are identified by their nominal time.
For example, in the case of a HDFS based dataset, the nominal time would be somewhere in the file path of the
-dataset instance: hdfs://foo:8020/usr/logs/2009/04/15/23/30. In the case of HCatalog table partitions, the nominal time
-would be part of some partition values: hcat://bar:8020/mydb/mytable/year=2009;month=04;dt=15;region=us.
+dataset instance: `hdfs://foo:8020/usr/logs/2009/04/15/23/30`. In the case of HCatalog table partitions, the nominal time
+would be part of some partition values: `hcat://bar:8020/mydb/mytable/year=2009;month=04;dt=15;region=us`.
-*Coordinator Action:* A coordinator action is a workflow job that is started when a set of conditions are met (input dataset instances are available).
+**Coordinator Action:** A coordinator action is a workflow job that is started when a set of conditions are met (input dataset instances are available).
-*Coordinator Application:* A coordinator application defines the conditions under which coordinator actions should be created (the frequency) and when the actions can be started. The coordinator application also defines a start and an end time. Normally, coordinator applications are parameterized. A Coordinator application is written in XML.
+**Coordinator Application:** A coordinator application defines the conditions under which coordinator actions should be created (the frequency) and when the actions can be started. The coordinator application also defines a start and an end time. Normally, coordinator applications are parameterized. A Coordinator application is written in XML.
-*Coordinator Job:* A coordinator job is an executable instance of a coordination definition. A job submission is done by submitting a job configuration that resolves all parameters in the application definition.
+**Coordinator Job:** A coordinator job is an executable instance of a coordination definition. A job submission is done by submitting a job configuration that resolves all parameters in the application definition.
-*Data pipeline:* A data pipeline is a connected set of coordinator applications that consume and produce interdependent datasets.
+**Data pipeline:** A data pipeline is a connected set of coordinator applications that consume and produce interdependent datasets.
-*Coordinator Definition Language:* The language used to describe datasets and coordinator applications.
+**Coordinator Definition Language:** The language used to describe datasets and coordinator applications.
-*Coordinator Engine:* A system that executes coordinator jobs.
+**Coordinator Engine:** A system that executes coordinator jobs.
----++ 3. Expression Language for Parameterization
+## 3. Expression Language for Parameterization
Coordinator application definitions can be parameterized with variables, built-in constants and built-in functions.
At execution time all the parameters are resolved into concrete values.
-The parameterization of workflow definitions it done using JSP Expression Language syntax from the [[http://jcp.org/aboutJava/communityprocess/final/jsr152/][JSP 2.0 Specification (JSP.2.3)]], allowing not only to support variables as parameters but also functions and complex expressions.
+The parameterization of workflow definitions it done using JSP Expression Language syntax from the [JSP 2.0 Specification (JSP.2.3)](http://jcp.org/aboutJava/communityprocess/final/jsr152/index.html), allowing not only to support variables as parameters but also functions and complex expressions.
EL expressions can be used in XML attribute values and XML text element values. They cannot be used in XML element and XML attribute names.
Refer to section #6.5 'Parameterization of Coordinator Applications' for more details.
----++ 4. Datetime, Frequency and Time-Period Representation
+## 4. Datetime, Frequency and Time-Period Representation
-Oozie processes coordinator jobs in a fixed timezone with no DST (typically =UTC=), this timezone is referred as 'Oozie
+Oozie processes coordinator jobs in a fixed timezone with no DST (typically `UTC`), this timezone is referred as 'Oozie
processing timezone'.
The Oozie processing timezone is used to resolve coordinator jobs start/end times, job pause times and the initial-instance
@@ -131,42 +146,42 @@ of datasets. Also, all coordinator dataset instance URI templates are resolved t
time-zone.
All the datetimes used in coordinator applications and job parameters to coordinator applications must be specified
-in the Oozie processing timezone. If Oozie processing timezone is =UTC=, the qualifier is *Z*. If Oozie processing
-time zone is other than =UTC=, the qualifier must be the GMT offset, =(+/-)####=.
+in the Oozie processing timezone. If Oozie processing timezone is `UTC`, the qualifier is **Z**. If Oozie processing
+time zone is other than `UTC`, the qualifier must be the GMT offset, `(+/-)####`.
-For example, a datetime in =UTC= is =2012-08-12T00:00Z=, the same datetime in =GMT+5:30= is =2012-08-12T05:30+0530=.
+For example, a datetime in `UTC` is `2012-08-12T00:00Z`, the same datetime in `GMT+5:30` is `2012-08-12T05:30+0530`.
-For simplicity, the rest of this specification uses =UTC= datetimes.
+For simplicity, the rest of this specification uses `UTC` datetimes.
-#datetime
----+++ 4.1. Datetime
+<a name="datetime"></a>
+### 4.1. Datetime
-If the Oozie processing timezone is =UTC=, all datetime values are always in
-[[http://en.wikipedia.org/wiki/Coordinated_Universal_Time][UTC]] down to a minute precision, 'YYYY-MM-DDTHH:mmZ'.
+If the Oozie processing timezone is `UTC`, all datetime values are always in
+[UTC](http://en.wikipedia.org/wiki/Coordinated_Universal_Time) down to a minute precision, 'YYYY-MM-DDTHH:mmZ'.
-For example =2009-08-10T13:10Z= is August 10th 2009 at 13:10 UTC.
+For example `2009-08-10T13:10Z` is August 10th 2009 at 13:10 UTC.
-If the Oozie processing timezone is a GMT offset =GMT(+/-)####=, all datetime values are always in
-[[http://en.wikipedia.org/wiki/ISO_8601][ISO 8601]] in the corresponding GMT offset down to a minute precision,
+If the Oozie processing timezone is a GMT offset `GMT(+/-)####`, all datetime values are always in
+[ISO 8601](http://en.wikipedia.org/wiki/ISO_8601) in the corresponding GMT offset down to a minute precision,
'YYYY-MM-DDTHH:mmGMT(+/-)####'.
-For example =2009-08-10T13:10+0530= is August 10th 2009 at 13:10 GMT+0530, India timezone.
+For example `2009-08-10T13:10+0530` is August 10th 2009 at 13:10 GMT+0530, India timezone.
----++++ 4.1.1 End of the day in Datetime Values
+#### 4.1.1 End of the day in Datetime Values
-It is valid to express the end of day as a '24:00' hour (i.e. =2009-08-10T24:00Z=).
+It is valid to express the end of day as a '24:00' hour (i.e. `2009-08-10T24:00Z`).
However, for all calculations and display, Oozie resolves such dates as the zero hour of the following day
-(i.e. =2009-08-11T00:00Z=).
+(i.e. `2009-08-11T00:00Z`).
----+++ 4.2. Timezone Representation
+### 4.2. Timezone Representation
There is no widely accepted standard to identify timezones.
Oozie Coordinator will understand the following timezone identifiers:
- * Generic NON-DST timezone identifier: =GMT[+/-]##:##= (i.e.: GMT+05:30)
- * UTC timezone identifier: =UTC= (i.e.: 2009-06-06T00:00Z)
+ * Generic NON-DST timezone identifier: `GMT[+/-]##:##` (i.e.: GMT+05:30)
+ * UTC timezone identifier: `UTC` (i.e.: 2009-06-06T00:00Z)
* ZoneInfo identifiers, with DST support, understood by Java JDK (about 600 IDs) (i.e.: America/Los_Angeles)
Due to DST shift from PST to PDT, it is preferred that GMT, UTC or Region/City timezone notation is used in
@@ -175,9 +190,9 @@ at a DST shift. If used directly, PST will not handle DST shift when time is swi
Oozie Coordinator must provide a tool for developers to list all supported timezone identifiers.
----+++ 4.3. Timezones and Daylight-Saving
+### 4.3. Timezones and Daylight-Saving
-While Oozie coordinator engine works in a fixed timezone with no DST (typically =UTC=), it provides DST support for coordinator applications.
+While Oozie coordinator engine works in a fixed timezone with no DST (typically `UTC`), it provides DST support for coordinator applications.
The baseline datetime for datasets and coordinator applications are expressed in UTC. The baseline datetime is the time of the first occurrence.
@@ -189,85 +204,89 @@ The timezone indicator enables Oozie coordinator engine to properly compute freq
Section #7 'Handling Timezones and Daylight Saving Time' explains how coordinator applications can be written to handle timezones and daylight-saving-time properly.
----+++ 4.4. Frequency and Time-Period Representation
+### 4.4. Frequency and Time-Period Representation
Frequency is used to capture the periodic intervals at which datasets that are produced, and coordinator applications are scheduled to run.
This time periods representation is also used to specify non-recurrent time-periods, for example a timeout interval.
-For datasets and coordinator applications the frequency time-period is applied =N= times to the baseline datetime to compute recurrent times.
+For datasets and coordinator applications the frequency time-period is applied `N` times to the baseline datetime to compute recurrent times.
Frequency is always expressed in minutes.
-Because the number of minutes in day may vary for timezones that observe daylight saving time, constants cannot be use to express frequencies greater than a day for datasets and coordinator applications for such timezones. For such uses cases, Oozie coordinator provides 2 EL functions, =${coord:days(int n)}= and =${coord:months(int n)}=.
+Because the number of minutes in day may vary for timezones that observe daylight saving time, constants cannot be use to express frequencies greater than a day for datasets and coordinator applications for such timezones. For such uses cases, Oozie coordinator provides 2 EL functions, `${coord:days(int n)}` and `${coord:months(int n)}`.
Frequencies can be expressed using EL constants and EL functions that evaluate to an positive integer number.
Coordinator Frequencies can also be expressed using cron syntax.
-*%GREEN% Examples: %ENDCOLOR%*
+**<font color="#008000"> Examples: </font>**
-| *EL Constant* | *Value* | *Example* |
-| =${coord:minutes(int n)}= | _n_ | =${coord:minutes(45)}= --> =45= |
-| =${coord:hours(int n)}= | _n * 60_ | =${coord:hours(3)}= --> =180= |
-| =${coord:days(int n)}= | _variable_ | =${coord:days(2)}= --> minutes in 2 full days from the current date |
-| =${coord:months(int n)}= | _variable_ | =${coord:months(1)}= --> minutes in a 1 full month from the current date |
-| =${cron syntax}= | _variable_ | =${0,10 15 * * 2-6}= --> a job that runs every weekday at 3:00pm and 3:10pm UTC time|
+| **EL Constant** | **Value** | **Example** |
+| --- | --- | --- |
+| `${coord:minutes(int n)}` | _n_ | `${coord:minutes(45)}` --> `45` |
+| `${coord:hours(int n)}` | _n * 60_ | `${coord:hours(3)}` --> `180` |
+| `${coord:days(int n)}` | _variable_ | `${coord:days(2)}` --> minutes in 2 full days from the current date |
+| `${coord:months(int n)}` | _variable_ | `${coord:months(1)}` --> minutes in a 1 full month from the current date |
+| `${cron syntax}` | _variable_ | `${0,10 15 * * 2-6}` --> a job that runs every weekday at 3:00pm and 3:10pm UTC time|
-Note that, though =${coord:days(int n)}= and =${coord:months(int n)}= EL functions are used to calculate minutes precisely including
+Note that, though `${coord:days(int n)}` and `${coord:months(int n)}` EL functions are used to calculate minutes precisely including
variations due to daylight saving time for Frequency representation, when specified for coordinator timeout interval, one day is
calculated as 24 hours and one month is calculated as 30 days for simplicity.
----++++ 4.4.1. The coord:days(int n) and coord:endOfDays(int n) EL functions
+#### 4.4.1. The coord:days(int n) and coord:endOfDays(int n) EL functions
-The =${coord:days(int n)}= and =${coord:endOfDays(int n)}= EL functions should be used to handle day based frequencies.
+The `${coord:days(int n)}` and `${coord:endOfDays(int n)}` EL functions should be used to handle day based frequencies.
Constant values should not be used to indicate a day based frequency (every 1 day, every 1 week, etc) because the number of hours in
every day is not always the same for timezones that observe daylight-saving time.
-It is a good practice to use always these EL functions instead of using a constant expression (i.e. =24 * 60=) even if the timezone
+It is a good practice to use always these EL functions instead of using a constant expression (i.e. `24 * 60`) even if the timezone
for which the application is being written for does not support daylight saving time. This makes application foolproof to country
legislation changes and also makes applications portable across timezones.
----+++++ 4.4.1.1. The coord:days(int n) EL function
+##### 4.4.1.1. The coord:days(int n) EL function
-The =${coord:days(int n)}= EL function returns the number of minutes for 'n' complete days starting with the day of the specified nominal time for which the computation is being done.
+The `${coord:days(int n)}` EL function returns the number of minutes for 'n' complete days starting with the day of the specified nominal time for which the computation is being done.
-The =${coord:days(int n)}= EL function includes *all* the minutes of the current day, regardless of the time of the day of the current nominal time.
+The `${coord:days(int n)}` EL function includes **all** the minutes of the current day, regardless of the time of the day of the current nominal time.
-*%GREEN% Examples: %ENDCOLOR%*
+**<font color="#008000"> Examples: </font>**
-| *Starting Nominal UTC time* | *Timezone* | *Usage* | *Value* | *First Occurrence* | *Comments* |
-| =2009-01-01T08:00Z= | =UTC= | =${coord:days(1)}= | 1440 | =2009-01-01T08:00Z= | total minutes on 2009JAN01 UTC time |
-| =2009-01-01T08:00Z= | =America/Los_Angeles= | =${coord:days(1)}= | 1440 | =2009-01-01T08:00Z= | total minutes in 2009JAN01 PST8PDT time |
-| =2009-01-01T08:00Z= | =America/Los_Angeles= | =${coord:days(2)}= | 2880 | =2009-01-01T08:00Z= | total minutes in 2009JAN01 and 2009JAN02 PST8PDT time |
+| **Starting Nominal UTC time** | **Timezone** | **Usage** | **Value** | **First Occurrence** | **Comments** |
+| --- | --- | --- | --- | --- | --- |
+| `2009-01-01T08:00Z` | `UTC` | `${coord:days(1)}` | 1440 | `2009-01-01T08:00Z` | total minutes on 2009JAN01 UTC time |
+| `2009-01-01T08:00Z` | `America/Los_Angeles` | `${coord:days(1)}` | 1440 | `2009-01-01T08:00Z` | total minutes in 2009JAN01 PST8PDT time |
+| `2009-01-01T08:00Z` | `America/Los_Angeles` | `${coord:days(2)}` | 2880 | `2009-01-01T08:00Z` | total minutes in 2009JAN01 and 2009JAN02 PST8PDT time |
| |||||
-| =2009-03-08T08:00Z= | =UTC= | =${coord:days(1)}= | 1440 | =2009-03-08T08:00Z= | total minutes on 2009MAR08 UTC time |
-| =2009-03-08T08:00Z= | =Europe/London= | =${coord:days(1)}= | 1440 | =2009-03-08T08:00Z= | total minutes in 2009MAR08 BST1BDT time |
-| =2009-03-08T08:00Z= | =America/Los_Angeles= | =${coord:days(1)}= | 1380 | =2009-03-08T08:00Z= | total minutes in 2009MAR08 PST8PDT time <br/> (2009MAR08 is DST switch in the US) |
-| =2009-03-08T08:00Z= | =UTC= | =${coord:days(2)}= | 2880 | =2009-03-08T08:00Z= | total minutes in 2009MAR08 and 2009MAR09 UTC time |
-| =2009-03-08T08:00Z= | =America/Los_Angeles= | =${coord:days(2)}= | 2820 | =2009-03-08T08:00Z= | total minutes in 2009MAR08 and 2009MAR09 PST8PDT time <br/> (2009MAR08 is DST switch in the US) |
-| =2009-03-09T08:00Z= | =America/Los_Angeles= | =${coord:days(1)}= | 1440 | =2009-03-09T07:00Z= | total minutes in 2009MAR09 PST8PDT time <br/> (2009MAR08 is DST ON, frequency tick is earlier in UTC) |
+| `2009-03-08T08:00Z` | `UTC` | `${coord:days(1)}` | 1440 | `2009-03-08T08:00Z` | total minutes on 2009MAR08 UTC time |
+| `2009-03-08T08:00Z` | `Europe/London` | `${coord:days(1)}` | 1440 | `2009-03-08T08:00Z` | total minutes in 2009MAR08 BST1BDT time |
+| `2009-03-08T08:00Z` | `America/Los_Angeles` | `${coord:days(1)}` | 1380 | `2009-03-08T08:00Z` | total minutes in 2009MAR08 PST8PDT time <br/> (2009MAR08 is DST switch in the US) |
+| `2009-03-08T08:00Z` | `UTC` | `${coord:days(2)}` | 2880 | `2009-03-08T08:00Z` | total minutes in 2009MAR08 and 2009MAR09 UTC time |
+| `2009-03-08T08:00Z` | `America/Los_Angeles` | `${coord:days(2)}` | 2820 | `2009-03-08T08:00Z` | total minutes in 2009MAR08 and 2009MAR09 PST8PDT time <br/> (2009MAR08 is DST switch in the US) |
+| `2009-03-09T08:00Z` | `America/Los_Angeles` | `${coord:days(1)}` | 1440 | `2009-03-09T07:00Z` | total minutes in 2009MAR09 PST8PDT time <br/> (2009MAR08 is DST ON, frequency tick is earlier in UTC) |
-For all these examples, the first occurrence of the frequency will be at =08:00Z= (UTC time).
+For all these examples, the first occurrence of the frequency will be at `08:00Z` (UTC time).
----+++++ 4.4.1.2. The coord:endOfDays(int n) EL function
+##### 4.4.1.2. The coord:endOfDays(int n) EL function
-The =${coord:endOfDays(int n)}= EL function is identical to the =${coord:days(int n)}= except that it shifts the first occurrence to the end of the day for the specified timezone before computing the interval in minutes.
+The `${coord:endOfDays(int n)}` EL function is identical to the `${coord:days(int n)}` except that it shifts the first occurrence to the end of the day for the specified timezone before computing the interval in minutes.
-*%GREEN% Examples: %ENDCOLOR%*
+**<font color="#008000"> Examples: </font>**
-| *Starting Nominal UTC time* | *Timezone* | *Usage* | *Value* | *First Occurrence* | *Comments* |
-| =2009-01-01T08:00Z= | =UTC= | =${coord:endOfDays(1)}= | 1440 | =2009-01-02T00:00Z= | first occurrence in 2009JAN02 00:00 UTC time, <br/> first occurrence shifted to the end of the UTC day |
-| =2009-01-01T08:00Z= | =America/Los_Angeles= | =${coord:endOfDays(1)}= | 1440 | =2009-01-02T08:00Z= | first occurrence in 2009JAN02 08:00 UTC time, <br/> first occurrence shifted to the end of the PST8PDT day |
-| =2009-01-01T08:01Z= | =America/Los_Angeles= | =${coord:endOfDays(1)}= | 1440 | =2009-01-02T08:00Z= | first occurrence in 2009JAN02 08:00 UTC time, <br/> first occurrence shifted to the end of the PST8PDT day |
-| =2009-01-01T18:00Z= | =America/Los_Angeles= | =${coord:endOfDays(1)}= | 1440 | =2009-01-02T08:00Z= | first occurrence in 2009JAN02 08:00 UTC time, <br/> first occurrence shifted to the end of the PST8PDT day |
+| **Starting Nominal UTC time** | **Timezone** | **Usage** | **Value** | **First Occurrence** | **Comments** |
+| --- | --- | --- | --- | --- | --- |
+| `2009-01-01T08:00Z` | `UTC` | `${coord:endOfDays(1)}` | 1440 | `2009-01-02T00:00Z` | first occurrence in 2009JAN02 00:00 UTC time, <br/> first occurrence shifted to the end of the UTC day |
+| `2009-01-01T08:00Z` | `America/Los_Angeles` | `${coord:endOfDays(1)}` | 1440 | `2009-01-02T08:00Z` | first occurrence in 2009JAN02 08:00 UTC time, <br/> first occurrence shifted to the end of the PST8PDT day |
+| `2009-01-01T08:01Z` | `America/Los_Angeles` | `${coord:endOfDays(1)}` | 1440 | `2009-01-02T08:00Z` | first occurrence in 2009JAN02 08:00 UTC time, <br/> first occurrence shifted to the end of the PST8PDT day |
+| `2009-01-01T18:00Z` | `America/Los_Angeles` | `${coord:endOfDays(1)}` | 1440 | `2009-01-02T08:00Z` | first occurrence in 2009JAN02 08:00 UTC time, <br/> first occurrence shifted to the end of the PST8PDT day |
| |||||
-| =2009-03-07T09:00Z= | =America/Los_Angeles= | =${coord:endOfDays(1)}= | 1380 | =2009-03-08T08:00Z= | first occurrence in 2009MAR08 08:00 UTC time <br/> first occurrence shifted to the end of the PST8PDT day |
-| =2009-03-08T07:00Z= | =America/Los_Angeles= | =${coord:endOfDays(1)}= | 1440 | =2009-03-08T08:00Z= | first occurrence in 2009MAR08 08:00 UTC time <br/> first occurrence shifted to the end of the PST8PDT day |
-| =2009-03-09T07:00Z= | =America/Los_Angeles= | =${coord:endOfDays(1)}= | 1440 | =2009-03-10T07:00Z= | first occurrence in 2009MAR10 07:00 UTC time <br/> (2009MAR08 is DST switch in the US), <br/> first occurrence shifted to the end of the PST8PDT day |
+| `2009-03-07T09:00Z` | `America/Los_Angeles` | `${coord:endOfDays(1)}` | 1380 | `2009-03-08T08:00Z` | first occurrence in 2009MAR08 08:00 UTC time <br/> first occurrence shifted to the end of the PST8PDT day |
+| `2009-03-08T07:00Z` | `America/Los_Angeles` | `${coord:endOfDays(1)}` | 1440 | `2009-03-08T08:00Z` | first occurrence in 2009MAR08 08:00 UTC time <br/> first occurrence shifted to the end of the PST8PDT day |
+| `2009-03-09T07:00Z` | `America/Los_Angeles` | `${coord:endOfDays(1)}` | 1440 | `2009-03-10T07:00Z` | first occurrence in 2009MAR10 07:00 UTC time <br/> (2009MAR08 is DST switch in the US), <br/> first occurrence shifted to the end of the PST8PDT day |
-<verbatim>
+
+```
<coordinator-app name="hello-coord" frequency="${coord:days(1)}"
start="2009-01-02T08:00Z" end="2009-01-04T08:00Z" timezone="America/Los_Angeles"
xmlns="uri:oozie:coordinator:0.5">
@@ -317,48 +336,51 @@ The =${coord:endOfDays(int n)}= EL function is identical to the =${coord:days(in
</workflow>
</action>
</coordinator-app>
-</verbatim>
+```
----++++ 4.4.2. The coord:months(int n) and coord:endOfMonths(int n) EL functions
+#### 4.4.2. The coord:months(int n) and coord:endOfMonths(int n) EL functions
-The =${coord:months(int n)}= and =${coord:endOfMonths(int n)}= EL functions should be used to handle month based frequencies.
+The `${coord:months(int n)}` and `${coord:endOfMonths(int n)}` EL functions should be used to handle month based frequencies.
Constant values cannot be used to indicate a month based frequency because the number of days in a month changes month to month and on leap years; plus the number of hours in every day of the month are not always the same for timezones that observe daylight-saving time.
----+++++ 4.4.2.1. The coord:months(int n) EL function
+##### 4.4.2.1. The coord:months(int n) EL function
-The =${coord:months(int n)}= EL function returns the number of minutes for 'n' complete months starting with the month of the current nominal time for which the computation is being done.
+The `${coord:months(int n)}` EL function returns the number of minutes for 'n' complete months starting with the month of the current nominal time for which the computation is being done.
-The =${coord:months(int n)}= EL function includes *all* the minutes of the current month, regardless of the day of the month of the current nominal time.
+The `${coord:months(int n)}` EL function includes **all** the minutes of the current month, regardless of the day of the month of the current nominal time.
-*%GREEN% Examples: %ENDCOLOR%*
+**<font color="#008000"> Examples: </font>**
-| *Starting Nominal UTC time* | *Timezone* | *Usage* | *Value* | *First Occurrence* | *Comments* |
-| =2009-01-01T08:00Z= | =UTC= | =${coord:months(1)}= | 44640 | =2009-01-01T08:00Z= |total minutes for 2009JAN UTC time |
-| =2009-01-01T08:00Z= | =America/Los_Angeles= | =${coord:months(1)}= | 44640 | =2009-01-01T08:00Z= | total minutes in 2009JAN PST8PDT time |
-| =2009-01-01T08:00Z= | =America/Los_Angeles= | =${coord:months(2)}= | 84960 | =2009-01-01T08:00Z= | total minutes in 2009JAN and 2009FEB PST8PDT time |
+| **Starting Nominal UTC time** | **Timezone** | **Usage** | **Value** | **First Occurrence** | **Comments** |
+| --- | --- | --- | --- | --- | --- |
+| `2009-01-01T08:00Z` | `UTC` | `${coord:months(1)}` | 44640 | `2009-01-01T08:00Z` |total minutes for 2009JAN UTC time |
+| `2009-01-01T08:00Z` | `America/Los_Angeles` | `${coord:months(1)}` | 44640 | `2009-01-01T08:00Z` | total minutes in 2009JAN PST8PDT time |
+| `2009-01-01T08:00Z` | `America/Los_Angeles` | `${coord:months(2)}` | 84960 | `2009-01-01T08:00Z` | total minutes in 2009JAN and 2009FEB PST8PDT time |
| |||||
-| =2009-03-08T08:00Z= | =UTC= | =${coord:months(1)}= | 44640 | =2009-03-08T08:00Z= | total minutes on 2009MAR UTC time |
-| =2009-03-08T08:00Z= | =Europe/London= | =${coord:months(1)}= | 44580 | =2009-03-08T08:00Z= | total minutes in 2009MAR BST1BDT time <br/> (2009MAR29 is DST switch in Europe) |
-| =2009-03-08T08:00Z= | =America/Los_Angeles= | =${coord:months(1)}= | 44580 | =2009-03-08T08:00Z= | total minutes in 2009MAR PST8PDT time <br/> (2009MAR08 is DST switch in the US) |
-| =2009-03-08T08:00Z= | =UTC= | =${coord:months(2)}= | 87840 | =2009-03-08T08:00Z= | total minutes in 2009MAR and 2009APR UTC time |
-| =2009-03-08T08:00Z= | =America/Los_Angeles= | =${coord:months(2)}= | 87780 | =2009-03-08T08:00Z= | total minutes in 2009MAR and 2009APR PST8PDT time <br/> (2009MAR08 is DST switch in US) |
+| `2009-03-08T08:00Z` | `UTC` | `${coord:months(1)}` | 44640 | `2009-03-08T08:00Z` | total minutes on 2009MAR UTC time |
+| `2009-03-08T08:00Z` | `Europe/London` | `${coord:months(1)}` | 44580 | `2009-03-08T08:00Z` | total minutes in 2009MAR BST1BDT time <br/> (2009MAR29 is DST switch in Europe) |
+| `2009-03-08T08:00Z` | `America/Los_Angeles` | `${coord:months(1)}` | 44580 | `2009-03-08T08:00Z` | total minutes in 2009MAR PST8PDT time <br/> (2009MAR08 is DST switch in the US) |
+| `2009-03-08T08:00Z` | `UTC` | `${coord:months(2)}` | 87840 | `2009-03-08T08:00Z` | total minutes in 2009MAR and 2009APR UTC time |
+| `2009-03-08T08:00Z` | `America/Los_Angeles` | `${coord:months(2)}` | 87780 | `2009-03-08T08:00Z` | total minutes in 2009MAR and 2009APR PST8PDT time <br/> (2009MAR08 is DST switch in US) |
+
+##### 4.4.2.2. The coord:endOfMonths(int n) EL function
----+++++ 4.4.2.2. The coord:endOfMonths(int n) EL function
+The `${coord:endOfMonths(int n)}` EL function is identical to the `${coord:months(int n)}` except that it shifts the first occurrence to the end of the month for the specified timezone before computing the interval in minutes.
-The =${coord:endOfMonths(int n)}= EL function is identical to the =${coord:months(int n)}= except that it shifts the first occurrence to the end of the month for the specified timezone before computing the interval in minutes.
+**<font color="#008000"> Examples: </font>**
-*%GREEN% Examples: %ENDCOLOR%*
+| **Starting Nominal UTC time** | **Timezone** | **Usage** | **Value** | **First Occurrence** | **Comments** |
+| --- | --- | --- | --- | --- | --- |
+| `2009-01-01T00:00Z` | `UTC` | `${coord:endOfMonths(1)}` | 40320 | `2009-02-01T00:00Z` | first occurrence in 2009FEB 00:00 UTC time |
+| `2009-01-01T08:00Z` | `UTC` | `${coord:endOfMonths(1)}` | 40320 | `2009-02-01T00:00Z` | first occurrence in 2009FEB 00:00 UTC time |
+| `2009-01-31T08:00Z` | `UTC` | `${coord:endOfMonths(1)}` | 40320 | `2009-02-01T00:00Z` | first occurrence in 2009FEB 00:00 UTC time |
+| `2009-01-01T08:00Z` | `America/Los_Angeles` | `${coord:endOfMonths(1)}` | 40320 | `2009-02-01T08:00Z` | first occurrence in 2009FEB 08:00 UTC time |
+| `2009-02-02T08:00Z` | `America/Los_Angeles` | `${coord:endOfMonths(1)}` | 44580 | `2009-03-01T08:00Z` | first occurrence in 2009MAR 08:00 UTC time |
+| `2009-02-01T08:00Z` | `America/Los_Angeles` | `${coord:endOfMonths(1)}` | 44580 | `2009-03-01T08:00Z` | first occurrence in 2009MAR 08:00 UTC time |
-| *Starting Nominal UTC time* | *Timezone* | *Usage* | *Value* | *First Occurrence* | *Comments* |
-| =2009-01-01T00:00Z= | =UTC= | =${coord:endOfMonths(1)}= | 40320 | =2009-02-01T00:00Z= | first occurrence in 2009FEB 00:00 UTC time |
-| =2009-01-01T08:00Z= | =UTC= | =${coord:endOfMonths(1)}= | 40320 | =2009-02-01T00:00Z= | first occurrence in 2009FEB 00:00 UTC time |
-| =2009-01-31T08:00Z= | =UTC= | =${coord:endOfMonths(1)}= | 40320 | =2009-02-01T00:00Z= | first occurrence in 2009FEB 00:00 UTC time |
-| =2009-01-01T08:00Z= | =America/Los_Angeles= | =${coord:endOfMonths(1)}= | 40320 | =2009-02-01T08:00Z= | first occurrence in 2009FEB 08:00 UTC time |
-| =2009-02-02T08:00Z= | =America/Los_Angeles= | =${coord:endOfMonths(1)}= | 44580 | =2009-03-01T08:00Z= | first occurrence in 2009MAR 08:00 UTC time |
-| =2009-02-01T08:00Z= | =America/Los_Angeles= | =${coord:endOfMonths(1)}= | 44580 | =2009-03-01T08:00Z= | first occurrence in 2009MAR 08:00 UTC time |
-<verbatim>
+```
<coordinator-app name="hello-coord" frequency="${coord:months(1)}"
start="2009-01-02T08:00Z" end="2009-04-02T08:00Z" timezone="America/Los_Angeles"
xmlns="uri:oozie:coordinator:0.5">
@@ -408,25 +430,27 @@ The =${coord:endOfMonths(int n)}= EL function is identical to the =${coord:month
</workflow>
</action>
</coordinator-app>
-</verbatim>
+```
----++++ 4.4.3. The coord:endOfWeeks(int n) EL function
+#### 4.4.3. The coord:endOfWeeks(int n) EL function
-The =${coord:endOfWeeks(int n)}= EL function shifts the first occurrence to the start of the week for the specified
+The `${coord:endOfWeeks(int n)}` EL function shifts the first occurrence to the start of the week for the specified
timezone before computing the interval in minutes. The start of the week depends on the Java's implementation of
-[[https://docs.oracle.com/javase/8/docs/api/java/util/Calendar.html#getFirstDayOfWeek--][Calendar.getFirstDayOfWeek()]]
+[Calendar.getFirstDayOfWeek()](https://docs.oracle.com/javase/8/docs/api/java/util/Calendar.html#getFirstDayOfWeek--)
i.e. first day of the week is SUNDAY in the U.S., MONDAY in France.
-*%GREEN% Examples: %ENDCOLOR%*
+**<font color="#008000"> Examples: </font>**
-| *Starting Nominal UTC time* | *Timezone* | *Usage* | *Value* | *First Occurrence* | *Comments* |
-| =2017-01-04T00:00Z= | =UTC= | =${coord:endOfWeeks(1)}= | 10080 | =2017-01-08T00:00Z= | first occurrence on 2017JAN08 08:00 UTC time |
-| =2017-01-04T08:00Z= | =UTC= | =${coord:endOfWeeks(1)}= | 10080 | =2017-01-08T08:00Z= | first occurrence on 2017JAN08 08:00 UTC time |
-| =2017-01-06T08:00Z= | =UTC= | =${coord:endOfWeeks(1)}= | 10080 | =2017-01-08T08:00Z= | first occurrence on 2017JAN08 08:00 UTC time |
-| =2017-01-04T08:00Z= | =America/Los_Angeles= | =${coord:endOfWeeks(1)}= | 10080 | =2017-01-08T08:00Z= | first occurrence in 2017JAN08 08:00 UTC time |
-| =2017-01-06T08:00Z= | =America/Los_Angeles= | =${coord:endOfWeeks(1)}= | 10080 | =2017-01-08T08:00Z= | first occurrence in 2017JAN08 08:00 UTC time |
+| **Starting Nominal UTC time** | **Timezone** | **Usage** | **Value** | **First Occurrence** | **Comments** |
+| --- | --- | --- | --- | --- | --- |
+| `2017-01-04T00:00Z` | `UTC` | `${coord:endOfWeeks(1)}` | 10080 | `2017-01-08T00:00Z` | first occurrence on 2017JAN08 08:00 UTC time |
+| `2017-01-04T08:00Z` | `UTC` | `${coord:endOfWeeks(1)}` | 10080 | `2017-01-08T08:00Z` | first occurrence on 2017JAN08 08:00 UTC time |
+| `2017-01-06T08:00Z` | `UTC` | `${coord:endOfWeeks(1)}` | 10080 | `2017-01-08T08:00Z` | first occurrence on 2017JAN08 08:00 UTC time |
+| `2017-01-04T08:00Z` | `America/Los_Angeles` | `${coord:endOfWeeks(1)}` | 10080 | `2017-01-08T08:00Z` | first occurrence in 2017JAN08 08:00 UTC time |
+| `2017-01-06T08:00Z` | `America/Los_Angeles` | `${coord:endOfWeeks(1)}` | 10080 | `2017-01-08T08:00Z` | first occurrence in 2017JAN08 08:00 UTC time |
-<verbatim>
+
+```
<coordinator-app name="hello-coord" frequency="${coord:endOfWeeks(1)}"
start="2017-01-04T08:00Z" end="2017-12-31T08:00Z" timezone="America/Los_Angeles"
xmlns="uri:oozie:coordinator:0.5">
@@ -476,9 +500,9 @@ timezone before computing the interval in minutes. The start of the week depends
</workflow>
</action>
</coordinator-app>
-</verbatim>
+```
----++++ 4.4.4. Cron syntax in coordinator frequency
+#### 4.4.4. Cron syntax in coordinator frequency
Oozie has historically allowed only very basic forms of scheduling: You could choose
to run jobs separated by a certain number of minutes, hours, days or weeks. That's
@@ -497,8 +521,9 @@ Cron is a standard time-based job scheduling mechanism in unix-like operating sy
administrators to setup jobs and maintain software environment. Cron syntax generally consists of five fields, minutes,
hours, date of month, month, and day of week respectively although multiple variations do exist.
-<verbatim>
-<coordinator-app name="cron-coord" frequency="0/10 1/2 * * *" start="${start}" end="${end}" timezone="UTC"
+
+```
+<coordinator-app name="cron-coord" frequency="0/10 1/2 ** ** *" start="${start}" end="${end}" timezone="UTC"
xmlns="uri:oozie:coordinator:0.2">
<action>
<workflow>
@@ -520,18 +545,19 @@ hours, date of month, month, and day of week respectively although multiple vari
</workflow>
</action>
</coordinator-app>
-</verbatim>
+```
Cron expressions are comprised of 5 required fields. The fields respectively are described as follows:
-| *Field name* | *Allowed Values* | *Allowed Special Characters* |
-| =Minutes= | =0-59= | , - * / |
-| =Hours= | =0-23= | , - * / |
-| =Day-of-month= | =1-31= | , - * ? / L W |
-| =Month= | =1-12 or JAN-DEC= | , - * / |
-| =Day-of-Week= | =1-7 or SUN-SAT= | , - * ? / L #|
+| **Field name** | **Allowed Values** | **Allowed Special Characters** |
+| --- | --- | --- |
+| `Minutes` | `0-59` | , - * / |
+| `Hours` | `0-23` | , - * / |
+| `Day-of-month` | `1-31` | , - * ? / L W |
+| `Month` | `1-12 or JAN-DEC` | , - * / |
+| `Day-of-Week` | `1-7 or SUN-SAT` | , - * ? / L #|
-The '*' character is used to specify all values. For example, "*" in the minute field means "every minute".
+The '**' character is used to specify all values. For example, "**" in the minute field means "every minute".
The '?' character is allowed for the day-of-month and day-of-week fields. It is used to specify 'no specific value'.
This is useful when you need to specify something in one of the two fields, but not the other.
@@ -585,21 +611,22 @@ The legal characters and the names of months and days of the week are not case s
If a user specifies an invalid cron syntax to run something on Feb, 30th for example: "0 10 30 2 *", the coordinator job
will not be created and an invalid coordinator frequency parse exception will be thrown.
-If a user has a coordinator job that materializes no action during run time, for example: frequency of "0 10 * * *" with
+If a user has a coordinator job that materializes no action during run time, for example: frequency of "0 10 ** ** *" with
start time of 2013-10-18T21:00Z and end time of 2013-10-18T22:00Z, the coordinator job submission will be rejected and
an invalid coordinator attribute exception will be thrown.
-*%GREEN% Examples: %ENDCOLOR%*
+**<font color="#008000"> Examples: </font>**
-| *Cron Expression* | *Meaning* |
-| 10 9 * * * | Runs everyday at 9:10am |
-| 10,30,45 9 * * * | Runs everyday at 9:10am, 9:30am, and 9:45am |
-| =0 * 30 JAN 2-6= | Runs at 0 minute of every hour on weekdays and 30th of January |
-| =0/20 9-17 * * 2-5= | Runs every Mon, Tue, Wed, and Thurs at minutes 0, 20, 40 from 9am to 5pm |
-| 1 2 L-3 * * | Runs every third-to-last day of month at 2:01am |
-| =1 2 6W 3 ?= | Runs on the nearest weekday to March, 6th every year at 2:01am |
-| =1 2 * 3 3#2= | Runs every second Tuesday of March at 2:01am every year |
-| =0 10,13 * * MON-FRI= | Runs every weekday at 10am and 1pm |
+| **Cron Expression** | **Meaning** |
+| --- | --- |
+| 10 9 ** ** * | Runs everyday at 9:10am |
+| 10,30,45 9 ** ** * | Runs everyday at 9:10am, 9:30am, and 9:45am |
+| `0 * 30 JAN 2-6` | Runs at 0 minute of every hour on weekdays and 30th of January |
+| `0/20 9-17 ** ** 2-5` | Runs every Mon, Tue, Wed, and Thurs at minutes 0, 20, 40 from 9am to 5pm |
+| 1 2 L-3 ** ** | Runs every third-to-last day of month at 2:01am |
+| `1 2 6W 3 ?` | Runs on the nearest weekday to March, 6th every year at 2:01am |
+| `1 2 * 3 3#2` | Runs every second Tuesday of March at 2:01am every year |
+| `0 10,13 ** ** MON-FRI` | Runs every weekday at 10am and 1pm |
NOTES:
@@ -619,7 +646,7 @@ NOTES:
no effort has been made to determine which interpretation CronExpression chooses.
An example would be "0 14-6 ? * FRI-MON".
----++ 5. Dataset
+## 5. Dataset
A dataset is a collection of data referred to by a logical name.
@@ -631,7 +658,7 @@ A dataset is a synchronous (produced at regular time intervals, it has an expect
A dataset instance is considered to be immutable while it is being consumed by coordinator jobs.
----+++ 5.1. Synchronous Datasets
+### 5.1. Synchronous Datasets
Instances of synchronous datasets are produced at regular time intervals, at an expected frequency. They are also referred to as "clocked datasets".
@@ -639,175 +666,190 @@ Synchronous dataset instances are identified by their nominal creation time. The
A synchronous dataset definition contains the following information:
- * *%BLUE% name: %ENDCOLOR%* The dataset name. It must be a valid Java identifier.
- * *%BLUE% frequency: %ENDCOLOR%* It represents the rate, in minutes at which data is _periodically_ created. The granularity is in minutes and can be expressed using EL expressions, for example: ${5 * HOUR}.
- * *%BLUE% initial-instance: %ENDCOLOR%* The UTC datetime of the initial instance of the dataset. The initial-instance also provides the baseline datetime to compute instances of the dataset using multiples of the frequency.
- * *%BLUE% timezone:%ENDCOLOR%* The timezone of the dataset.
- * *%BLUE% uri-template:%ENDCOLOR%* The URI template that identifies the dataset and can be resolved into concrete URIs to identify a particular dataset instance. The URI template is constructed using:
- * *%BLUE% constants %ENDCOLOR%* See the allowable EL Time Constants below. Ex: ${YEAR}/${MONTH}.
- * *%BLUE% variables %ENDCOLOR%* Variables must be resolved at the time a coordinator job is submitted to the coordinator engine. They are normally provided a job parameters (configuration properties). Ex: ${market}/${language}
- * *%BLUE% done-flag:%ENDCOLOR%* This flag denotes when a dataset instance is ready to be consumed.
+ * **<font color="#0000ff"> name: </font>** The dataset name. It must be a valid Java identifier.
+ * **<font color="#0000ff"> frequency: </font>*** It represents the rate, in minutes at which data is _periodically_ created. The granularity is in minutes and can be expressed using EL expressions, for example: ${5 ** HOUR}.
+ * **<font color="#0000ff"> initial-instance: </font>** The UTC datetime of the initial instance of the dataset. The initial-instance also provides the baseline datetime to compute instances of the dataset using multiples of the frequency.
+ * **<font color="#0000ff"> timezone:</font>** The timezone of the dataset.
+ * **<font color="#0000ff"> uri-template:</font>** The URI template that identifies the dataset and can be resolved into concrete URIs to identify a particular dataset instance. The URI template is constructed using:
+ * **<font color="#0000ff"> constants </font>** See the allowable EL Time Constants below. Ex: ${YEAR}/${MONTH}.
+ * **<font color="#0000ff"> variables </font>** Variables must be resolved at the time a coordinator job is submitted to the coordinator engine. They are normally provided a job parameters (configuration properties). Ex: ${market}/${language}
+ * **<font color="#0000ff"> done-flag:</font>** This flag denotes when a dataset instance is ready to be consumed.
* If the done-flag is omitted the coordinator will wait for the presence of a _SUCCESS file in the directory (Note: MapReduce jobs create this on successful completion automatically).
* If the done-flag is present but empty, then the existence of the directory itself indicates that the dataset is ready.
* If the done-flag is present but non-empty, Oozie will check for the presence of the named file within the directory, and will be considered ready (done) when the file exists.
The following EL constants can be used within synchronous dataset URI templates:
-| *EL Constant* | *Resulting Format* | *Comments* |
-| =YEAR= | _YYYY_ | 4 digits representing the year |
-| =MONTH= | _MM_ | 2 digits representing the month of the year, January = 1 |
-| =DAY= | _DD_ | 2 digits representing the day of the month |
-| =HOUR= | _HH_ | 2 digits representing the hour of the day, in 24 hour format, 0 - 23 |
-| =MINUTE= | _mm_ | 2 digits representing the minute of the hour, 0 - 59 |
+| **EL Constant** | **Resulting Format** | **Comments** |
+| --- | --- | --- |
+| `YEAR` | _YYYY_ | 4 digits representing the year |
+| `MONTH` | _MM_ | 2 digits representing the month of the year, January = 1 |
+| `DAY` | _DD_ | 2 digits representing the day of the month |
+| `HOUR` | _HH_ | 2 digits representing the hour of the day, in 24 hour format, 0 - 23 |
+| `MINUTE` | _mm_ | 2 digits representing the minute of the hour, 0 - 59 |
+
+**<font color="#800080">Syntax: </font>**
-*%PURPLE% Syntax: %ENDCOLOR%*
-<verbatim>
+```
<dataset name="[NAME]" frequency="[FREQUENCY]"
initial-instance="[DATETIME]" timezone="[TIMEZONE]">
<uri-template>[URI TEMPLATE]</uri-template>
<done-flag>[FILE NAME]</done-flag>
</dataset>
-</verbatim>
+```
IMPORTANT: The values of the EL constants in the dataset URIs (in HDFS) are expected in UTC. Oozie Coordinator takes care of the timezone conversion when performing calculations.
-*%GREEN% Examples: %ENDCOLOR%*
+**<font color="#008000"> Examples: </font>**
-1. *A dataset produced once every day at 00:15 PST8PDT and done-flag is set to empty:*
+1. **A dataset produced once every day at 00:15 PST8PDT and done-flag is set to empty:**
-<verbatim>
- <dataset name="logs" frequency="${coord:days(1)}"
- initial-instance="2009-02-15T08:15Z" timezone="America/Los_Angeles">
- <uri-template>
- hdfs://foo:8020/app/logs/${market}/${YEAR}${MONTH}/${DAY}/data
- </uri-template>
- <done-flag></done-flag>
- </dataset>
-</verbatim>
+ ```
+ <dataset name="logs" frequency="${coord:days(1)}"
+ initial-instance="2009-02-15T08:15Z" timezone="America/Los_Angeles">
+ <uri-template>
+ hdfs://foo:8020/app/logs/${market}/${YEAR}${MONTH}/${DAY}/data
+ </uri-template>
+ <done-flag></done-flag>
+ </dataset>
+ ```
-The dataset would resolve to the following URIs and Coordinator looks for the existence of the directory itself:
-<verbatim>
- [market] will be replaced with user given property.
+ The dataset would resolve to the following URIs and Coordinator looks for the existence of the directory itself:
- hdfs://foo:8020/usr/app/[market]/2009/02/15/data
- hdfs://foo:8020/usr/app/[market]/2009/02/16/data
- hdfs://foo:8020/usr/app/[market]/2009/02/17/data
- ...
-</verbatim>
+ ```
+ [market] will be replaced with user given property.
-2. *A dataset available on the 10th of each month and done-flag is default '_SUCCESS':*
+ hdfs://foo:8020/usr/app/[market]/2009/02/15/data
+ hdfs://foo:8020/usr/app/[market]/2009/02/16/data
+ hdfs://foo:8020/usr/app/[market]/2009/02/17/data
+ ...
+ ```
-<verbatim>
- <dataset name="stats" frequency="${coord:months(1)}"
- initial-instance="2009-01-10T10:00Z" timezone="America/Los_Angeles">
- <uri-template>hdfs://foo:8020/usr/app/stats/${YEAR}/${MONTH}/data</uri-template>
- </dataset>
-</verbatim>
-The dataset would resolve to the following URIs:
+2. **A dataset available on the 10th of each month and done-flag is default '_SUCCESS':**
-<verbatim>
- hdfs://foo:8020/usr/app/stats/2009/01/data
- hdfs://foo:8020/usr/app/stats/2009/02/data
- hdfs://foo:8020/usr/app/stats/2009/03/data
- ...
-</verbatim>
-The dataset instances are not ready until '_SUCCESS' exists in each path:
+ ```
+ <dataset name="stats" frequency="${coord:months(1)}"
+ initial-instance="2009-01-10T10:00Z" timezone="America/Los_Angeles">
+ <uri-template>hdfs://foo:8020/usr/app/stats/${YEAR}/${MONTH}/data</uri-template>
+ </dataset>
+ ```
-<verbatim>
- hdfs://foo:8020/usr/app/stats/2009/01/data/_SUCCESS
- hdfs://foo:8020/usr/app/stats/2009/02/data/_SUCCESS
- hdfs://foo:8020/usr/app/stats/2009/03/data/_SUCCESS
- ...
-</verbatim>
+ The dataset would resolve to the following URIs:
-3. *A dataset available at the end of every quarter and done-flag is 'trigger.dat':*
+ ```
+ hdfs://foo:8020/usr/app/stats/2009/01/data
+ hdfs://foo:8020/usr/app/stats/2009/02/data
+ hdfs://foo:8020/usr/app/stats/2009/03/data
+ ...
+ ```
-<verbatim>
- <dataset name="stats" frequency="${coord:months(3)}"
- initial-instance="2009-01-31T20:00Z" timezone="America/Los_Angeles">
- <uri-template>
- hdfs://foo:8020/usr/app/stats/${YEAR}/${MONTH}/data
- </uri-template>
- <done-flag>trigger.dat</done-flag>
- </dataset>
-</verbatim>
+ The dataset instances are not ready until '_SUCCESS' exists in each path:
-The dataset would resolve to the following URIs:
-<verbatim>
- hdfs://foo:8020/usr/app/stats/2009/01/data
- hdfs://foo:8020/usr/app/stats/2009/04/data
- hdfs://foo:8020/usr/app/stats/2009/07/data
- ...
-</verbatim>
+ ```
+ hdfs://foo:8020/usr/app/stats/2009/01/data/_SUCCESS
+ hdfs://foo:8020/usr/app/stats/2009/02/data/_SUCCESS
+ hdfs://foo:8020/usr/app/stats/2009/03/data/_SUCCESS
+ ...
+ ```
-The dataset instances are not ready until 'trigger.dat' exists in each path:
-<verbatim>
- hdfs://foo:8020/usr/app/stats/2009/01/data/trigger.dat
- hdfs://foo:8020/usr/app/stats/2009/04/data/trigger.dat
- hdfs://foo:8020/usr/app/stats/2009/07/data/trigger.dat
- ...
-</verbatim>
+3. **A dataset available at the end of every quarter and done-flag is 'trigger.dat':**
-4. *Normally the URI template of a dataset has a precision similar to the frequency:*
+ ```
+ <dataset name="stats" frequency="${coord:months(3)}"
+ initial-instance="2009-01-31T20:00Z" timezone="America/Los_Angeles">
+ <uri-template>
+ hdfs://foo:8020/usr/app/stats/${YEAR}/${MONTH}/data
+ </uri-template>
+ <done-flag>trigger.dat</done-flag>
+ </dataset>
+ ```
-<verbatim>
- <dataset name="logs" frequency="${coord:days(1)}"
- initial-instance="2009-01-01T10:30Z" timezone="America/Los_Angeles">
- <uri-template>
- hdfs://foo:8020/usr/app/logs/${YEAR}/${MONTH}/${DAY}/data
- </uri-template>
- </dataset>
-</verbatim>
+ The dataset would resolve to the following URIs:
-The dataset would resolve to the following URIs:
-<verbatim>
- hdfs://foo:8020/usr/app/logs/2009/01/01/data
- hdfs://foo:8020/usr/app/logs/2009/01/02/data
- hdfs://foo:8020/usr/app/logs/2009/01/03/data
- ...
-</verbatim>
+ ```
+ hdfs://foo:8020/usr/app/stats/2009/01/data
+ hdfs://foo:8020/usr/app/stats/2009/04/data
+ hdfs://foo:8020/usr/app/stats/2009/07/data
+ ...
+ ```
-5. *However, if the URI template has a finer precision than the dataset frequency:*
+ The dataset instances are not ready until 'trigger.dat' exists in each path:
-<verbatim>
- <dataset name="logs" frequency="${coord:days(1)}"
- initial-instance="2009-01-01T10:30Z" timezone="America/Los_Angeles">
- <uri-template>
- hdfs://foo:8020/usr/app/logs/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}/data
- </uri-template>
- </dataset>
-</verbatim>
-The dataset resolves to the following URIs with fixed values for the finer precision template variables:
+ ```
+ hdfs://foo:8020/usr/app/stats/2009/01/data/trigger.dat
+ hdfs://foo:8020/usr/app/stats/2009/04/data/trigger.dat
+ hdfs://foo:8020/usr/app/stats/2009/07/data/trigger.dat
+ ...
+ ```
-<verbatim>
- hdfs://foo:8020/usr/app/logs/2009/01/01/10/30/data
- hdfs://foo:8020/usr/app/logs/2009/01/02/10/30/data
- hdfs://foo:8020/usr/app/logs/2009/01/03/10/30/data
- ...
-</verbatim>
----+++ 5.2. Dataset URI-Template types
+4. **Normally the URI template of a dataset has a precision similar to the frequency:**
+
-Each dataset URI could be a HDFS path URI denoting a HDFS directory: hdfs://foo:8020/usr/logs/20090415 or a
-HCatalog partition URI identifying a set of table partitions: hcat://bar:8020/logsDB/logsTable/dt=20090415;region=US.
+ ```
+ <dataset name="logs" frequency="${coord:days(1)}"
+ initial-instance="2009-01-01T10:30Z" timezone="America/Los_Angeles">
+ <uri-template>
+ hdfs://foo:8020/usr/app/logs/${YEAR}/${MONTH}/${DAY}/data
+ </uri-template>
+ </dataset>
+ ```
+
+ The dataset would resolve to the following URIs:
+
+
+ ```
+ hdfs://foo:8020/usr/app/logs/2009/01/01/data
+ hdfs://foo:8020/usr/app/logs/2009/01/02/data
+ hdfs://foo:8020/usr/app/logs/2009/01/03/data
+ ...
+ ```
+
+5. **However, if the URI template has a finer precision than the dataset frequency:**
+
+
+ ```
+ <dataset name="logs" frequency="${coord:days(1)}"
+ initial-instance="2009-01-01T10:30Z" timezone="America/Los_Angeles">
+ <uri-template>
+ hdfs://foo:8020/usr/app/logs/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}/data
+ </uri-template>
+ </dataset>
+ ```
+
+ The dataset resolves to the following URIs with fixed values for the finer precision template variables:
+
+
+ ```
+ hdfs://foo:8020/usr/app/logs/2009/01/01/10/30/data
+ hdfs://foo:8020/usr/app/logs/2009/01/02/10/30/data
+ hdfs://foo:8020/usr/app/logs/2009/01/03/10/30/data
+ ...
+ ```
+
+### 5.2. Dataset URI-Template types
+
+Each dataset URI could be a HDFS path URI denoting a HDFS directory: `hdfs://foo:8020/usr/logs/20090415` or a
+HCatalog partition URI identifying a set of table partitions: `hcat://bar:8020/logsDB/logsTable/dt=20090415;region=US`.
HCatalog enables table and storage management for Pig, Hive and MapReduce. The format to specify a HCatalog table partition URI is
-hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value];...
+`hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value];...`
For example,
-<verbatim>
+
+```
<dataset name="logs" frequency="${coord:days(1)}"
initial-instance="2009-02-15T08:15Z" timezone="America/Los_Angeles">
<uri-template>
@@ -815,19 +857,20 @@ For example,
</uri-template>
<done-flag></done-flag>
</dataset>
-</verbatim>
+```
----+++ 5.3. Asynchronous Datasets
+### 5.3. Asynchronous Datasets
* TBD
----+++ 5.4. Dataset Definitions
+### 5.4. Dataset Definitions
Dataset definitions are grouped in XML files.
-*IMPORTANT:* Please note that if an XML namespace version is specified for the coordinator-app element in the coordinator.xml file, no namespace needs to be defined separately for the datasets element (even if the dataset is defined in a separate file). Specifying it at multiple places might result in xml errors while submitting the coordinator job.
+**IMPORTANT:** Please note that if an XML namespace version is specified for the coordinator-app element in the coordinator.xml file, no namespace needs to be defined separately for the datasets element (even if the dataset is defined in a separate file). Specifying it at multiple places might result in xml errors while submitting the coordinator job.
-*%PURPLE% Syntax: %ENDCOLOR%*
+**<font color="#800080">Syntax: </font>**
-<verbatim>
+
+```
<!-- Synchronous datasets -->
<datasets>
<include>[SHARED_DATASETS]</include>
@@ -838,11 +881,12 @@ Dataset definitions are grouped in XML files.
</dataset>
...
</datasets>
-</verbatim>
+```
+
+**<font color="#008000"> Example: </font>**
-*%GREEN% Example: %ENDCOLOR%*
-<verbatim>
+```
<datasets>
.
<include>hdfs://foo:8020/app/dataset-definitions/globallogs.xml</include>
@@ -860,74 +904,74 @@ Dataset definitions are grouped in XML files.
</dataset>
.
</datasets>
-</verbatim>
+```
----++ 6. Coordinator Application
+## 6. Coordinator Application
----+++ 6.1. Concepts
+### 6.1. Concepts
----++++ 6.1.1. Coordinator Application
+#### 6.1.1. Coordinator Application
A coordinator application is a program that triggers actions (commonly workflow jobs) when a set of conditions are met. Conditions can be a time frequency, the availability of new dataset instances or other external events.
Types of coordinator applications:
- * *Synchronous:* Its coordinator actions are created at specified time intervals.
+ * **Synchronous:** Its coordinator actions are created at specified time intervals.
Coordinator applications are normally parameterized.
----++++ 6.1.2. Coordinator Job
+#### 6.1.2. Coordinator Job
To create a coordinator job, a job configuration that resolves all coordinator application parameters must be provided to the coordinator engine.
A coordinator job is a running instance of a coordinator application running from a start time to an end time. The start
time must be earlier than the end time.
-At any time, a coordinator job is in one of the following status: *PREP, RUNNING, RUNNINGWITHERROR, PREPSUSPENDED, SUSPENDED, SUSPENDEDWITHERROR, PREPPAUSED, PAUSED, PAUSEDWITHERROR, SUCCEEDED, DONEWITHERROR, KILLED, FAILED*.
+At any time, a coordinator job is in one of the following status: **PREP, RUNNING, RUNNINGWITHERROR, PREPSUSPENDED, SUSPENDED, SUSPENDEDWITHERROR, PREPPAUSED, PAUSED, PAUSEDWITHERROR, SUCCEEDED, DONEWITHERROR, KILLED, FAILED**.
Valid coordinator job status transitions are:
- * *PREP --> PREPSUSPENDED | PREPPAUSED | RUNNING | KILLED*
- * *RUNNING --> RUNNINGWITHERROR | SUSPENDED | PAUSED | SUCCEEDED | KILLED*
- * *RUNNINGWITHERROR --> RUNNING | SUSPENDEDWITHERROR | PAUSEDWITHERROR | DONEWITHERROR | KILLED | FAILED*
- * *PREPSUSPENDED --> PREP | KILLED*
- * *SUSPENDED --> RUNNING | KILLED*
- * *SUSPENDEDWITHERROR --> RUNNINGWITHERROR | KILLED*
- * *PREPPAUSED --> PREP | KILLED*
- * *PAUSED --> SUSPENDED | RUNNING | KILLED*
- * *PAUSEDWITHERROR --> SUSPENDEDWITHERROR | RUNNINGWITHERROR | KILLED*
- * *FAILED | KILLED --> IGNORED*
- * *IGNORED --> RUNNING*
+ * **PREP --> PREPSUSPENDED | PREPPAUSED | RUNNING | KILLED**
+ * **RUNNING --> RUNNINGWITHERROR | SUSPENDED | PAUSED | SUCCEEDED | KILLED**
+ * **RUNNINGWITHERROR --> RUNNING | SUSPENDEDWITHERROR | PAUSEDWITHERROR | DONEWITHERROR | KILLED | FAILED**
+ * **PREPSUSPENDED --> PREP | KILLED**
+ * **SUSPENDED --> RUNNING | KILLED**
+ * **SUSPENDEDWITHERROR --> RUNNINGWITHERROR | KILLED**
+ * **PREPPAUSED --> PREP | KILLED**
+ * **PAUSED --> SUSPENDED | RUNNING | KILLED**
+ * **PAUSEDWITHERROR --> SUSPENDEDWITHERROR | RUNNINGWITHERROR | KILLED**
+ * **FAILED | KILLED --> IGNORED**
+ * **IGNORED --> RUNNING**
-When a coordinator job is submitted, oozie parses the coordinator job XML. Oozie then creates a record for the coordinator with status *PREP* and returns a unique ID. The coordinator is also started immediately if pause time is not set.
+When a coordinator job is submitted, oozie parses the coordinator job XML. Oozie then creates a record for the coordinator with status **PREP** and returns a unique ID. The coordinator is also started immediately if pause time is not set.
-When a user requests to suspend a coordinator job that is in *PREP* state, oozie puts the job in status *PREPSUSPENDED*. Similarly, when pause time reaches for a coordinator job with *PREP* status, oozie puts the job in status *PREPPAUSED*.
+When a user requests to suspend a coordinator job that is in **PREP** state, oozie puts the job in status **PREPSUSPENDED**. Similarly, when pause time reaches for a coordinator job with **PREP** status, oozie puts the job in status **PREPPAUSED**.
-Conversely, when a user requests to resume a *PREPSUSPENDED* coordinator job, oozie puts the job in status *PREP*. And when pause time is reset for a coordinator job and job status is *PREPPAUSED*, oozie puts the job in status *PREP*.
+Conversely, when a user requests to resume a **PREPSUSPENDED** coordinator job, oozie puts the job in status **PREP**. And when pause time is reset for a coordinator job and job status is **PREPPAUSED**, oozie puts the job in status **PREP**.
-When a coordinator job starts, oozie puts the job in status *RUNNING* and start materializing workflow jobs based on job frequency. If any workflow job goes to *FAILED/KILLED/TIMEDOUT* state, the coordinator job is put in *RUNNINGWITHERROR*
+When a coordinator job starts, oozie puts the job in status **RUNNING** and start materializing workflow jobs based on job frequency. If any workflow job goes to **FAILED/KILLED/TIMEDOUT** state, the coordinator job is put in **RUNNINGWITHERROR**
-When a user requests to kill a coordinator job, oozie puts the job in status *KILLED* and it sends kill to all submitted workflow jobs.
+When a user requests to kill a coordinator job, oozie puts the job in status **KILLED** and it sends kill to all submitted workflow jobs.
-When a user requests to suspend a coordinator job that is in *RUNNING* status, oozie puts the job in status *SUSPENDED* and it suspends all submitted workflow jobs. Similarly, when a user requests to suspend a coordinator job that is in *RUNNINGWITHERROR* status, oozie puts the job in status *SUSPENDEDWITHERROR* and it suspends all submitted workflow jobs.
+When a user requests to suspend a coordinator job that is in **RUNNING** status, oozie puts the job in status **SUSPENDED** and it suspends all submitted workflow jobs. Similarly, when a user requests to suspend a coordinator job that is in **RUNNINGWITHERROR** status, oozie puts the job in status **SUSPENDEDWITHERROR** and it suspends all submitted workflow jobs.
-When pause time reaches for a coordinator job that is in *RUNNING* status, oozie puts the job in status *PAUSED*. Similarly, when pause time reaches for a coordinator job that is in *RUNNINGWITHERROR* status, oozie puts the job in status *PAUSEDWITHERROR*.
+When pause time reaches for a coordinator job that is in **RUNNING** status, oozie puts the job in status **PAUSED**. Similarly, when pause time reaches for a coordinator job that is in **RUNNINGWITHERROR** status, oozie puts the job in status **PAUSEDWITHERROR**.
-Conversely, when a user requests to resume a *SUSPENDED* coordinator job, oozie puts the job in status *RUNNING*. Also, when a user requests to resume a *SUSPENDEDWITHERROR* coordinator job, oozie puts the job in status *RUNNINGWITHERROR*. And when pause time is reset for a coordinator job and job status is *PAUSED*, oozie puts the job in status *RUNNING*. Also, when the pause time is reset for a coordinator job and job status is *PAUSEDWITHERROR*, oozie puts the job in status *RUNNINGWITHERROR*
+Conversely, when a user requests to resume a **SUSPENDED** coordinator job, oozie puts the job in status **RUNNING**. Also, when a user requests to resume a **SUSPENDEDWITHERROR** coordinator job, oozie puts the job in status **RUNNINGWITHERROR**. And when pause time is reset for a coordinator job and job status is **PAUSED**, oozie puts the job in status **RUNNING**. Also, when the pause time is reset for a coordinator job and job status is **PAUSEDWITHERROR**, oozie puts the job in status **RUNNINGWITHERROR**
-A coordinator job creates workflow jobs (commonly coordinator actions) only for the duration of the coordinator job and only if the coordinator job is in *RUNNING* status. If the coordinator job has been suspended, when resumed it will create all the coordinator actions that should have been created during the time it was suspended, actions will not be lost, they will delayed.
+A coordinator job creates workflow jobs (commonly coordinator actions) only for the duration of the coordinator job and only if the coordinator job is in **RUNNING** status. If the coordinator job has been suspended, when resumed it will create all the coordinator actions that should have been created during the time it was suspended, actions will not be lost, they will delayed.
When the coordinator job materialization finishes and all workflow jobs finish, oozie updates the coordinator status accordingly.
-For example, if all workflows are *SUCCEEDED*, oozie puts the coordinator job into *SUCCEEDED* status.
-If all workflows are *FAILED*, oozie puts the coordinator job into *FAILED* status. If all workflows are *KILLED*, the coordinator
-job status changes to KILLED. However, if any workflow job finishes with not *SUCCEEDED* and combination of *KILLED*, *FAILED* or
-*TIMEOUT*, oozie puts the coordinator job into *DONEWITHERROR*. If all coordinator actions are *TIMEDOUT*, oozie puts the
-coordinator job into *DONEWITHERROR*.
+For example, if all workflows are **SUCCEEDED**, oozie puts the coordinator job into **SUCCEEDED** status.
+If all workflows are **FAILED**, oozie puts the coordinator job into **FAILED** status. If all workflows are **KILLED**, the coordinator
+job status changes to KILLED. However, if any workflow job finishes with not **SUCCEEDED** and combination of **KILLED**, **FAILED** or
+**TIMEOUT**, oozie puts the coordinator job into **DONEWITHERROR**. If all coordinator actions are **TIMEDOUT**, oozie puts the
+coordinator job into **DONEWITHERROR**.
-A coordinator job in *FAILED* or *KILLED* status can be changed to *IGNORED* status. A coordinator job in *IGNORED* status can be changed to
- *RUNNING* status.
+A coordinator job in **FAILED** or **KILLED** status can be changed to **IGNORED** status. A coordinator job in **IGNORED** status can be changed to
+ **RUNNING** status.
----++++ 6.1.3. Coordinator Action
+#### 6.1.3. Coordinator Action
A coordinator job creates and executes coordinator actions.
@@ -935,169 +979,171 @@ A coordinator action is normally a workflow job that consumes and produces datas
Once an coordinator action is created (this is also referred as the action being materialized), the coordinator action will be in waiting until all required inputs for execution are satisfied or until the waiting times out.
----+++++ 6.1.3.1. Coordinator Action Creation (Materialization)
+##### 6.1.3.1. Coordinator Action Creation (Materialization)
A coordinator job has one driver event that determines the creation (materialization) of its coordinator actions (typically a workflow job).
* For synchronous coordinator jobs the driver event is the frequency of the coordinator job.
----+++++ 6.1.3.2. Coordinator Action Status
+##### 6.1.3.2. Coordinator Action Status
-Once a coordinator action has been created (materialized) the coordinator action qualifies for execution. At this point, the action status is *WAITING*.
+Once a coordinator action has been created (materialized) the coordinator action qualifies for execution. At this point, the action status is **WAITING**.
-A coordinator action in *WAITING* status must wait until all its input events are available before is ready for execution. When a coordinator action is ready for execution its status is *READY*.
+A coordinator action in **WAITING** status must wait until all its input events are available before is ready for execution. When a coordinator action is ready for execution its status is **READY**.
-A coordinator action in *WAITING* status may timeout before it becomes ready for execution. Then the action status is *TIMEDOUT*.
+A coordinator action in **WAITING** status may timeout before it becomes ready for execution. Then the action status is **TIMEDOUT**.
-A coordinator action may remain in *READY* status for a while, without starting execution, due to the concurrency execution policies of the coordinator job.
+A coordinator action may remain in **READY** status for a while, without starting execution, due to the concurrency execution policies of the coordinator job.
-A coordinator action in *READY* or *WAITING* status changes to *SKIPPED* status if the execution strategy is LAST_ONLY and the
+A coordinator action in **READY** or **WAITING** status changes to **SKIPPED** status if the execution strategy is LAST_ONLY and the
current time is past the next action's nominal time. See section 6.3 for more details.
-A coordinator action in *READY* or *WAITING* status changes to *SKIPPED* status if the execution strategy is NONE and the
+A coordinator action in **READY** or **WAITING** status changes to **SKIPPED** status if the execution strategy is NONE and the
current time is past the action's nominal time + 1 minute. See section 6.3 for more details.
-A coordinator action in *READY* status changes to *SUBMITTED* status if total current *RUNNING* and *SUBMITTED* actions are less than concurrency execution limit.
+A coordinator action in **READY** status changes to **SUBMITTED** status if total current **RUNNING** and **SUBMITTED** actions are less than concurrency execution limit.
-A coordinator action in *SUBMITTED* status changes to *RUNNING* status when the workflow engine start execution of the coordinator action.
+A coordinator action in **SUBMITTED** status changes to **RUNNING** status when the workflow engine start execution of the coordinator action.
-A coordinator action is in *RUNNING* status until the associated workflow job completes its execution. Depending on the workflow job completion status, the coordinator action will be in *SUCCEEDED*, *KILLED* or *FAILED* status.
+A coordinator action is in **RUNNING** status until the associated workflow job completes its execution. Depending on the workflow job completion status, the coordinator action will be in **SUCCEEDED**, **KILLED** or **FAILED** status.
-A coordinator action in *WAITING*, *READY*, *SUBMITTED* or *RUNNING* status can be killed, changing to *KILLED* status.
+A coordinator action in **WAITING**, **READY**, **SUBMITTED** or **RUNNING** status can be killed, changing to **KILLED** status.
-A coordinator action in *SUBMITTED* or *RUNNING* status can also fail, changing to *FAILED* status.
+A coordinator action in **SUBMITTED** or **RUNNING** status can also fail, changing to **FAILED** status.
-A coordinator action in *FAILED*, *KILLED*, or *TIMEDOUT* status can be changed to *IGNORED* status. A coordinator action in *IGNORED* status can be
- rerun, changing to *WAITING* status.
+A coordinator action in **FAILED**, **KILLED**, or **TIMEDOUT** status can be changed to **IGNORED** status. A coordinator action in **IGNORED** status can be
+ rerun, changing to **WAITING** status.
Valid coordinator action status transitions are:
- * *WAITING --> READY | TIMEDOUT | SKIPPED | KILLED*
- * *READY --> SUBMITTED | SKIPPED | KILLED*
- * *SUBMITTED --> RUNNING | KILLED | FAILED*
- * *RUNNING --> SUCCEEDED | KILLED | FAILED*
- * *FAILED | KILLED | TIMEDOUT --> IGNORED*
- * *IGNORED --> WAITING*
+ * **WAITING --> READY | TIMEDOUT | SKIPPED | KILLED**
+ * **READY --> SUBMITTED | SKIPPED | KILLED**
+ * **SUBMITTED --> RUNNING | KILLED | FAILED**
+ * **RUNNING --> SUCCEEDED | KILLED | FAILED**
+ * **FAILED | KILLED | TIMEDOUT --> IGNORED**
+ * **IGNORED --> WAITING**
----++++ 6.1.4. Input Events
+#### 6.1.4. Input Events
The Input events of a coordinator application specify the input conditions that are required in order to execute a coordinator action.
In the current specification input events are restricted to dataset instances availability.
-All the datasets instances defined as input events must be available for the coordinator action to be ready for execution ( *READY* status).
+All the datasets instances defined as input events must be available for the coordinator action to be ready for execution ( **READY** status).
Input events are normally parameterized. For example, the last 24 hourly instances of the 'searchlogs' dataset.
Input events can be refer to multiple instances of multiple datasets. For example, the last 24 hourly instances of the 'searchlogs' dataset and the last weekly instance of the 'celebrityRumours' dataset.
----++++ 6.1.5. Output Events
+#### 6.1.5. Output Events
A coordinator action can produce one or more dataset(s) instances as output.
Dataset instances produced as output by one coordinator actions may be consumed as input by another coordinator action(s) of other coordinator job(s).
-The chaining of coordinator jobs via the datasets they produce and consume is referred as a *data pipeline.*
+The chaining of coordinator jobs via the datasets they produce and consume is referred as a **data pipeline.**
In the current specification coordinator job output events are restricted to dataset instances.
----++++ 6.1.6. Coordinator Action Execution Policies
+#### 6.1.6. Coordinator Action Execution Policies
The execution policies for the actions of a coordinator job can be defined in the coordinator application.
* Timeout: A coordinator job can specify the timeout for its coordinator actions, this is, how long the coordinator action will be in *WAITING* or *READY* status before giving up on its execution.
- * Concurrency: A coordinator job can specify the concurrency for its coordinator actions, this is, how many coordinator actions are allowed to run concurrently ( *RUNNING* status) before the coordinator engine starts throttling them.
+ * Concurrency: A coordinator job can specify the concurrency for its coordinator actions, this is, how many coordinator actions are allowed to run concurrently ( **RUNNING** status) before the coordinator engine starts throttling them.
* Execution strategy: A coordinator job can specify the execution strategy of its coordinator actions when there is backlog of coordinator actions in the coordinator engine. The different execution strategies are 'oldest first', 'newest first', 'none' and 'last one only'. A backlog normally happens because of delayed input data, concurrency control or because manual re-runs of coordinator jobs.
* Throttle: A coordinator job can specify the materialization or creation throttle value for its coordinator actions, this is, how many maximum coordinator actions are allowed to be in WAITING state concurrently.
----++++ 6.1.7. Data Pipeline Application
+#### 6.1.7. Data Pipeline Application
Commonly, multiple workflow applications are chained together to form a more complex application.
-Workflow applications are run on regular basis, each of one of them at their own frequency. The data consumed and produced by these workflow applications is relative to the nominal time of workflow job that is processing the data. This is a *coordinator application*.
+Workflow applications are run on regular basis, each of one of them at their own frequency. The data consumed and produced by these workflow applications is relative to the nominal time of workflow job that is processing the data. This is a **coordinator application**.
-The output of multiple workflow jobs of a single workflow application is then consumed by a single workflow job of another workflow application, this is done on regular basis as well. These workflow jobs are triggered by recurrent actions of coordinator jobs. This is a set of *coordinator jobs* that inter-depend on each other via the data they produce and consume.
+The output of multiple workflow jobs of a single workflow application is then consumed by a single workflow job of another workflow application, this is done on regular basis as well. These workflow jobs are triggered by recurrent actions of coordinator jobs. This is a set of **coordinator jobs** that inter-depend on each other via the data they produce and consume.
-This set of interdependent *coordinator applications* is referred as a *data pipeline application*.
+This set of interdependent **coordinator applications** is referred as a **data pipeline application**.
----+++ 6.2. Synchronous Coordinator Application Example
+### 6.2. Synchronous Coordinator Application Example
- * The =checkouts= synchronous dataset is created every 15 minutes by an online checkout store.
- * The =hourlyRevenue= synchronous dataset is created every hour and contains the hourly revenue.
- * The =dailyRevenue= synchronous dataset is created every day and contains the daily revenue.
- * The =monthlyRevenue= synchronous dataset is created every month and contains the monthly revenue.
+ * The `checkouts` synchronous dataset is created every 15 minutes by an online checkout store.
+ * The `hourlyRevenue` synchronous dataset is created every hour and contains the hourly revenue.
+ * The `dailyRevenue` synchronous dataset is created every day and contains the daily revenue.
+ * The `monthlyRevenue` synchronous dataset is created every month and contains the monthly revenue.
- * The =revenueCalculator-wf= workflow consumes checkout data and produces as output the corresponding revenue.
- * The =rollUpRevenue-wf= workflow consumes revenue data and produces a consolidated output.
+ * The `revenueCalculator-wf` workflow consumes checkout data and produces as output the corresponding revenue.
+ * The `rollUpRevenue-wf` workflow consumes revenue data and produces a consolidated output.
- * The =hourlyRevenue-coord= coordinator job triggers, every hour, a =revenueCalculator-wf= workflow. It specifies as input the last 4 =checkouts= dataset instances and it specifies as output a new instance of the =hourlyRevenue= dataset.
- * The =dailyRollUpRevenue-coord= coordinator job triggers, every day, a =rollUpRevenue-wf= workflow. It specifies as input the last 24 =hourlyRevenue= dataset instances and it specifies as output a new instance of the =dailyRevenue= dataset.
- * The =monthlyRollUpRevenue-coord= coordinator job triggers, once a month, a =rollUpRevenue-wf= workflow. It specifies as input all the =dailyRevenue= dataset instance of the month and it specifies as output a new instance of the =monthlyRevenue= dataset.
+ * The `hourlyRevenue-coord` coordinator job triggers, every hour, a `revenueCalculator-wf` workflow. It specifies as input the last 4 `checkouts` dataset instances and it specifies as output a new instance of the `hourlyRevenue` dataset.
+ * The `dailyRollUpRevenue-coord` coordinator job triggers, every day, a `rollUpRevenue-wf` workflow. It specifies as input the last 24 `hourlyRevenue` dataset instances and it specifies as output a new instance of the `dailyRevenue` dataset.
+ * The `monthlyRollUpRevenue-coord` coordinator job triggers, once a month, a `rollUpRevenue-wf` workflow. It specifies as input all the `dailyRevenue` dataset instance of the month and it specifies as output a new instance of the `monthlyRevenue` dataset.
This example contains describes all the components that conform a data pipeline: datasets, coordinator jobs and coordinator actions (workflows).
The coordinator actions (the workflows) are completely agnostic of datasets and their frequencies, they just use them as input and output data (i.e. HDFS files or directories). Furthermore, as the example shows, the same workflow can be used to process similar datasets of different frequencies.
-The frequency of the =hourlyRevenue-coord= coordinator job is 1 hour, this means that every hour a coordinator action is created. A coordinator action will be executed only when the 4 =checkouts= dataset instances for the corresponding last hour are available, until then the coordinator action will remain as created (materialized), in *WAITING* status. Once the 4 dataset instances for the corresponding last hour are available, the coordinator action will be executed and it will start a =revenueCalculator-wf= workflow job.
+The frequency of the `hourlyRevenue-coord` coordinator job is 1 hour, this means that every hour a coordinator action is created. A coordinator action will be executed only when the 4 `checkouts` dataset instances for the corresponding last hour are available, until then the coordinator action will remain as created (materialized), in **WAITING** status. Once the 4 dataset instances for the corresponding last hour are available, the coordinator action will be executed and it will start a `revenueCalculator-wf` workflow job.
----+++ 6.3. Synchronous Coordinator Application Definition
+### 6.3. Synchronous Coordinator Application Definition
A synchronous coordinator definition is a is defined by a name, start time and end time, the frequency of creation of its coordinator actions, the input events, the output events and action control information:
- * *%BLUE% start: %ENDCOLOR%* The start datetime for the job. Starting at this time actions will be materialized. Refer to section #3 'Datetime Representation' for syntax details.
- * *%BLUE% end: %ENDCOLOR%* The end datetime for the job. When actions will stop being materialized. Refer to section #3 'Datetime Representation' for syntax details.
- * *%BLUE% timezone:%ENDCOLOR%* The timezone of the coordinator application.
- * *%BLUE% frequency: %ENDCOLOR%* The frequency, in minutes, to materialize actions. Refer to section #4 'Time Interval Representation' for syntax details.
+ * **<font color="#0000ff"> start: </font>** The start datetime for the job. Starting at this time actions will be materialized. Refer to section #3 'Datetime Representation' for syntax details.
+ * **<font color="#0000ff"> end: </font>** The end datetime for the job. When actions will stop being materialized. Refer to section #3 'Datetime Representation' for syntax details.
+ * **<font color="#0000ff"> timezone:</font>** The timezone of the coordinator application.
+ * **<font color="#0000ff"> frequency: </font>** The frequency, in minutes, to materialize actions. Refer to section #4 'Time Interval Representation' for syntax details.
* Control information:
- * *%BLUE% timeout: %ENDCOLOR%* The maximum time, in minutes, that a materialized action will be waiting for the additional conditions to be satisfied before being discarded. A timeout of =0= indicates that at the time of materialization all the other conditions must be satisfied, else the action will be discarded. A timeout of =0= indicates that if all the input events are not satisfied at the time of action materialization, the action should timeout immediately. A timeout of =-1= indicates no timeout, the materialized action will wait forever for the other conditions to be satisfied. The default value is =-1=. The timeout can only cause a =WAITING= action to transition to =TIMEDOUT=; once the data dependency is satisified, a =WAITING= action transitions to =READY=, and the timeout no longer has any affect, even if the action hasn't transitioned to =SUBMITTED= or =RUNNING= when it expires.
- * *%BLUE% concurrency: %ENDCOLOR%* The maximum number of actions for this job that can be running at the same time. This value allows to materialize and submit multiple instances of the coordinator app, and allows operations to catchup on delayed processing. The default value is =1=.
- * *%BLUE% execution: %ENDCOLOR%* Specifies the execution order if multiple instances of the coordinator job have satisfied their execution criteria. Valid values are:
- * =FIFO= (oldest first) *default*.
- * =LIFO= (newest first).
- * =LAST_ONLY= (see explanation below).
- * =NONE= (see explanation below).
- * *%BLUE% throttle: %ENDCOLOR%* The maximum coordinator actions are allowed to be in WAITING state concurrently. The default value is =12=.
- * *%BLUE% datasets: %ENDCOLOR%* The datasets coordinator application uses.
- * *%BLUE% input-events: %ENDCOLOR%* The coordinator job input events.
- * *%BLUE% data-in: %ENDCOLOR%* It defines one job input condition that resolves to one or more instances of a dataset.
- * *%BLUE% name: %ENDCOLOR%* input condition name.
- * *%BLUE% dataset: %ENDCOLOR%* dataset name.
- * *%BLUE% instance: %ENDCOLOR%* refers to a single dataset instance (the time for a synchronous dataset).
- * *%BLUE% start-instance: %ENDCOLOR%* refers to the beginning of an instance range (the time for a synchronous dataset).
- * *%BLUE% end-instance: %ENDCOLOR%* refers to the end of an instance range (the time for a synchronous dataset).
- * *%BLUE% output-events: %ENDCOLOR%* The coordinator job output events.
- * *%BLUE% data-out: %ENDCOLOR%* It defines one job output that resolves to a dataset instance.
- * *%BLUE% name: %ENDCOLOR%* output name.
- * *%BLUE% dataset: %ENDCOLOR%* dataset name.
- * *%BLUE% instance: %ENDCOLOR%* dataset instance that will be generated by coordinator action.
- * *%BLUE% nocleanup: %ENDCOLOR%* disable cleanup of the output dataset in rerun if true, even when nocleanup option is not used in CLI command.
- * *%BLUE% action: %ENDCOLOR%* The coordinator action to execute.
- * *%BLUE% workflow: %ENDCOLOR%* The workflow job invocation. Workflow job properties can refer to the defined data-in and data-out elements.
-
-*LAST_ONLY:* While =FIFO= and =LIFO= simply specify the order in which READY actions should be executed, =LAST_ONLY= can actually
-cause some actions to be SKIPPED and is a little harder to understand. When =LAST_ONLY= is set, an action that is =WAITING=
-or =READY= will be =SKIPPED= when the current time is past the next action's nominal time. For example, suppose action 1 and 2
-are both =READY=, the current time is 5:00pm, and action 2's nominal time is 5:10pm. In 10 minutes from now, at 5:10pm, action 1
-will become SKIPPED, assuming it doesn't transition to =SUBMITTED= (or a terminal state) before then. This sounds similar to the
+ * **<font color="#0000ff"> timeout: </font>** The maximum time, in minutes, that a materialized action will be waiting for the additional conditions to be satisfied before being discarded. A timeout of `0` indicates that at the time of materialization all the other conditions must be satisfied, else the action will be discarded. A timeout of `0` indicates that if all the input events are not satisfied at the time of action materialization, the action should timeout immediately. A timeout of `-1` indicates no timeout, the materialized action will wait forever for the other conditions to be satisfied. The default value is `-1`. The t
<TRUNCATED>
[04/11] oozie git commit: OOZIE-2734 [docs] Switch from TWiki to
Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_QuickStart.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_QuickStart.twiki b/docs/src/site/twiki/DG_QuickStart.twiki
deleted file mode 100644
index d6a0069..0000000
--- a/docs/src/site/twiki/DG_QuickStart.twiki
+++ /dev/null
@@ -1,230 +0,0 @@
-<noautolink>
-
-[[index][::Go back to Oozie Documentation Index::]]
-
----+!! Oozie Quick Start
-
-These instructions install and run Oozie using an embedded Jetty server and an embedded Derby database.
-
-For detailed install and configuration instructions refer to [[AG_Install][Oozie Install]].
-
-%TOC%
-
----++ Building Oozie
-
----+++ System Requirements:
- * Unix box (tested on Mac OS X and Linux)
- * Java JDK 1.8+
- * Maven 3.0.1+
- * Hadoop 2.6.0+
- * Pig 0.10.1+
-
-JDK commands (java, javac) must be in the command path.
-
-The Maven command (mvn) must be in the command path.
-
----+++ Building Oozie
-
-Download a source distribution of Oozie from the "Releases" drop down menu on the [[http://oozie.apache.org][Oozie site]].
-
-Expand the source distribution =tar.gz= and change directories into it.
-
-The simplest way to build Oozie is to run the =mkdistro.sh= script:
-<verbatim>
-$ bin/mkdistro.sh [-DskipTests]
-
-Running =mkdistro.sh= will create the binary distribution of Oozie. By default, oozie war will not contain hadoop and
-hcatalog libraries, however they are required for oozie to work. There are 2 options to add these libraries:
-
-1. At install time, copy the hadoop and hcatalog libraries to libext and run oozie-setup.sh to setup Oozie. This is
-suitable when same oozie package needs to be used in multiple set-ups with different hadoop/hcatalog versions.
-
-2. Build with -Puber which will bundle the required libraries in the oozie war. Further, the following options are
-available to customise the versions of the dependencies:
--Dhadoop.version=<version> - default 2.6.0
--Ptez - Bundle tez jars in hive and pig sharelibs. Useful if you want to use tez
-+as the execution engine for those applications.
--Dpig.version=<version> - default 0.16.0
--Dpig.classifier=<classifier> - default h2
--Dsqoop.version=<version> - default 1.4.3
--Dsqoop.classifier=<classifier> - default hadoop100
--Djetty.version=<version> - default 9.3.20.v20170531
--Dopenjpa.version=<version> - default 2.2.2
--Dxerces.version=<version> - default 2.10.0
--Dcurator.version=<version> - default 2.5.0
--Dhive.version=<version - default 1.2.0
--Dhbase.version=<version> - default 1.2.3
--Dtez.version=<version> - default 0.8.4
-
-*IMPORTANT:* Profile hadoop-3 must be activated if building against Hadoop 3
-</verbatim>
-
-More details on building Oozie can be found on the [[ENG_Building][Building Oozie]] page.
-
----++ Server Installation
-
----+++ System Requirements
-
- * Unix (tested in Linux and Mac OS X)
- * Java 1.8+
- * Hadoop
- * [[http://hadoop.apache.org][Apache Hadoop]] (tested with 1.2.1 & 2.6.0+)
- * ExtJS library (optional, to enable Oozie webconsole)
- * [[http://archive.cloudera.com/gplextras/misc/ext-2.2.zip][ExtJS 2.2]]
-
-The Java 1.8+ =bin= directory should be in the command path.
-
----+++ Server Installation
-
-*IMPORTANT:* Oozie ignores any set value for =OOZIE_HOME=, Oozie computes its home automatically.
-
- * Build an Oozie binary distribution
- * Download a Hadoop binary distribution
- * Download ExtJS library (it must be version 2.2)
-
-*NOTE:* The ExtJS library is not bundled with Oozie because it uses a different license.
-
-*NOTE:* Oozie UI browser compatibility Chrome (all), Firefox (3.5), Internet Explorer (8.0), Opera (10.5).
-
-*NOTE:* It is recommended to use a Oozie Unix user for the Oozie server.
-
-Expand the Oozie distribution =tar.gz=.
-
-Expand the Hadoop distribution =tar.gz= (as the Oozie Unix user).
-
-#HadoopProxyUser
-
-*NOTE:* Configure the Hadoop cluster with proxyuser for the Oozie process.
-
-The following two properties are required in Hadoop core-site.xml:
-
-<verbatim>
- <!-- OOZIE -->
- <property>
- <name>hadoop.proxyuser.[OOZIE_SERVER_USER].hosts</name>
- <value>[OOZIE_SERVER_HOSTNAME]</value>
- </property>
- <property>
- <name>hadoop.proxyuser.[OOZIE_SERVER_USER].groups</name>
- <value>[USER_GROUPS_THAT_ALLOW_IMPERSONATION]</value>
- </property>
-</verbatim>
-
-Replace the capital letter sections with specific values and then restart Hadoop.
-
-The ExtJS library is optional (only required for the Oozie web-console to work)
-
-*IMPORTANT:* all Oozie server scripts (=oozie-setup.sh=, =oozied.sh=, =oozie-start.sh=, =oozie-run.sh=
-and =oozie-stop.sh=) run only under the Unix user that owns the Oozie installation directory,
-if necessary use =sudo -u OOZIE_USER= when invoking the scripts.
-
-As of Oozie 3.3.2, use of =oozie-start.sh=, =oozie-run.sh=, and =oozie-stop.sh= has
-been deprecated and will print a warning. The =oozied.sh= script should be used
-instead; passing it =start=, =run=, or =stop= as an argument will perform the
-behaviors of =oozie-start.sh=, =oozie-run.sh=, and =oozie-stop.sh= respectively.
-
-Create a *libext/* directory in the directory where Oozie was expanded.
-
-If using the ExtJS library copy the ZIP file to the *libext/* directory. If hadoop and hcatalog libraries are not
-already included in the war, add the corresponding libraries to *libext/* directory.
-
-A "sharelib create -fs fs_default_name [-locallib sharelib]" command is available when running oozie-setup.sh
-for uploading new sharelib into hdfs where the first argument is the default fs name
-and the second argument is the Oozie sharelib to install, it can be a tarball or the expanded version of it.
-If the second argument is omitted, the Oozie sharelib tarball from the Oozie installation directory will be used.
-Upgrade command is deprecated, one should use create command to create new version of sharelib.
-Sharelib files are copied to new lib_<timestamped> directory. At start, server picks the sharelib from latest time-stamp directory.
-While starting server also purge sharelib directory which is older than sharelib retention days
-(defined as oozie.service.ShareLibService.temp.sharelib.retention.days and 7 days is default).
-
-db create|upgrade|postupgrade -run [-sqlfile <FILE>] command is for create, upgrade or postupgrade oozie db with an
-optional sql file
-
-Run the =oozie-setup.sh= script to configure Oozie with all the components added to the *libext/* directory.
-
-<verbatim>
-$ bin/oozie-setup.sh sharelib create -fs <FS_URI> [-locallib <PATH>]
- sharelib upgrade -fs <FS_URI> [-locallib <PATH>]
- db create|upgrade|postupgrade -run [-sqlfile <FILE>]
-</verbatim>
-
-*IMPORTANT*: If the Oozie server needs to establish secure connection with an external server with a self-signed certificate,
-make sure you specify the location of a truststore that contains required certificates. It can be done by configuring
-=oozie.https.truststore.file= in =oozie-site.xml=, or by setting the =javax.net.ssl.trustStore= system property.
-If it is set in both places, the value passed as system property will be used.
-
-Create the Oozie DB using the 'ooziedb.sh' command line tool:
-
-<verbatim>
-$ bin/ooziedb.sh create -sqlfile oozie.sql -run
-
-Validate DB Connection.
-DONE
-Check DB schema does not exist
-DONE
-Check OOZIE_SYS table does not exist
-DONE
-Create SQL schema
-DONE
-DONE
-Create OOZIE_SYS table
-DONE
-
-Oozie DB has been created for Oozie version '3.2.0'
-
-$
-</verbatim>
-
-Start Oozie as a daemon process run:
-
-<verbatim>
-$ bin/oozied.sh start
-</verbatim>
-
-To start Oozie as a foreground process run:
-
-<verbatim>
-$ bin/oozied.sh run
-</verbatim>
-
-Check the Oozie log file =logs/oozie.log= to ensure Oozie started properly.
-
-Using the Oozie command line tool check the status of Oozie:
-
-<verbatim>
-$ bin/oozie admin -oozie http://localhost:11000/oozie -status
-</verbatim>
-
-Using a browser go to the [[http://localhost:11000/oozie][Oozie web console]], Oozie status should be *NORMAL*.
-
-Refer to the [[DG_Examples][Running the Examples]] document for details on running the examples.
-
----++ Client Installation
-
----+++ System Requirements
-
- * Unix (tested in Linux and Mac OS X)
- * Java 1.8+
-
-The Java 1.8+ =bin= directory should be in the command path.
-
----+++ Client Installation
-
-Copy and expand the =oozie-client= TAR.GZ file bundled with the distribution. Add the =bin/= directory to the =PATH=.
-
-Refer to the [[DG_CommandLineTool][Command Line Interface Utilities]] document for a full reference of the =oozie=
-command line tool.
-
-NOTE: The Oozie server installation includes the Oozie client. The Oozie client should be installed in remote machines
-only.
-
-#OozieShareLib
----++ Oozie Share Lib Installation
-
-Oozie share lib has been installed by oozie-setup.sh create command explained in the earlier section.
-
-See the [[WorkflowFunctionalSpec#ShareLib][Workflow Functional Specification]] and [[AG_Install#Oozie_Share_Lib][Installation]] for more information about the Oozie ShareLib.
-
-[[index][::Go back to Oozie Documentation Index::]]
-
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_SLAMonitoring.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_SLAMonitoring.twiki b/docs/src/site/twiki/DG_SLAMonitoring.twiki
index c91c227..0831b93 100644
--- a/docs/src/site/twiki/DG_SLAMonitoring.twiki
+++ b/docs/src/site/twiki/DG_SLAMonitoring.twiki
@@ -1,12 +1,12 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
----+!! Oozie SLA Monitoring
+[::Go back to Oozie Documentation Index::](index.html)
-%TOC%
+# Oozie SLA Monitoring
----++ Overview
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Overview
Critical jobs can have certain SLA requirements associated with them. This SLA can be in terms of time
i.e. a maximum allowed time limit associated with when the job should start, by when should it end,
@@ -23,27 +23,30 @@ Oozie now also has a SLA tab in the Oozie UI, where users can query for SLA info
of how their jobs fared against their SLAs.
----++ Oozie Server Configuration
+## Oozie Server Configuration
-Refer to [[AG_Install#Notifications_Configuration][Notifications Configuration]] for configuring Oozie server to track
+Refer to [Notifications Configuration](AG_Install.html#Notifications_Configuration) for configuring Oozie server to track
SLA for jobs and send notifications.
----++ SLA Tracking
+## SLA Tracking
Oozie allows tracking SLA for meeting the following criteria:
+
* Start time
* End time
* Job Duration
----++++ Event Status
+### Event Status
Corresponding to each of these 3 criteria, your jobs are processed for whether Met or Miss i.e.
+
* START_MET, START_MISS
* END_MET, END_MISS
* DURATION_MET, DURATION_MISS
----++++ SLA Status
+### SLA Status
Expected end-time is the most important criterion for majority of users while deciding overall SLA Met or Miss.
Hence the _"SLA_Status"_ for a job will transition through these four stages
+
* Not_Started <-- Job not yet begun
* In_Process <-- Job started and is running, and SLAs are being tracked
* Met <-- caused by an END_MET
@@ -52,19 +55,26 @@ Hence the _"SLA_Status"_ for a job will transition through these four stages
In addition to overshooting expected end-time, and END_MISS (and so an eventual SLA MISS) also occurs when the
job does not end successfully e.g. goes to error state - Failed/Killed/Error/Timedout.
----++ Configuring SLA in Applications
+## Configuring SLA in Applications
-To make your jobs trackable for SLA, you simply need to add the =<sla:info>= tag to your workflow application definition.
-If you were already using the existing SLA schema in your workflows (Schema xmlns:sla="uri:oozie:sla:0.1"), you don't need to
+To make your jobs trackable for SLA, you simply need to add the `<sla:info>` tag to your workflow application definition.
+If you were already using the existing SLA schema
+in your workflows (Schema xmlns:sla="uri:oozie:sla:0.1"), you don't need to
do anything extra to receive SLA notifications via JMS messages. This new SLA monitoring framework is backward-compatible -
-no need to change application XML for now and you can continue to fetch old records via the [[DG_CommandLineTool#SLAOperations][command line API]].
-However, usage of old schema and API is deprecated and we strongly recommend using new schema.
- * New SLA schema is 'uri:oozie:sla:0.2'
- * In order to use new SLA schema, you will need to upgrade your workflow/coordinator schema to 0.5 i.e. 'uri:oozie:workflow:0.5'
+no need to change application XML for now and you can continue to fetch old records via the [command line API](DG_CommandLineTool.html#SLAOperations).
+However, usage of old schema
+and API is deprecated and we strongly recommend using new schema.
----+++ SLA Definition in Workflow
+ * New SLA schema
+is 'uri:oozie:sla:0.2'
+ * In order to use new SLA schema,
+ you will need to upgrade your workflow/coordinator schema
+to 0.5 i.e. 'uri:oozie:workflow:0.5'
+
+### SLA Definition in Workflow
Example:
-<verbatim>
+
+```
<workflow-app name="test-wf-job-sla"
xmlns="uri:oozie:workflow:0.5"
xmlns:sla="uri:oozie:sla:0.2">
@@ -97,25 +107,28 @@ Example:
<sla:alert-contact>joe@example.com</sla:alert-contact>
</sla:info>
</workflow-app>
-</verbatim>
+```
-For the list of tags usable under =<sla:info>=, refer to [[WorkflowFunctionalSpec#SLASchema][Schemas Appendix]].
-This new schema is much more compact and meaningful, getting rid of redundant and unused tags.
+For the list of tags usable under `<sla:info>`, refer to [Schemas Appendix](WorkflowFunctionalSpec.html#SLASchema).
+This new schema
+is much more compact and meaningful, getting rid of redundant and unused tags.
- * ==nominal-time==: As the name suggests, this is the time relative to which your jobs' SLAs will be calculated. Generally since Oozie workflows are aligned with synchronous data dependencies, this nominal time can be parameterized to be passed the value of your coordinator nominal time. Nominal time is also required in case of independent workflows and you can specify the time in which you expect the workflow to be run if you don't have a synchronous dataset associated with it.
- * ==should-start==: Relative to =nominal-time= this is the amount of time (along with time-unit - MINUTES, HOURS, DAYS) within which your job should *start running* to meet SLA. This is optional.
- * ==should-end==: Relative to =nominal-time= this is the amount of time (along with time-unit - MINUTES, HOURS, DAYS) within which your job should *finish* to meet SLA.
- * ==max-duration==: This is the maximum amount of time (along with time-unit - MINUTES, HOURS, DAYS) your job is expected to run. This is optional.
- * ==alert-events==: Specify the types of events for which *Email* alerts should be sent. Allowable values in this comma-separated list are start_miss, end_miss and duration_miss. *_met events can generally be deemed low priority and hence email alerting for these is not necessary. However, note that this setting is only for alerts via *email* alerts and not via JMS messages, where all events send out notifications, and user can filter them using desired selectors. This is optional and only applicable when alert-contact is configured.
- * ==alert-contact==: Specify a comma separated list of email addresses where you wish your alerts to be sent. This is optional and need not be configured if you just want to view your job SLA history in the UI and do not want to receive email alerts.
+ * `nominal-time`: As the name suggests, this is the time relative to which your jobs' SLAs will be calculated. Generally since Oozie workflows are aligned with synchronous data dependencies, this nominal time can be parameterized to be passed the value of your coordinator nominal time. Nominal time is also required in case of independent workflows and you can specify the time in which you expect the workflow to be run if you don't have a synchronous dataset associated with it.
+ * `should-start`: Relative to `nominal-time` this is the amount of time (along with time-unit - MINUTES, HOURS, DAYS) within which your job should *start running* to meet SLA. This is optional.
+ * `should-end`: Relative to `nominal-time` this is the amount of time (along with time-unit - MINUTES, HOURS, DAYS) within which your job should *finish* to meet SLA.
+ * `max-duration`: This is the maximum amount of time (along with time-unit - MINUTES, HOURS, DAYS) your job is expected to run. This is optional.
+ * `alert-events`: Specify the types of events for which **Email** alerts should be sent. Allowable values in this comma-separated list are start_miss, end_miss and duration_miss. *_met events can generally be deemed low priority and hence email alerting for these is not necessary. However, note that this setting is only for alerts via *email* alerts and not via JMS messages, where all events send out notifications, and user can filter them using desired selectors. This is optional and only applicable when alert-contact is configured.
+ * `alert-contact`: Specify a comma separated list of email addresses where you wish your alerts to be sent. This is optional and need not be configured if you just want to view your job SLA history in the UI and do not want to receive email alerts.
NOTE: All tags can be parameterized as a EL function or a fixed value.
-Same schema can be applied to and embedded under Workflow-Action as well as Coordinator-Action XML.
+Same schema
+can be applied to and embedded under Workflow-Action as well as Coordinator-Action XML.
+
+### SLA Definition in Workflow Action
----+++ SLA Definition in Workflow Action
-<verbatim>
+```
<workflow-app name="test-wf-action-sla" xmlns="uri:oozie:workflow:0.5" xmlns:sla="uri:oozie:sla:0.2">
<start to="grouper"/>
<action name="grouper">
@@ -130,10 +143,11 @@ Same schema can be applied to and embedded under Workflow-Action as well as Coor
</action>
<end name="end"/>
</workflow-app>
-</verbatim>
+```
----+++ SLA Definition in Coordinator Action
-<verbatim>
+### SLA Definition in Coordinator Action
+
+```
<coordinator-app name="test-coord-sla" frequency="${coord:days(1)}" freq_timeunit="DAY"
end_of_duration="NONE" start="2013-06-20T08:01Z" end="2013-12-01T08:01Z"
timezone="America/Los_Angeles" xmlns="uri:oozie:coordinator:0.4" xmlns:sla="uri:oozie:sla:0.2">
@@ -147,22 +161,24 @@ Same schema can be applied to and embedded under Workflow-Action as well as Coor
</sla:info>
</action>
</coordinator-app>
-</verbatim>
+```
----++ Accessing SLA Information
+## Accessing SLA Information
SLA information is accessible via the following ways:
+
* Through the SLA tab of the Oozie Web UI.
* JMS messages sent to a configured JMS provider for instantaneous tracking.
* RESTful API to query for SLA summary.
- * As an =Instrumentation.Counter= entry that is accessible via RESTful API and reflects to the number of all SLA tracked external
- entities. Name of this counter is =sla-calculator.sla-map=.
+ * As an `Instrumentation.Counter` entry that is accessible via RESTful API and reflects to the number of all SLA tracked external
+ entities. Name of this counter is `sla-calculator.sla-map`.
For JMS Notifications, you have to have a message broker in place, on which Oozie publishes messages and you can
hook on a subscriber to receive those messages. For more info on setting up and consuming JMS messages, refer
-[[DG_JMSNotifications][JMS Notifications]] documentation.
+[JMS Notifications](DG_JMSNotifications.html) documentation.
In the REST API, the following filters can be applied while fetching SLA information:
+
* app_name - Application name
* id - id of the workflow job, workflow action or coordinator action
* parent_id - Parent id of the workflow job, workflow action or coordinator action
@@ -179,14 +195,16 @@ the number of milliseconds that have elapsed since January 1, 1970 00:00:00.000
The examples below demonstrate the use of REST API and explains the JSON response.
----+++ Scenario 1: Workflow Job Start_Miss
-*Request:*
-<verbatim>
+### Scenario 1: Workflow Job Start_Miss
+**Request:**
+
+```
GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=nominal_start=2013-06-18T00:01Z;nominal_end=2013-06-23T00:01Z;app_name=my-sla-app
-</verbatim>
+```
+
+**JSON Response**
-*JSON Response*
-<verbatim>
+```
{
id : "000056-1238791320234-oozie-joe-W"
@@ -208,16 +226,18 @@ GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=nominal_start=2013-06-1
upstreamApps: "dependent-app-1, dependent-app-2"
}
-</verbatim>
+```
+
+### Scenario 2: Workflow Action End_Miss
+**Request:**
----+++ Scenario 2: Workflow Action End_Miss
-*Request:*
-<verbatim>
+```
GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=parent_id=000056-1238791320234-oozie-joe-W
-</verbatim>
+```
-*JSON Response*
-<verbatim>
+**JSON Response**
+
+```
{
id : "000056-1238791320234-oozie-joe-W@map-reduce-action"
@@ -239,16 +259,18 @@ GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=parent_id=000056-123879
upstreamApps: "dependent-app-1, dependent-app-2"
}
-</verbatim>
+```
+
+### Scenario 3: Coordinator Action Duration_Miss
+**Request:**
----+++ Scenario 3: Coordinator Action Duration_Miss
-*Request:*
-<verbatim>
+```
GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=id=000001-1238791320234-oozie-joe-C
-</verbatim>
+```
+
+**JSON Response**
-*JSON Response*
-<verbatim>
+```
{
id : "000001-1238791320234-oozie-joe-C@2"
@@ -270,19 +292,21 @@ GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=id=000001-1238791320234
upstreamApps: "dependent-app-1, dependent-app-2"
}
-</verbatim>
+```
Scenario #3 is particularly interesting because it is an overall "MET" because it met its expected End-time,
but it is "Duration_Miss" because the actual run (between actual start and actual end) exceeded expected duration.
----+++ Scenario 4: All Coordinator actions in a Bundle
-*Request:*
-<verbatim>
+### Scenario 4: All Coordinator actions in a Bundle
+**Request:**
+
+```
GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=bundle=1234567-150130225116604-oozie-B;event_status=END_MISS
-</verbatim>
+```
+
+**JSON Response**
-*JSON Response*
-<verbatim>
+```
{
id : "000001-1238791320234-oozie-joe-C@1"
parentId : "000001-1238791320234-oozie-joe-C"
@@ -323,13 +347,14 @@ GET <oozie-host>:<port>/oozie/v2/sla?timezone=GMT&filter=bundle=1234567-15013022
actualDuration: 3360000 <-- (actual duration in milliseconds)
durationDelay: -4 <-- (duration delay in minutes)
}
-</verbatim>
+```
Scenario #4 (All Coordinator actions in a Bundle) is to get SLA information of all coordinator actions under bundle job in one call.
startDelay/durationDelay/endDelay values returned indicate how much delay compared to expected time (positive values in case of MISS, and negative values in case of MET).
----+++ Sample Email Alert
-<verbatim>
+### Sample Email Alert
+
+```
Subject: OOZIE - SLA END_MISS (AppName=wf-sla-job, JobID=0000004-130610225200680-oozie-oozi-W)
@@ -353,77 +378,82 @@ SLA Details:
Expected End Time - Mon Jun 10 23:38:00 UTC 2013
Expected Duration (in mins) - 5
Actual Duration (in mins) - -1
-</verbatim>
+```
----+++ Changing job SLA definition and alerting
+### Changing job SLA definition and alerting
Following are ways to enable/disable SLA alerts for coordinator actions.
----++++ 1. Specify in Bundle XML during submission.
+#### 1. Specify in Bundle XML during submission.
Following properties can be specified in bundle xml as properties for coordinator.
-=oozie.sla.disable.alerts.older.than= this property can be specified in hours, the SLA notification for
+`oozie.sla.disable.alerts.older.than` this property can be specified in hours, the SLA notification for
coord actions will be disabled whose nominal is time older then this value. Default is 48 hours.
-<verbatim>
+
+```
<property>
<name>oozie.sla.disable.alerts.older.than</name>
<value>12</value>
</property>
-</verbatim>
+```
+
+`oozie.sla.disable.alerts` List of coord actions to be disabled. Value can be specified as list of coord actions or date range.
-=oozie.sla.disable.alerts= List of coord actions to be disabled. Value can be specified as list of coord actions or date range.
-<verbatim>
+```
<property>
<name>oozie.sla.disable.alerts</name>
<value>1,3-4,7-10</value>
</property>
-</verbatim>
+```
Will disable alert for coord actions 1,3,5,7,8,9,10
-=oozie.sla.enable.alerts= List of coord actions to be enabled. Value can be specified as list of coord actions or date range.
-<verbatim>
+`oozie.sla.enable.alerts` List of coord actions to be enabled. Value can be specified as list of coord actions or date range.
+
+```
<property>
<name>oozie.sla.enable.alerts</name>
<value>2009-01-01T01:00Z::2009-05-31T23:59Z</value>
</property>
-</verbatim>
+```
This will enable SLA alert for coord actions whose nominal time is in between (inclusive) 2009-01-01T01:00Z and 2009-05-31T23:59Z.
ALL keyword can be specified to specify all actions. Below property will disable SLA notifications for all coord actions.
-<verbatim>
+
+```
<property>
<name>oozie.sla.disable.alerts</name>
<value>ALL</value>
</property>
-</verbatim>
+```
----++++ 2. Specify during Coordinator job submission or update
+#### 2. Specify during Coordinator job submission or update
Above properties can be specified in job.properties in
-[[DG_CommandLineTool#Updating_coordinator_definition_and_properties][Coord job update command]],
-in [[DG_CommandLineTool#Submitting_a_Workflow_Coordinator_or_Bundle_Job][Coord job submit command]]
-or in [[DG_CommandLineTool#Running_a_Workflow_Coordinator_or_Bundle_Job][Coord job run command]]
+[Coord job update command](DG_CommandLineTool.html#Updating_coordinator_definition_and_properties),
+in [Coord job submit command](DG_CommandLineTool.html#Submitting_a_Workflow_Coordinator_or_Bundle_Job)
+or in [Coord job run command](DG_CommandLineTool.html#Running_a_Workflow_Coordinator_or_Bundle_Job)
----++++ 3. Change using command line
-Refer [[DG_CommandLineTool#Changing_job_SLA_definition_and_alerting][Changing job SLA definition and alerting]] for commandline usage.
+#### 3. Change using command line
+Refer [Changing job SLA definition and alerting](DG_CommandLineTool.html#Changing_job_SLA_definition_and_alerting) for commandline usage.
----++++ 4. Change using REST API
-Refer the REST API [[WebServicesAPI#Changing_job_SLA_definition_and_alerting][Changing job SLA definition and alerting]].
+#### 4. Change using REST API
+Refer the REST API [Changing job SLA definition and alerting](WebServicesAPI.html#Changing_job_SLA_definition_and_alerting).
----++ In-memory SLA entries and database content
+## In-memory SLA entries and database content
-There are special circumstances when the in-memory =SLACalcStatus= entries can exist without the workflow or coordinator job or
+There are special circumstances when the in-memory `SLACalcStatus` entries can exist without the workflow or coordinator job or
action instances in database. For example:
- * SLA tracked database content may already have been deleted, and =SLA_SUMMARY= entry is not present anymore in database
- * SLA tracked database content and =SLA_SUMMARY= entry aren't yet present in database
-By the time =SLAService= scheduled job will be running, SLA map contents are checked. When the =SLA_SUMMARY= entry for the in-memory
+ * SLA tracked database content may already have been deleted, and `SLA_SUMMARY` entry is not present anymore in database
+ * SLA tracked database content and `SLA_SUMMARY` entry aren't yet present in database
+
+By the time `SLAService` scheduled job will be running, SLA map contents are checked. When the `SLA_SUMMARY` entry for the in-memory
SLA entry is missing, a counter is increased. When this counter reaches the server-wide preconfigured value
-=oozie.sla.service.SLAService.maximum.retry.count= (by default =3=), in-memory SLA entry will get purged.
+`oozie.sla.service.SLAService.maximum.retry.count` (by default `3`), in-memory SLA entry will get purged.
----++ Known issues
+## Known issues
There are two known issues when you define SLA for a workflow action.
* If there are decision nodes and SLA is defined for a workflow action not in the execution path because of the decision node, you will still get an SLA_MISS notification.
* If you have dangling action nodes in your workflow definition and SLA is defined for it, you will still get an SLA_MISS notification.
-[[index][::Go back to Oozie Documentation Index::]]
+[::Go back to Oozie Documentation Index::](index.html)
+
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_ShellActionExtension.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_ShellActionExtension.twiki b/docs/src/site/twiki/DG_ShellActionExtension.twiki
index 5894c28..eff4b08 100644
--- a/docs/src/site/twiki/DG_ShellActionExtension.twiki
+++ b/docs/src/site/twiki/DG_ShellActionExtension.twiki
@@ -1,39 +1,39 @@
-<<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
+
+[::Go back to Oozie Documentation Index::](index.html)
-----
----+!! Oozie Shell Action Extension
+# Oozie Shell Action Extension
-%TOC%
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-#ShellAction
----++ Shell Action
+<a name="ShellAction"></a>
+## Shell Action
-The =shell= action runs a Shell command.
+The `shell` action runs a Shell command.
The workflow job will wait until the Shell command completes before
continuing to the next action.
-To run the Shell job, you have to configure the =shell= action with the
-=job-tracker=, =name-node= and Shell =exec= elements as
+To run the Shell job, you have to configure the `shell` action with the
+`job-tracker`, `name-node` and Shell `exec` elements as
well as the necessary arguments and configuration.
-A =shell= action can be configured to create or delete HDFS directories
+A `shell` action can be configured to create or delete HDFS directories
before starting the Shell job.
-Shell _launcher_ configuration can be specified with a file, using the =job-xml=
-element, and inline, using the =configuration= elements.
+Shell _launcher_ configuration can be specified with a file, using the `job-xml`
+element, and inline, using the `configuration` elements.
Oozie EL expressions can be used in the inline configuration. Property
-values specified in the =configuration= element override values specified
-in the =job-xml= file.
+values specified in the `configuration` element override values specified
+in the `job-xml` file.
-Note that YARN =yarn.resourcemanager.address= (=resource-manager=) and HDFS =fs.default.name= (=name-node=) properties
+Note that YARN `yarn.resourcemanager.address` (`resource-manager`) and HDFS `fs.default.name` (`name-node`) properties
must not be present in the inline configuration.
-As with Hadoop =map-reduce= jobs, it is possible to add files and
+As with Hadoop `map-reduce` jobs, it is possible to add files and
archives in order to make them available to the Shell job. Refer to the
[WorkflowFunctionalSpec#FilesArchives][Adding Files and Archives for the Job]
section for more information about this feature.
@@ -45,9 +45,10 @@ command must follow the following requirements:
* The format of the output must be a valid Java Properties file.
* The size of the output must not exceed 2KB.
-*Syntax:*
+**Syntax:**
+
-<verbatim>
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<action name="[NODE-NAME]">
@@ -86,49 +87,50 @@ command must follow the following requirements:
</action>
...
</workflow-app>
-</verbatim>
+```
-The =prepare= element, if present, indicates a list of paths to delete
-or create before starting the job. Specified paths must start with =hdfs://HOST:PORT=.
+The `prepare` element, if present, indicates a list of paths to delete
+or create before starting the job. Specified paths must start with `hdfs://HOST:PORT`.
-The =job-xml= element, if present, specifies a file containing configuration
-for the Shell job. As of schema 0.2, multiple =job-xml= elements are allowed in order to
-specify multiple =job.xml= files.
+The `job-xml` element, if present, specifies a file containing configuration
+for the Shell job. As of schema 0.2, multiple `job-xml` elements are allowed in order to
+specify multiple `job.xml` files.
-The =configuration= element, if present, contains configuration
+The `configuration` element, if present, contains configuration
properties that are passed to the Shell job.
-The =exec= element must contain the path of the Shell command to
+The `exec` element must contain the path of the Shell command to
execute. The arguments of Shell command can then be specified
-using one or more =argument= element.
+using one or more `argument` element.
-The =argument= element, if present, contains argument to be passed to
+The `argument` element, if present, contains argument to be passed to
the Shell command.
-The =env-var= element, if present, contains the environment to be passed
-to the Shell command. =env-var= should contain only one pair of environment variable
+The `env-var` element, if present, contains the environment to be passed
+to the Shell command. `env-var` should contain only one pair of environment variable
and value. If the pair contains the variable such as $PATH, it should follow the
Unix convention such as PATH=$PATH:mypath. Don't use ${PATH} which will be
substituted by Oozie's EL evaluator.
-A =shell= action creates a Hadoop configuration. The Hadoop configuration is made available as a local file to the
+A `shell` action creates a Hadoop configuration. The Hadoop configuration is made available as a local file to the
Shell application in its running directory. The exact file path is exposed to the spawned shell using the environment
-variable called =OOZIE_ACTION_CONF_XML=.The Shell application can access the environment variable to read the action
+variable called `OOZIE_ACTION_CONF_XML`.The Shell application can access the environment variable to read the action
configuration XML file path.
-
-If the =capture-output= element is present, it indicates Oozie to capture output of the STDOUT of the shell command
+
+If the `capture-output` element is present, it indicates Oozie to capture output of the STDOUT of the shell command
execution. The Shell command output must be in Java Properties file format and it must not exceed 2KB. From within the
-workflow definition, the output of an Shell action node is accessible via the =String action:output(String node,
-String key)= function (Refer to section '4.2.6 Action EL Functions').
+workflow definition, the output of an Shell action node is accessible via the `String action:output(String node,
+String key)` function (Refer to section '4.2.6 Action EL Functions').
All the above elements can be parameterized (templatized) using EL
expressions.
-*Example:*
+**Example:**
How to run any shell script or perl script or CPP executable
-<verbatim>
+
+```
<workflow-app xmlns='uri:oozie:workflow:1.0' name='shell-wf'>
<start to='shell1' />
<action name='shell1'>
@@ -154,11 +156,12 @@ How to run any shell script or perl script or CPP executable
</kill>
<end name='end' />
</workflow-app>
-</verbatim>
+```
The corresponding job properties file used to submit Oozie job could be as follows:
-<verbatim>
+
+```
oozie.wf.application.path=hdfs://localhost:8020/user/kamrul/workflows/script
#Execute is expected to be in the Workflow directory.
@@ -173,11 +176,12 @@ resourceManager=localhost:8032
nameNode=hdfs://localhost:8020
queueName=default
-</verbatim>
+```
How to run any java program bundles in a jar.
-<verbatim>
+
+```
<workflow-app xmlns='uri:oozie:workflow:1.0' name='shell-wf'>
<start to='shell1' />
<action name='shell1'>
@@ -204,11 +208,12 @@ How to run any java program bundles in a jar.
</kill>
<end name='end' />
</workflow-app>
-</verbatim>
+```
The corresponding job properties file used to submit Oozie job could be as follows:
-<verbatim>
+
+```
oozie.wf.application.path=hdfs://localhost:8020/user/kamrul/workflows/script
#Hello.jar file is expected to be in the Workflow directory.
@@ -217,42 +222,45 @@ EXEC=Hello.jar
resourceManager=localhost:8032
nameNode=hdfs://localhost:8020
queueName=default
-</verbatim>
+```
----+++ Shell Action Configuration
+### Shell Action Configuration
-=oozie.action.shell.setup.hadoop.conf.dir= - Generates a config directory with various core/hdfs/yarn/mapred-site.xml files and points =HADOOP_CONF_DIR= and =YARN_CONF_DIR= env-vars to it, before the Script is invoked. XML is sourced from the action configuration. Useful when the Shell script passed uses various =hadoop= commands. Default is false.
-=oozie.action.shell.setup.hadoop.conf.dir.write.log4j.properties= - When =oozie.action.shell.setup.hadoop.conf.dir= is enabled, toggle if a log4j.properties file should also be written under the configuration files directory. Default is true.
-=oozie.action.shell.setup.hadoop.conf.dir.log4j.content= - When =oozie.action.shell.setup.hadoop.conf.dir.write.log4j.properties= is enabled, the content to write into the log4j.properties file under the configuration files directory. Default is a simple console based stderr logger, as presented below:
-<verbatim>
+ * `oozie.action.shell.setup.hadoop.conf.dir` - Generates a config directory with various core/hdfs/yarn/mapred-site.xml files and points `HADOOP_CONF_DIR` and `YARN_CONF_DIR` env-vars to it, before the Script is invoked. XML is sourced from the action configuration. Useful when the Shell script passed uses various `hadoop` commands. Default is false.
+ * `oozie.action.shell.setup.hadoop.conf.dir.write.log4j.properties` - When `oozie.action.shell.setup.hadoop.conf.dir` is enabled, toggle if a log4j.properties file should also be written under the configuration files directory. Default is true.
+ * `oozie.action.shell.setup.hadoop.conf.dir.log4j.content` - When `oozie.action.shell.setup.hadoop.conf.dir.write.log4j.properties` is enabled, the content to write into the log4j.properties file under the configuration files directory. Default is a simple console based stderr logger, as presented below:
+
+```
log4j.rootLogger=${hadoop.root.logger}
hadoop.root.logger=INFO,console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
-</verbatim>
+```
----+++ Shell Action Logging
+### Shell Action Logging
Shell action's stdout and stderr output are redirected to the Oozie Launcher map-reduce job task STDOUT that runs the shell command.
From Oozie web-console, from the Shell action pop up using the 'Console URL' link, it is possible
to navigate to the Oozie Launcher map-reduce job task logs via the Hadoop job-tracker web-console.
----+++ Shell Action Limitations
+### Shell Action Limitations
Although Shell action can execute any shell command, there are some limitations.
+
* No interactive command is supported.
* Command can't be executed as different user using sudo.
- * User has to explicitly upload the required 3rd party packages (such as jar, so lib, executable etc). Oozie provides a way using <file> and <archive> tag through Hadoop's Distributed Cache to upload.
+ * User has to explicitly upload the required 3rd party packages (such as jar, so lib, executable etc). Oozie provides a way using \<file\> and \<archive\> tag through Hadoop's Distributed Cache to upload.
* Since Oozie will execute the shell command into a Hadoop compute node, the default installation of utility in the compute node might not be fixed. However, the most common unix utilities are usually installed on all compute nodes. It is important to note that Oozie could only support the commands that are installed into the compute nodes or that are uploaded through Distributed Cache.
----++ Appendix, Shell XML-Schema
+## Appendix, Shell XML-Schema
----+++ AE.A Appendix A, Shell XML-Schema
+### AE.A Appendix A, Shell XML-Schema
----++++ Shell Action Schema Version 1.0
-<verbatim>
+#### Shell Action Schema Version 1.0
+
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:shell="uri:oozie:shell-action:1.0"
elementFormDefault="qualified"
@@ -283,10 +291,11 @@ Although Shell action can execute any shell command, there are some limitations.
</xs:complexType>
.
</xs:schema>
-</verbatim>
+```
+
+#### Shell Action Schema Version 0.3
----++++ Shell Action Schema Version 0.3
-<verbatim>
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:shell="uri:oozie:shell-action:0.3" elementFormDefault="qualified"
targetNamespace="uri:oozie:shell-action:0.3">
@@ -341,10 +350,11 @@ Although Shell action can execute any shell command, there are some limitations.
</xs:complexType>
</xs:schema>
-</verbatim>
+```
----++++ Shell Action Schema Version 0.2
-<verbatim>
+#### Shell Action Schema Version 0.2
+
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:shell="uri:oozie:shell-action:0.2" elementFormDefault="qualified"
targetNamespace="uri:oozie:shell-action:0.2">
@@ -399,10 +409,11 @@ Although Shell action can execute any shell command, there are some limitations.
</xs:complexType>
</xs:schema>
-</verbatim>
+```
+
+#### Shell Action Schema Version 0.1
----++++ Shell Action Schema Version 0.1
-<verbatim>
+```
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -476,8 +487,8 @@ Although Shell action can execute any shell command, there are some limitations.
</xs:complexType>
</xs:schema>
-</verbatim>
+```
+
+[::Go back to Oozie Documentation Index::](index.html)
-[[index][::Go back to Oozie Documentation Index::]]
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_SparkActionExtension.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_SparkActionExtension.twiki b/docs/src/site/twiki/DG_SparkActionExtension.twiki
index ce80e45..5a56cca 100644
--- a/docs/src/site/twiki/DG_SparkActionExtension.twiki
+++ b/docs/src/site/twiki/DG_SparkActionExtension.twiki
@@ -1,36 +1,37 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
+
+[::Go back to Oozie Documentation Index::](index.html)
-----
----+!! Oozie Spark Action Extension
+# Oozie Spark Action Extension
-%TOC%
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
----++ Spark Action
+## Spark Action
-The =spark= action runs a Spark job.
+The `spark` action runs a Spark job.
The workflow job will wait until the Spark job completes before
continuing to the next action.
-To run the Spark job, you have to configure the =spark= action with
-the =resource-manager=, =name-node=, Spark =master= elements as
+To run the Spark job, you have to configure the `spark` action with
+the `resource-manager`, `name-node`, Spark `master` elements as
well as the necessary elements, arguments and configuration.
-Spark options can be specified in an element called =spark-opts=.
+Spark options can be specified in an element called `spark-opts`.
-A =spark= action can be configured to create or delete HDFS directories
+A `spark` action can be configured to create or delete HDFS directories
before starting the Spark job.
Oozie EL expressions can be used in the inline configuration. Property
-values specified in the =configuration= element override values specified
-in the =job-xml= file.
+values specified in the `configuration` element override values specified
+in the `job-xml` file.
+
+**Syntax:**
-*Syntax:*
-<verbatim>
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<action name="[NODE-NAME]">
@@ -67,41 +68,43 @@ in the =job-xml= file.
</action>
...
</workflow-app>
-</verbatim>
+```
-The =prepare= element, if present, indicates a list of paths to delete
-or create before starting the job. Specified paths must start with =hdfs://HOST:PORT=.
+The `prepare` element, if present, indicates a list of paths to delete
+or create before starting the job. Specified paths must start with `hdfs://HOST:PORT`.
-The =job-xml= element, if present, specifies a file containing configuration
-for the Spark job. Multiple =job-xml= elements are allowed in order to
-specify multiple =job.xml= files.
+The `job-xml` element, if present, specifies a file containing configuration
+for the Spark job. Multiple `job-xml` elements are allowed in order to
+specify multiple `job.xml` files.
-The =configuration= element, if present, contains configuration
+The `configuration` element, if present, contains configuration
properties that are passed to the Spark job.
-The =master= element indicates the url of the Spark Master. Ex: spark://host:port, mesos://host:port, yarn-cluster, yarn-client,
+The `master` element indicates the url of the Spark Master. Ex: `spark://host:port`, `mesos://host:port`, yarn-cluster, yarn-client,
or local.
-The =mode= element if present indicates the mode of spark, where to run spark driver program. Ex: client,cluster. This is typically
-not required because you can specify it as part of =master= (i.e. master=yarn, mode=client is equivalent to master=yarn-client).
-A local =master= always runs in client mode.
+The `mode` element if present indicates the mode of spark, where to run spark driver program. Ex: client,cluster. This is typically
+not required because you can specify it as part of `master` (i.e. master`yarn, mode`client is equivalent to master=yarn-client).
+A local `master` always runs in client mode.
+
+Depending on the `master` (and `mode`) entered, the Spark job will run differently as follows:
-Depending on the =master= (and =mode=) entered, the Spark job will run differently as follows:
* local mode: everything runs here in the Launcher Job.
* yarn-client mode: the driver runs here in the Launcher Job and the executor in Yarn.
* yarn-cluster mode: the driver and executor run in Yarn.
-The =name= element indicates the name of the spark application.
+The `name` element indicates the name of the spark application.
-The =class= element if present, indicates the spark's application main class.
+The `class` element if present, indicates the spark's application main class.
-The =jar= element indicates a comma separated list of jars or python files.
+The `jar` element indicates a comma separated list of jars or python files.
-The =spark-opts= element, if present, contains a list of Spark options that can be passed to Spark. Spark configuration
+The `spark-opts` element, if present, contains a list of Spark options that can be passed to Spark. Spark configuration
options can be passed by specifying '--conf key=value' or other Spark CLI options.
Values containing whitespaces can be enclosed by double quotes.
-Some examples of the =spark-opts= element:
+Some examples of the `spark-opts` element:
+
* '--conf key=value'
* '--conf key1=value1 value2'
* '--conf key1="value1 value2"'
@@ -109,32 +112,35 @@ Some examples of the =spark-opts= element:
* '--conf key=value --verbose --properties-file user.properties'
There are several ways to define properties that will be passed to Spark. They are processed in the following order:
- * propagated from =oozie.service.SparkConfigurationService.spark.configurations=
- * read from a localized =spark-defaults.conf= file
- * read from a file defined in =spark-opts= via the =--properties-file=
- * properties defined in =spark-opts= element
+
+ * propagated from `oozie.service.SparkConfigurationService.spark.configurations`
+ * read from a localized `spark-defaults.conf` file
+ * read from a file defined in `spark-opts` via the `--properties-file`
+ * properties defined in `spark-opts` element
(The latter takes precedence over the former.)
-The server propagated properties, the =spark-defaults.conf= and the user-defined properties file are merged together into a
-single properties file as Spark handles only one file in its =--properties-file= option.
+The server propagated properties, the `spark-defaults.conf` and the user-defined properties file are merged together into a
+single properties file as Spark handles only one file in its `--properties-file` option.
-The =arg= element if present, contains arguments that can be passed to spark application.
+The `arg` element if present, contains arguments that can be passed to spark application.
-In case some property values are present both in =spark-defaults.conf= and as property key/value pairs generated by Oozie, the user
-configured values from =spark-defaults.conf= are prepended to the ones generated by Oozie, as part of the Spark arguments list.
+In case some property values are present both in `spark-defaults.conf` and as property key/value pairs generated by Oozie, the user
+configured values from `spark-defaults.conf` are prepended to the ones generated by Oozie, as part of the Spark arguments list.
Following properties to prepend to Spark arguments:
- * =spark.executor.extraClassPath=
- * =spark.driver.extraClassPath=
- * =spark.executor.extraJavaOptions=
- * =spark.driver.extraJavaOptions=
+
+ * `spark.executor.extraClassPath`
+ * `spark.driver.extraClassPath`
+ * `spark.executor.extraJavaOptions`
+ * `spark.driver.extraJavaOptions`
All the above elements can be parameterized (templatized) using EL
expressions.
-*Example:*
+**Example:**
+
-<verbatim>
+```
<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
...
<action name="myfirstsparkjob">
@@ -165,34 +171,35 @@ expressions.
</action>
...
</workflow-app>
-</verbatim>
+```
----+++ Spark Action Logging
+### Spark Action Logging
Spark action logs are redirected to the Oozie Launcher map-reduce job task STDOUT/STDERR that runs Spark.
From Oozie web-console, from the Spark action pop up using the 'Console URL' link, it is possible
to navigate to the Oozie Launcher map-reduce job task logs via the Hadoop job-tracker web-console.
----+++ Spark on YARN
+### Spark on YARN
To ensure that your Spark job shows up in the Spark History Server, make sure to specify these three Spark configuration properties
-either in =spark-opts= with =--conf= or from =oozie.service.SparkConfigurationService.spark.configurations= in oozie-site.xml.
+either in `spark-opts` with `--conf` or from `oozie.service.SparkConfigurationService.spark.configurations` in oozie-site.xml.
1. spark.yarn.historyServer.address=SPH-HOST:18088
-2. spark.eventLog.dir=hdfs://NN:8020/user/spark/applicationHistory
+2. spark.eventLog.dir=`hdfs://NN:8020/user/spark/applicationHistory`
3. spark.eventLog.enabled=true
----+++ PySpark with Spark Action
+### PySpark with Spark Action
To submit PySpark scripts with Spark Action, pyspark dependencies must be available in sharelib or in workflow's lib/ directory.
-For more information, please refer to [[AG_Install#Oozie_Share_Lib][installation document.]]
+For more information, please refer to [installation document.](AG_Install.html#Oozie_Share_Lib)
-*Example:*
+**Example:**
-<verbatim>
+
+```
<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
....
<action name="myfirstpysparkjob">
@@ -220,24 +227,24 @@ For more information, please refer to [[AG_Install#Oozie_Share_Lib][installation
</action>
...
</workflow-app>
-</verbatim>
+```
-The =jar= element indicates python file. Refer to the file by it's localized name, because only local files are allowed
-in PySpark. The py file should be in the lib/ folder next to the workflow.xml or added using the =file= element so that
+The `jar` element indicates python file. Refer to the file by it's localized name, because only local files are allowed
+in PySpark. The py file should be in the lib/ folder next to the workflow.xml or added using the `file` element so that
it's localized to the working directory with just its name.
----+++ Using Symlink in <jar>
+### Using Symlink in \<jar\>
-A symlink must be specified using =[[WorkflowFunctionalSpec#a3.2.2.
-1_Adding_Files_and_Archives_for_the_Job][file]]= element. Then, you can use
-the symlink name in =jar= element.
+A symlink must be specified using [file](WorkflowFunctionalSpec.html#a3.2.2.1_Adding_Files_and_Archives_for_the_Job) element. Then, you can use
+the symlink name in `jar` element.
-*Example:*
+**Example:**
Specifying relative path for symlink:
-Make sure that the file is within the application directory i.e. =oozie.wf.application.path= .
-<verbatim>
+Make sure that the file is within the application directory i.e. `oozie.wf.application.path` .
+
+```
<spark xmlns="uri:oozie:spark-action:1.0">
...
<jar>py-spark-example-symlink.py</jar>
@@ -246,10 +253,11 @@ Make sure that the file is within the application directory i.e. =oozie.wf.appli
<file>py-spark.py#py-spark-example-symlink.py</file>
...
</spark>
-</verbatim>
+```
Specifying full path for symlink:
-<verbatim>
+
+```
<spark xmlns="uri:oozie:spark-action:1.0">
...
<jar>spark-example-symlink.jar</jar>
@@ -258,16 +266,17 @@ Specifying full path for symlink:
<file>hdfs://localhost:8020/user/testjars/all-oozie-examples.jar#spark-example-symlink.jar</file>
...
</spark>
-</verbatim>
+```
+
+## Appendix, Spark XML-Schema
----++ Appendix, Spark XML-Schema
+### AE.A Appendix A, Spark XML-Schema
----+++ AE.A Appendix A, Spark XML-Schema
+#### Spark Action Schema Version 1.0
----++++ Spark Action Schema Version 1.0
-<verbatim>
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:spark="uri:oozie:spark-action:1.0" elementFormDefault="qualified"
targetNamespace="uri:oozie:spark-action:1.0">
@@ -300,10 +309,11 @@ Specifying full path for symlink:
</xs:complexType>
.
</xs:schema>
-</verbatim>
+```
----++++ Spark Action Schema Version 0.2
-<verbatim>
+#### Spark Action Schema Version 0.2
+
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:spark="uri:oozie:spark-action:0.2" elementFormDefault="qualified"
targetNamespace="uri:oozie:spark-action:0.2">
@@ -359,10 +369,11 @@ Specifying full path for symlink:
</xs:complexType>
</xs:schema>
-</verbatim>
+```
+
+#### Spark Action Schema Version 0.1
----++++ Spark Action Schema Version 0.1
-<verbatim>
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:spark="uri:oozie:spark-action:0.1" elementFormDefault="qualified"
targetNamespace="uri:oozie:spark-action:0.1">
@@ -416,10 +427,10 @@ Specifying full path for symlink:
</xs:complexType>
</xs:schema>
-</verbatim>
-[[index][::Go back to Oozie Documentation Index::]]
+```
+[::Go back to Oozie Documentation Index::](index.html)
+
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_SqoopActionExtension.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_SqoopActionExtension.twiki b/docs/src/site/twiki/DG_SqoopActionExtension.twiki
index 0317d0c..b186c5a 100644
--- a/docs/src/site/twiki/DG_SqoopActionExtension.twiki
+++ b/docs/src/site/twiki/DG_SqoopActionExtension.twiki
@@ -1,46 +1,47 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
+
+[::Go back to Oozie Documentation Index::](index.html)
-----
----+!! Oozie Sqoop Action Extension
+# Oozie Sqoop Action Extension
-%TOC%
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
----++ Sqoop Action
+## Sqoop Action
-*IMPORTANT:* The Sqoop action requires Apache Hadoop 1.x or 2.x.
+**IMPORTANT:** The Sqoop action requires Apache Hadoop 1.x or 2.x.
-The =sqoop= action runs a Sqoop job.
+The `sqoop` action runs a Sqoop job.
The workflow job will wait until the Sqoop job completes before
continuing to the next action.
-To run the Sqoop job, you have to configure the =sqoop= action with the =resource-manager=, =name-node= and Sqoop =command=
-or =arg= elements as well as configuration.
+To run the Sqoop job, you have to configure the `sqoop` action with the `resource-manager`, `name-node` and Sqoop `command`
+or `arg` elements as well as configuration.
-A =sqoop= action can be configured to create or delete HDFS directories
+A `sqoop` action can be configured to create or delete HDFS directories
before starting the Sqoop job.
-Sqoop configuration can be specified with a file, using the =job-xml=
-element, and inline, using the =configuration= elements.
+Sqoop configuration can be specified with a file, using the `job-xml`
+element, and inline, using the `configuration` elements.
Oozie EL expressions can be used in the inline configuration. Property
-values specified in the =configuration= element override values specified
-in the =job-xml= file.
+values specified in the `configuration` element override values specified
+in the `job-xml` file.
-Note that YARN =yarn.resourcemanager.address= / =resource-manager= and HDFS =fs.default.name= / =name-node= properties must not
+Note that YARN `yarn.resourcemanager.address` / `resource-manager` and HDFS `fs.default.name` / `name-node` properties must not
be present in the inline configuration.
-As with Hadoop =map-reduce= jobs, it is possible to add files and
+As with Hadoop `map-reduce` jobs, it is possible to add files and
archives in order to make them available to the Sqoop job. Refer to the
[WorkflowFunctionalSpec#FilesArchives][Adding Files and Archives for the Job]
section for more information about this feature.
-*Syntax:*
+**Syntax:**
+
-<verbatim>
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<action name="[NODE-NAME]">
@@ -73,40 +74,41 @@ section for more information about this feature.
</action>
...
</workflow-app>
-</verbatim>
+```
-The =prepare= element, if present, indicates a list of paths to delete
-or create before starting the job. Specified paths must start with =hdfs://HOST:PORT=.
+The `prepare` element, if present, indicates a list of paths to delete
+or create before starting the job. Specified paths must start with `hdfs://HOST:PORT`.
-The =job-xml= element, if present, specifies a file containing configuration
-for the Sqoop job. As of schema 0.3, multiple =job-xml= elements are allowed in order to
-specify multiple =job.xml= files.
+The `job-xml` element, if present, specifies a file containing configuration
+for the Sqoop job. As of schema 0.3, multiple `job-xml` elements are allowed in order to
+specify multiple `job.xml` files.
-The =configuration= element, if present, contains configuration
+The `configuration` element, if present, contains configuration
properties that are passed to the Sqoop job.
-*Sqoop command*
+**Sqoop command**
-The Sqoop command can be specified either using the =command= element or multiple =arg=
+The Sqoop command can be specified either using the `command` element or multiple `arg`
elements.
-When using the =command= element, Oozie will split the command on every space
+When using the `command` element, Oozie will split the command on every space
into multiple arguments.
-When using the =arg= elements, Oozie will pass each argument value as an argument to Sqoop.
+When using the `arg` elements, Oozie will pass each argument value as an argument to Sqoop.
-The =arg= variant should be used when there are spaces within a single argument.
+The `arg` variant should be used when there are spaces within a single argument.
Consult the Sqoop documentation for a complete list of valid Sqoop commands.
All the above elements can be parameterized (templatized) using EL
expressions.
-*Examples:*
+**Examples:**
-Using the =command= element:
+Using the `command` element:
-<verbatim>
+
+```
<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
...
<action name="myfirsthivejob">
@@ -129,11 +131,12 @@ Using the =command= element:
</action>
...
</workflow-app>
-</verbatim>
+```
+
+The same Sqoop action using `arg` elements:
-The same Sqoop action using =arg= elements:
-<verbatim>
+```
<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
...
<action name="myfirstsqoopjob">
@@ -164,20 +167,20 @@ The same Sqoop action using =arg= elements:
</action>
...
</workflow-app>
-</verbatim>
+```
-NOTE: The =arg= elements syntax, while more verbose, allows to have spaces in a single argument, something useful when
+NOTE: The `arg` elements syntax, while more verbose, allows to have spaces in a single argument, something useful when
using free from queries.
----+++ Sqoop Action Counters
+### Sqoop Action Counters
The counters of the map-reduce job run by the Sqoop action are available to be used in the workflow via the
-[[WorkflowFunctionalSpec#HadoopCountersEL][hadoop:counters() EL function]].
+[hadoop:counters() EL function](WorkflowFunctionalSpec.html#HadoopCountersEL).
-If the Sqoop action run an import all command, the =hadoop:counters()= EL will return the aggregated counters
+If the Sqoop action run an import all command, the `hadoop:counters()` EL will return the aggregated counters
of all map-reduce jobs run by the Sqoop import all command.
----+++ Sqoop Action Logging
+### Sqoop Action Logging
Sqoop action logs are redirected to the Oozie Launcher map-reduce job task STDOUT/STDERR that runs Sqoop.
@@ -185,14 +188,15 @@ From Oozie web-console, from the Sqoop action pop up using the 'Console URL' lin
to navigate to the Oozie Launcher map-reduce job task logs via the Hadoop job-tracker web-console.
The logging level of the Sqoop action can set in the Sqoop action configuration using the
-property =oozie.sqoop.log.level=. The default value is =INFO=.
+property `oozie.sqoop.log.level`. The default value is `INFO`.
----++ Appendix, Sqoop XML-Schema
+## Appendix, Sqoop XML-Schema
----+++ AE.A Appendix A, Sqoop XML-Schema
+### AE.A Appendix A, Sqoop XML-Schema
----++++ Sqoop Action Schema Version 1.0
-<verbatim>
+#### Sqoop Action Schema Version 1.0
+
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:sqoop="uri:oozie:sqoop-action:1.0"
elementFormDefault="qualified"
@@ -223,10 +227,11 @@ property =oozie.sqoop.log.level=. The default value is =INFO=.
</xs:complexType>
.
</xs:schema>
-</verbatim>
+```
+
+#### Sqoop Action Schema Version 0.3
----++++ Sqoop Action Schema Version 0.3
-<verbatim>
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:sqoop="uri:oozie:sqoop-action:0.3" elementFormDefault="qualified"
targetNamespace="uri:oozie:sqoop-action:0.3">
@@ -279,10 +284,11 @@ property =oozie.sqoop.log.level=. The default value is =INFO=.
</xs:complexType>
</xs:schema>
-</verbatim>
+```
----++++ Sqoop Action Schema Version 0.2
-<verbatim>
+#### Sqoop Action Schema Version 0.2
+
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:sqoop="uri:oozie:sqoop-action:0.2" elementFormDefault="qualified"
targetNamespace="uri:oozie:sqoop-action:0.2">
@@ -335,8 +341,8 @@ property =oozie.sqoop.log.level=. The default value is =INFO=.
</xs:complexType>
.
</xs:schema>
-</verbatim>
+```
+
+[::Go back to Oozie Documentation Index::](index.html)
-[[index][::Go back to Oozie Documentation Index::]]
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_SshActionExtension.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_SshActionExtension.twiki b/docs/src/site/twiki/DG_SshActionExtension.twiki
index 5a51d49..e53e1c3 100644
--- a/docs/src/site/twiki/DG_SshActionExtension.twiki
+++ b/docs/src/site/twiki/DG_SshActionExtension.twiki
@@ -1,16 +1,16 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
+
+[::Go back to Oozie Documentation Index::](index.html)
-----
----+!! Oozie Ssh Action Extension
+# Oozie Ssh Action Extension
-%TOC%
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
----++ Ssh Action
+## Ssh Action
-The =ssh= action starts a shell command on a remote machine as a remote secure shell in background. The workflow job
+The `ssh` action starts a shell command on a remote machine as a remote secure shell in background. The workflow job
will wait until the remote shell command completes before continuing to the next action.
The shell command must be present in the remote machine and it must be available for execution via the command path.
@@ -32,9 +32,10 @@ Note: Ssh Action will fail if oozie fails to ssh connect to host for action stat
The first retry will wait a configurable period of time ( 3 seconds by default) before check.
The following retries will wait 2 times of previous wait time.
-*Syntax:*
+**Syntax:**
+
-<verbatim>
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<action name="[NODE-NAME]">
@@ -50,32 +51,33 @@ The following retries will wait 2 times of previous wait time.
</action>
...
</workflow-app>
-</verbatim>
+```
-The =host= indicates the user and host where the shell will be executed.
+The `host` indicates the user and host where the shell will be executed.
-*IMPORTANT:* The =oozie.action.ssh.allow.user.at.host= property, in the =oozie-site.xml= configuration, indicates if
+**IMPORTANT:** The `oozie.action.ssh.allow.user.at.host` property, in the `oozie-site.xml` configuration, indicates if
an alternate user than the one submitting the job can be used for the ssh invocation. By default this property is set
-to =true=.
+to `true`.
-The =command= element indicates the shell command to execute.
+The `command` element indicates the shell command to execute.
-The =args= element, if present, contains parameters to be passed to the shell command. If more than one =args= element
-is present they are concatenated in order. When an =args= element contains a space, even when quoted, it will be considered as
-separate arguments (i.e. "Hello World" becomes "Hello" and "World"). Starting with ssh schema 0.2, you can use the =arg= element
-(note that this is different than the =args= element) to specify arguments that have a space in them (i.e. "Hello World" is
-preserved as "Hello World"). You can use either =args= elements, =arg= elements, or neither; but not both in the same action.
+The `args` element, if present, contains parameters to be passed to the shell command. If more than one `args` element
+is present they are concatenated in order. When an `args` element contains a space, even when quoted, it will be considered as
+separate arguments (i.e. "Hello World" becomes "Hello" and "World"). Starting with ssh schema 0.2, you can use the `arg` element
+(note that this is different than the `args` element) to specify arguments that have a space in them (i.e. "Hello World" is
+preserved as "Hello World"). You can use either `args` elements, `arg` elements, or neither; but not both in the same action.
-If the =capture-output= element is present, it indicates Oozie to capture output of the STDOUT of the ssh command
+If the `capture-output` element is present, it indicates Oozie to capture output of the STDOUT of the ssh command
execution. The ssh command output must be in Java Properties file format and it must not exceed 2KB. From within the
workflow definition, the output of an ssh action node is accessible via the =String action:output(String node,
String key)= function (Refer to section '4.2.6 Action EL Functions').
-The configuration of the =ssh= action can be parameterized (templatized) using EL expressions.
+The configuration of the `ssh` action can be parameterized (templatized) using EL expressions.
+
+**Example:**
-*Example:*
-<verbatim>
+```
<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
...
<action name="myssjob">
@@ -90,22 +92,23 @@ The configuration of the =ssh= action can be parameterized (templatized) using E
</action>
...
</workflow-app>
-</verbatim>
+```
-In the above example, the =uploaddata= shell command is executed with two arguments, =jdbc:derby://foo.com:1527/myDB=
-and =hdfs://foobar.com:8020/usr/tucu/myData=.
+In the above example, the `uploaddata` shell command is executed with two arguments, `jdbc:derby://foo.com:1527/myDB`
+and `hdfs://foobar.com:8020/usr/tucu/myData`.
-The =uploaddata= shell must be available in the remote host and available in the command path.
+The `uploaddata` shell must be available in the remote host and available in the command path.
-The output of the command will be ignored because the =capture-output= element is not present.
+The output of the command will be ignored because the `capture-output` element is not present.
----++ Appendix, Ssh XML-Schema
+## Appendix, Ssh XML-Schema
----+++ AE.A Appendix A, Ssh XML-Schema
+### AE.A Appendix A, Ssh XML-Schema
----++++ Ssh Action Schema Version 0.2
+#### Ssh Action Schema Version 0.2
-<verbatim>
+
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:ssh="uri:oozie:ssh-action:0.2" elementFormDefault="qualified"
targetNamespace="uri:oozie:ssh-action:0.2">
@@ -127,11 +130,12 @@ The output of the command will be ignored because the =capture-output= element i
<xs:complexType name="FLAG"/>
.
</xs:schema>
-</verbatim>
+```
+
+#### Ssh Action Schema Version 0.1
----++++ Ssh Action Schema Version 0.1
-<verbatim>
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:ssh="uri:oozie:ssh-action:0.1" elementFormDefault="qualified"
targetNamespace="uri:oozie:ssh-action:0.1">
@@ -150,8 +154,8 @@ The output of the command will be ignored because the =capture-output= element i
<xs:complexType name="FLAG"/>
.
</xs:schema>
-</verbatim>
+```
+
+[::Go back to Oozie Documentation Index::](index.html)
-[[index][::Go back to Oozie Documentation Index::]]
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_WorkflowReRun.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_WorkflowReRun.twiki b/docs/src/site/twiki/DG_WorkflowReRun.twiki
index 88d982b..c128681 100644
--- a/docs/src/site/twiki/DG_WorkflowReRun.twiki
+++ b/docs/src/site/twiki/DG_WorkflowReRun.twiki
@@ -1,33 +1,34 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
----+!! Workflow ReRrun
+[::Go back to Oozie Documentation Index::](index.html)
-%TOC%
----++ Configs
+# Workflow ReRrun
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+## Configs
* oozie.wf.application.path
- * Only one of following two configurations is mandatory. Both should not be defined at the same time
+ * Only one of following two configurations is mandatory. Both should not be defined at the same time
* oozie.wf.rerun.skip.nodes
* oozie.wf.rerun.failnodes
* Skip nodes are comma separated list of action names. They can be any action nodes including decision node.
- * The valid value of =oozie.wf.rerun.failnodes= is true or false.
+ * The valid value of `oozie.wf.rerun.failnodes` is true or false.
* If secured hadoop version is used, the following two properties needs to be specified as well
* mapreduce.jobtracker.kerberos.principal
* dfs.namenode.kerberos.principal.
* Configurations can be passed as -D param.
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -rerun 14-20090525161321-oozie-joe -Doozie.wf.rerun.skip.nodes=<>
-</verbatim>
+```
----++ Pre-Conditions
+## Pre-Conditions
* Workflow with id wfId should exist.
* Workflow with id wfId should be in SUCCEEDED/KILLED/FAILED.
* If specified , nodes in the config oozie.wf.rerun.skip.nodes must be completed successfully.
----++ ReRun
+## ReRun
* Reloads the configs.
* If no configuration is passed, existing coordinator/workflow configuration will be used. If configuration is passed then, it will be merged with existing workflow configuration. Input configuration will take the precedence.
@@ -36,6 +37,6 @@ $ oozie job -oozie http://localhost:11000/oozie -rerun 14-20090525161321-oozie-j
* Deletes the actions that are not skipped from the DB and copies data from old Workflow Instance to new one for skipped actions.
* Action handler will skip the nodes given in the config with the same exit transition as before.
-[[index][::Go back to Oozie Documentation Index::]]
+[::Go back to Oozie Documentation Index::](index.html)
+
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/ENG_Building.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/ENG_Building.twiki b/docs/src/site/twiki/ENG_Building.twiki
deleted file mode 100644
index b861026..0000000
--- a/docs/src/site/twiki/ENG_Building.twiki
+++ /dev/null
@@ -1,270 +0,0 @@
-<noautolink>
-
-[[index][::Go back to Oozie Documentation Index::]]
-
----+!! Building Oozie
-
-%TOC%
-
----++ System Requirements
-
- * Unix box (tested on Mac OS X and Linux)
- * Java JDK 1.8+
- * [[http://maven.apache.org/][Maven 3.0.1+]]
- * [[http://hadoop.apache.org/core/releases.html][Hadoop 2.6.0+]]
- * [[http://hadoop.apache.org/pig/releases.html][Pig 0.10.1+]]
-
-JDK commands (java, javac) must be in the command path.
-
-The Maven command (mvn) must be in the command path.
-
----++ Oozie Documentation Generation
-
-To generate the documentation, Oozie uses a patched Doxia plugin for Maven with improved twiki support.
-
-The source of the modified plugin is available in the Oozie GitHub repository, in the =ydoxia= branch.
-
-To build and install it locally run the following command in the =ydoxia= branch:
-
-<verbatim>
-$ mvn install
-</verbatim>
-
-#SshSetup
----++ Passphrase-less SSH Setup
-
-*NOTE: SSH actions are deprecated in Oozie 2.*
-
-To run SSH Testcases and for easier Hadoop start/stop configure SSH to localhost to be passphrase-less.
-
-Create your SSH keys without a passphrase and add the public key to the authorized file:
-
-<verbatim>
-$ ssh-keygen -t dsa
-$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys2
-</verbatim>
-
-Test that you can ssh without password:
-
-<verbatim>
-$ ssh localhost
-</verbatim>
-
----++ Building with different Java Versions
-
-Oozie requires a minimum Java version of 1.8. Any newer version can be used but by default bytecode will be generated
-which is compatible with 1.8. This can be changed by specifying the build property *targetJavaVersion*.
-
----++ Building and Testing Oozie
-
-The JARs for the specified Hadoop and Pig versions must be available in one of the Maven repositories defined in Oozie
-main 'pom.xml' file. Or they must be installed in the local Maven cache.
-
----+++ Examples Running Oozie Testcases with Different Configurations
-
-*Using embedded Hadoop minicluster with 'simple' authentication:*
-
-<verbatim>
-$ mvn clean test
-</verbatim>
-
-*Using a Hadoop cluster with 'simple' authentication:*
-
-<verbatim>
-$ mvn clean test -Doozie.test.hadoop.minicluster=false
-</verbatim>
-
-*Using embedded Hadoop minicluster with 'simple' authentication and Derby database:*
-
-<verbatim>
-$ mvn clean test -Doozie.test.hadoop.minicluster=false -Doozie.test.db=derby
-</verbatim>
-
-*Using a Hadoop cluster with 'kerberos' authentication:*
-
-<verbatim>
-$ mvn clean test -Doozie.test.hadoop.minicluster=false -Doozie.test.hadoop.security=kerberos
-</verbatim>
-
-NOTE: The embedded minicluster cannot be used when testing with 'kerberos' authentication.
-
-*Using a custom Oozie configuration for testcases:*
-
-<verbatim>
-$ mvn clean test -Doozie.test.config.file=/home/tucu/custom-oozie-sitel.xml
-</verbatim>
-
-*Running the testcases with different databases:*
-
-<verbatim>
-$ mvn clean test -Doozie.test.db=[hsqldb*|derby|mysql|postgres|oracle]
-</verbatim>
-
-Using =mysql= and =oracle= enables profiles that will include their JARs files in the build. If using
- =oracle=, the Oracle JDBC JAR file must be manually installed in the local Maven cache (the JAR is
-not available in public Maven repos).
-
----+++ Build Options Reference
-
-All these options can be set using *-D*.
-
-Except for the options marked with =(*)=, the options can be specified in the =test.properties= in the root
-of the Oozie project. The options marked with =(*)= are used in Maven POMs, thus they don't take effect if
-specified in the =test.properties= file (which is loaded by the =XTestCase= class at class initialization time).
-
-*hadoop.version* =(*)=: indicates the Hadoop version you wish to build Oozie against specifically. It will
-substitute this value in the Oozie POM properties and pull the corresponding Hadoop artifacts from Maven.
-The default version is 2.6.0 and that is the minimum supported Hadoop version.
-
-*generateSite* (*): generates Oozie documentation, default is undefined (no documentation is generated)
-
-*skipTests* (*): skips the execution of all testcases, no value required, default is undefined
-
-*test*= (*): runs a single test case, to run a test give the test class name without package and extension, no default
-
-*oozie.test.db*= (*): indicates the database to use for running the testcases, supported values are 'hsqldb', 'derby',
- 'mysql', 'postgres' and 'oracle'; default value is 'hsqldb'. For each database there is
- =core/src/test/resources/DATABASE-oozie-site.xml= file preconfigured.
-
-*oozie.test.properties* (*): indicates the file to load the test properties from, by default is =test.properties=.
-Having this option allows having different test properties sets, for example: minicluster, simple & kerberos.
-
-*oozie.test.waitfor.ratio*= : multiplication factor for testcases using waitfor, the ratio is used to adjust the
-effective time out. For slow machines the ratio should be increased. The default value is =1=.
-
-*oozie.test.config.file*= : indicates a custom Oozie configuration file for running the testcases. The specified file
-must be an absolute path. For example, it can be useful to specify different database than HSQL for running the
-testcases.
-
-*oozie.test.hadoop.minicluster*= : indicates if Hadoop minicluster should be started for testcases, default value 'true'
-
-*oozie.test.job.tracker*= : indicates the URI of the JobTracker when using a Hadoop cluster for testing, default value
-'localhost:8021'
-
-*oozie.test.name.node*= : indicates the URI of the NameNode when using a Hadoop cluster for testing, default value
-'hdfs://localhost:8020'
-
-*oozie.test.hadoop.security*= : indicates the type of Hadoop authentication for testing, valid values are 'simple' or
-'kerberos, default value 'simple'
-
-*oozie.test.kerberos.keytab.file*= : indicates the location of the keytab file, default value
-'${user.home}/oozie.keytab'
-
-*oozie.test.kerberos.realm*= : indicates the Kerberos real, default value 'LOCALHOST'
-
-*oozie.test.kerberos.oozie.principal*= : indicates the Kerberos principal for oozie, default value
-'${user.name}/localhost'
-
-*oozie.test.kerberos.jobtracker.principal*= : indicates the Kerberos principal for the JobTracker, default value
-'mapred/localhost'
-
-*oozie.test.kerberos.namenode.principal*= : indicates the Kerberos principal for the NameNode, default value
-'hdfs/localhost'
-
-*oozie.test.user.oozie*= : specifies the user ID used to start Oozie server in testcases, default value
-is =${user.name}=.
-
-*oozie.test.user.test*= : specifies primary user ID used as the user submitting jobs to Oozie Server in testcases,
-default value is =test=.
-
-*oozie.test.user.test2*= : specifies secondary user ID used as the user submitting jobs to Oozie Server in testcases,
-default value is =test2=.
-
-*oozie.test.user.test3*= : specifies secondary user ID used as the user submitting jobs to Oozie Server in testcases,
-default value is =test3=.
-
-*oozie.test.group*= : specifies group ID used as group when submitting jobs to Oozie Server in testcases,
-default value is =testg=.
-
-NOTE: The users/group specified in *oozie.test.user.test2*, *oozie.test.user.test3*= and *oozie.test.user.group*=
-are used for the authorization testcases only.
-
-*oozie.test.dir*= : specifies the directory where the =oozietests= directory will be created, default value is =/tmp=.
-The =oozietests= directory is used by testcases when they need a local filesystem directory.
-
-*hadoop.log.dir*= : specifies the directory where Hadoop minicluster will write its logs during testcases, default
-value is =/tmp=.
-
-*test.exclude*= : specifies a testcase class (just the class name) to exclude for the tests run, for example =TestSubmitCommand=.
-
-*test.exclude.pattern*= : specifies one or more patterns for testcases to exclude, for example =**/Test*Command.java=.
-
----+++ Testing Map Reduce Pipes Action
-
-Pipes testcases require Hadoop's *wordcount-simple* pipes binary example to run. The *wordcount-simple* pipes binary
-should be compiled for the build platform and copied into Oozie's *core/src/test/resources/* directory. The binary file
-must be named *wordcount-simple*.
-
-If the *wordcount-simple* pipes binary file is not available the testcase will do a NOP and it will print to its output
-file the following message 'SKIPPING TEST: TestPipesMain, binary 'wordcount-simple' not available in the classpath'.
-
-There are 2 testcases that use the *wordcount-simple* pipes binary, *TestPipesMain* and *TestMapReduceActionExecutor*,
-the 'SKIPPING TEST..." message would appear in the testcase log file of both testcases.
-
----+++ Testing using dist_test and grind
-
-Testing using [[https://github.com/cloudera/dist_test][dist_test]] framework with
-[[https://github.com/cloudera/dist_test/blob/master/docs/grind.md][grind]] front end might not work using the default 3.0.2
-version of the maven dependency plugin. It is necessary to downgrade to version 2.10 using
-<code>-Dmaven-dependency-plugin.version=2.10</code> .
-
-Maven flags for grind can be specified using <code>GRIND_MAVEN_FLAGS</code> environment variable:
-
-<verbatim>
-export GRIND_MAVEN_FLAGS=-Dmaven.dependency.plugin.version=2.10
-grind test --java-version 8
-</verbatim>
-
----++ Building an Oozie Distribution
-
-An Oozie distribution bundles an embedded Jetty server.
-
-The simplest way to build Oozie is to run the =mkdistro.sh= script:
-<verbatim>
-$ bin/mkdistro.sh [-DskipTests]
-Running =mkdistro.sh= will create the binary distribution of Oozie. The following options are available to customise
-the versions of the dependencies:
--Puber - Bundle required hadoop and hcatalog libraries in oozie war
--Dhadoop.version=<version> - default 2.6.0
--Ptez - Bundle tez jars in hive and pig sharelibs. Useful if you want to use tez
-as the execution engine for those applications.
--Dpig.version=<version> - default 0.16.0
--Dpig.classifier=<classifier> - default h2
--Dsqoop.version=<version> - default 1.4.3
--Dsqoop.classifier=<classifier> - default hadoop100
--jetty.version=<version> - default 9.3.20.v20170531
--Dopenjpa.version=<version> - default 2.2.2
--Dxerces.version=<version> - default 2.10.0
--Dcurator.version=<version> - default 2.5.0
--Dhive.version=<version> - default 1.2.0
--Dhbase.version=<version> - default 1.2.3
--Dtez.version=<version> - default 0.8.4
-</verbatim>
-
-*IMPORTANT:* Profile hadoop-3 must be activated if building against Hadoop 3
-
-The following properties should be specified when building a release:
-
- * -DgenerateDocs : forces the generation of Oozie documentation
- * -Dbuild.time= : timestamps the distribution
- * -Dvc.revision= : specifies the source control revision number of the distribution
- * -Dvc.url= : specifies the source control URL of the distribution
-
-The provided <code>bin/mkdistro.sh</code> script runs the above Maven invocation setting all these properties to the
-right values (the 'vc.*' properties are obtained from the local git repository).
-
----++ IDE Setup
-
-Eclipse and IntelliJ can use directly Oozie Maven project files.
-
-The only special consideration is that the following source directories from the =client= module must be added to
-the =core= module source path:
-
- * =client/src/main/java= : as source directory
- * =client/src/main/resources= : as source directory
- * =client/src/test/java= : as test-source directory
- * =client/src/test/resources= : as test-source directory
-
-[[index][::Go back to Oozie Documentation Index::]]
-
-</noautolink>
[10/11] oozie git commit: OOZIE-2734 [docs] Switch from TWiki to
Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/AG_Install.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/AG_Install.twiki b/docs/src/site/twiki/AG_Install.twiki
index b8031c8..270b98f 100644
--- a/docs/src/site/twiki/AG_Install.twiki
+++ b/docs/src/site/twiki/AG_Install.twiki
@@ -1,51 +1,52 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
----+!! Oozie Installation and Configuration
+[::Go back to Oozie Documentation Index::](index.html)
-%TOC%
+# Oozie Installation and Configuration
----++ Basic Setup
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-Follow the instructions at [[DG_QuickStart][Oozie Quick Start]].
+## Basic Setup
----++ Environment Setup
+Follow the instructions at [Oozie Quick Start](DG_QuickStart.html).
-*IMPORTANT:* Oozie ignores any set value for =OOZIE_HOME=, Oozie computes its home automatically.
+## Environment Setup
-When running Oozie with its embedded Jetty server, the =conf/oozie-env.sh= file can be
+**IMPORTANT:** Oozie ignores any set value for `OOZIE_HOME`, Oozie computes its home automatically.
+
+When running Oozie with its embedded Jetty server, the `conf/oozie-env.sh` file can be
used to configure the following environment variables used by Oozie:
-*JETTY_OPTS* : settings for the Embedded Jetty that runs Oozie. Java System properties
+**JETTY_OPTS** : settings for the Embedded Jetty that runs Oozie. Java System properties
for Oozie should be specified in this variable. No default value.
-*OOZIE_CONFIG_FILE* : Oozie configuration file to load from Oozie configuration directory.
-Default value =oozie-site.xml=.
+**OOZIE_CONFIG_FILE** : Oozie configuration file to load from Oozie configuration directory.
+Default value `oozie-site.xml`.
-*OOZIE_LOGS* : Oozie logs directory. Default value =logs/= directory in the Oozie installation
+**OOZIE_LOGS** : Oozie logs directory. Default value `logs/` directory in the Oozie installation
directory.
-*OOZIE_LOG4J_FILE* : Oozie Log4J configuration file to load from Oozie configuration directory.
-Default value =oozie-log4j.properties=.
+**OOZIE_LOG4J_FILE** : Oozie Log4J configuration file to load from Oozie configuration directory.
+Default value `oozie-log4j.properties`.
-*OOZIE_LOG4J_RELOAD* : Reload interval of the Log4J configuration file, in seconds.
-Default value =10=
+**OOZIE_LOG4J_RELOAD** : Reload interval of the Log4J configuration file, in seconds.
+Default value `10`
-*OOZIE_CHECK_OWNER* : If set to =true=, Oozie setup/start/run/stop scripts will check that the
+**OOZIE_CHECK_OWNER** : If set to `true`, Oozie setup/start/run/stop scripts will check that the
owner of the Oozie installation directory matches the user invoking the script. The default
-value is undefined and interpreted as a =false=.
+value is undefined and interpreted as a `false`.
+
+**OOZIE_INSTANCE_ID** : The instance id of the Oozie server. When using HA, each server instance should have a unique instance id.
+Default value `${OOZIE_HTTP_HOSTNAME}`
-*OOZIE_INSTANCE_ID* : The instance id of the Oozie server. When using HA, each server instance should have a unique instance id.
-Default value =${OOZIE_HTTP_HOSTNAME}=
+## Oozie Server Setup
----++ Oozie Server Setup
+The `oozie-setup.sh` script prepares the embedded Jetty server to run Oozie.
-The =oozie-setup.sh= script prepares the embedded Jetty server to run Oozie.
+The `oozie-setup.sh` script options are:
-The =oozie-setup.sh= script options are:
-<verbatim>
+```
Usage : oozie-setup.sh <Command and OPTIONS>
sharelib create -fs FS_URI [-locallib SHARED_LIBRARY] [-extralib EXTRA_SHARED_LIBRARY] [-concurrency CONCURRENCY]
(create sharelib for oozie,
@@ -76,24 +77,25 @@ Usage : oozie-setup.sh <Command and OPTIONS>
import <file> imports the oozie database from the zip file
created by export
(without options prints this usage information)
-</verbatim>
+```
-If a directory =libext/= is present in Oozie installation directory, the =oozie-setup.sh= script will
-include all JARs in Jetty's =webapp/WEB_INF/lib/= directory.
+If a directory `libext/` is present in Oozie installation directory, the `oozie-setup.sh` script will
+include all JARs in Jetty's `webapp/WEB_INF/lib/` directory.
-If the ExtJS ZIP file is present in the =libext/= directory, it will be added to the Jetty's =webapp/= directory as well.
-The ExtJS library file name be =ext-2.2.zip=.
+If the ExtJS ZIP file is present in the `libext/` directory, it will be added to the Jetty's `webapp/` directory as well.
+The ExtJS library file name be `ext-2.2.zip`.
----+++ Setting Up Oozie with an Alternate Tomcat
+### Setting Up Oozie with an Alternate Tomcat
-Use the =addtowar.sh= script to prepare the Oozie server only if Oozie will run with a different
+Use the `addtowar.sh` script to prepare the Oozie server only if Oozie will run with a different
servlet container than the embedded Jetty provided with the distribution.
-The =addtowar.sh= script adds Hadoop JARs, JDBC JARs and the ExtJS library to the Oozie WAR file.
+The `addtowar.sh` script adds Hadoop JARs, JDBC JARs and the ExtJS library to the Oozie WAR file.
+
+The `addtowar.sh` script options are:
-The =addtowar.sh= script options are:
-<verbatim>
+```
Usage : addtowar <OPTIONS>
Options: -inputwar INPUT_OOZIE_WAR
-outputwar OUTPUT_OOZIE_WAR
@@ -101,21 +103,21 @@ The =addtowar.sh= script options are:
[-extjs EXTJS_PATH]
[-jars JARS_PATH] (multiple JAR path separated by ':')
[-secureWeb WEB_XML_PATH] (path to secure web.xml)
-</verbatim>
+```
-The original =oozie.war= file is in the Oozie server installation directory.
+The original `oozie.war` file is in the Oozie server installation directory.
-After the Hadoop JARs and the ExtJS library has been added to the =oozie.war= file Oozie is ready to run.
+After the Hadoop JARs and the ExtJS library has been added to the `oozie.war` file Oozie is ready to run.
-Delete any previous deployment of the =oozie.war= from the servlet container (if using Tomcat, delete
-=oozie.war= and =oozie= directory from Tomcat's =webapps/= directory)
+Delete any previous deployment of the `oozie.war` from the servlet container (if using Tomcat, delete
+`oozie.war` and `oozie` directory from Tomcat's `webapps/` directory)
-Deploy the prepared =oozie.war= file (the one that contains the Hadoop JARs and the ExtJS library) in the
-servlet container (if using Tomcat, copy the prepared =oozie.war= file to Tomcat's =webapps/= directory).
+Deploy the prepared `oozie.war` file (the one that contains the Hadoop JARs and the ExtJS library) in the
+servlet container (if using Tomcat, copy the prepared `oozie.war` file to Tomcat's `webapps/` directory).
-*IMPORTANT:* Only one Oozie instance can be deployed per Tomcat instance.
+**IMPORTANT:** Only one Oozie instance can be deployed per Tomcat instance.
----++ Database Configuration
+## Database Configuration
Oozie works with HSQL, Derby, MySQL, Oracle, PostgreSQL or SQL Server databases.
@@ -125,23 +127,24 @@ Oozie bundles the JDBC drivers for HSQL, Embedded Derby and PostgreSQL.
HSQL is normally used for test cases as it is an in-memory database and all data is lost every time Oozie is stopped.
-If using Derby, MySQL, Oracle, PostgreSQL, or SQL Server, the Oozie database schema must be created using the =ooziedb.sh= command
+If using Derby, MySQL, Oracle, PostgreSQL, or SQL Server, the Oozie database schema must be created using the `ooziedb.sh` command
line tool.
-If using MySQL, Oracle, or SQL Server, the corresponding JDBC driver JAR file must be copied to Oozie's =libext/= directory and
-it must be added to Oozie WAR file using the =bin/addtowar.sh= or the =oozie-setup.sh= scripts using the =-jars= option.
+If using MySQL, Oracle, or SQL Server, the corresponding JDBC driver JAR file must be copied to Oozie's `libext/` directory and
+it must be added to Oozie WAR file using the `bin/addtowar.sh` or the `oozie-setup.sh` scripts using the `-jars` option.
-*IMPORTANT:* It is recommended to set the database's timezone to GMT (consult your database's documentation on how to do this).
+**IMPORTANT:** It is recommended to set the database's timezone to GMT (consult your database's documentation on how to do this).
Databases don't handle Daylight Saving Time shifts correctly, and may cause problems if you run any Coordinators with actions
scheduled to materialize during the 1 hour period where we "fall back". For Derby, you can add '-Duser.timezone=GMT'
-to =JETTY_OPTS= in oozie-env.sh to set this. Alternatively, if using MySQL, you can have Oozie use GMT with MySQL without
+to `JETTY_OPTS` in oozie-env.sh to set this. Alternatively, if using MySQL, you can have Oozie use GMT with MySQL without
setting MySQL's timezone to GMT by adding 'useLegacyDatetimeCode=false&serverTimezone=GMT' arguments to the JDBC
-URL, =oozie.service.JPAService.jdbc.url=. Be advised that changing the timezone on an existing Oozie database while Coordinators
+URL, `oozie.service.JPAService.jdbc.url`. Be advised that changing the timezone on an existing Oozie database while Coordinators
are already running may cause Coordinators to shift by the offset of their timezone from GMT once after making this change.
The SQL database used by Oozie is configured using the following configuration properties (default values shown):
-<verbatim>
+
+```
oozie.db.schema.name=oozie
oozie.service.JPAService.create.db.schema=false
oozie.service.JPAService.validate.db.connection=false
@@ -150,19 +153,20 @@ The SQL database used by Oozie is configured using the following configuration p
oozie.service.JPAService.jdbc.username=sa
oozie.service.JPAService.jdbc.password=
oozie.service.JPAService.pool.max.active.conn=10
-</verbatim>
+```
-*NOTE:* If the =oozie.db.schema.create= property is set to =true= (default value is =false=) the Oozie tables
-will be created automatically without having to use the =ooziedb= command line tool. Setting this property to
- =true= it is recommended only for development.
+**NOTE:** If the `oozie.db.schema.create` property is set to `true` (default value is `false`) the Oozie tables
+will be created automatically without having to use the `ooziedb` command line tool. Setting this property to
+ `true` it is recommended only for development.
-*NOTE:* If the =oozie.db.schema.create= property is set to true, the =oozie.service.JPAService.validate.db.connection=
-property value is ignored and Oozie handles it as set to =false=.
+**NOTE:** If the `oozie.db.schema.create` property is set to true, the `oozie.service.JPAService.validate.db.connection`
+property value is ignored and Oozie handles it as set to `false`.
-Once =oozie-site.xml= has been configured with the database configuration execute the =ooziedb.sh= command line tool to
+Once `oozie-site.xml` has been configured with the database configuration execute the `ooziedb.sh` command line tool to
create the database:
-<verbatim>
+
+```
$ bin/ooziedb.sh create -sqlfile oozie.sql -run
Validate DB Connection.
@@ -182,18 +186,19 @@ Oozie DB has been created for Oozie version '3.2.0'
The SQL commands have been written to: oozie.sql
$
-</verbatim>
+```
-NOTE: If using MySQL, Oracle, or SQL Server, copy the corresponding JDBC driver JAR file to the =libext/= directory before running
-the =ooziedb.sh= command line tool.
+NOTE: If using MySQL, Oracle, or SQL Server, copy the corresponding JDBC driver JAR file to the `libext/` directory before running
+the `ooziedb.sh` command line tool.
-NOTE: If instead using the '-run' option, the '-sqlfile <FILE>' option is used, then all the
+NOTE: If instead using the '-run' option, the `-sqlfile <FILE>` option is used, then all the
database changes will be written to the specified file and the database won't be modified.
-If using HSQL there is no need to use the =ooziedb= command line tool as HSQL is an in-memory database. Use the
+If using HSQL there is no need to use the `ooziedb` command line tool as HSQL is an in-memory database. Use the
following configuration properties in the oozie-site.xml:
-<verbatim>
+
+```
oozie.db.schema.name=oozie
oozie.service.JPAService.create.db.schema=true
oozie.service.JPAService.validate.db.connection=false
@@ -202,24 +207,25 @@ following configuration properties in the oozie-site.xml:
oozie.service.JPAService.jdbc.username=sa
oozie.service.JPAService.jdbc.password=
oozie.service.JPAService.pool.max.active.conn=10
-</verbatim>
+```
If you are interested in fine tuning how Oozie can retry database operations on failing database connectivity or errors, you can
set following properties to other values. Here are the default ones:
-<verbatim>
+
+```
oozie.service.JPAService.retry.initial-wait-time.ms=100
oozie.service.JPAService.retry.maximum-wait-time.ms=30000
oozie.service.JPAService.retry.max-retries=10
-</verbatim>
+```
-If you set either =oozie.service.JPAService.retry.max-retries= or =oozie.service.JPAService.retry.maximum-wait-time.ms= to =0=,
+If you set either `oozie.service.JPAService.retry.max-retries` or `oozie.service.JPAService.retry.maximum-wait-time.ms` to `0`,
no retry attempts will be made on any database connectivity issues. Exact settings for these properties depend also on how much load
is on Oozie regarding workflow and coordinator jobs.
-The database operation retry functionality kicks in when there is a =javax.persistence.PersistenceException= those root cause is not
-part of the normal everyday operation - filtered against a blacklist consisting of descendants like =NoSuchResultException=,
-=NonUniqueResultException=, and the like. This way Oozie won't retry database operations on errors that are more related to the
+The database operation retry functionality kicks in when there is a `javax.persistence.PersistenceException` those root cause is not
+part of the normal everyday operation - filtered against a blacklist consisting of descendants like `NoSuchResultException`,
+`NonUniqueResultException`, and the like. This way Oozie won't retry database operations on errors that are more related to the
current query, or otherwise part of the everyday life. This way it's ensured that this blacklist is database agnostic.
It has been tested with a MySQL / failing every minute 10 seconds / an Oozie coordinator job of an Oozie workflow consisting of four
@@ -227,20 +233,21 @@ workflow actions (some of them are asynchronous). On this setup Oozie was recove
To set up such a failing MySQL scenario following has to be performed:
- * Set =oozie.service.JPAService.connection.data.source= to =org.apache.oozie.util.db.BasicDataSourceWrapper=
- within =oozie-site.xml=
- * Set =oozie.service.JPAService.jdbc.driver= to =org.apache.oozie.util.db.FailingMySQLDriverWrapper= within =oozie-site.xml=
+ * Set `oozie.service.JPAService.connection.data.source` to `org.apache.oozie.util.db.BasicDataSourceWrapper`
+ within `oozie-site.xml`
+ * Set `oozie.service.JPAService.jdbc.driver` to `org.apache.oozie.util.db.FailingMySQLDriverWrapper` within `oozie-site.xml`
* Restart Oozie server
* Submit / start some workflows, coordinators etc.
- * See how Oozie is retrying on injected database errors by looking at the Oozie server logs, grepping =JPAException= instances
- with following message prefix: <verbatim>Deliberately failing to prepare statement.</verbatim>
+ * See how Oozie is retrying on injected database errors by looking at the Oozie server logs, grepping `JPAException` instances
+ with following message prefix: `Deliberately failing to prepare statement.`
----++ Database Migration
+## Database Migration
Oozie provides an easy way to switch between databases without losing any data. Oozie servers should be stopped during the
database migration process.
The export of the database can be done using the following command:
-<verbatim>
+
+```
$ bin/oozie-setup.sh export /tmp/oozie_db.zip
1 rows exported from OOZIE_SYS
50 rows exported from WF_JOBS
@@ -251,14 +258,15 @@ $ bin/oozie-setup.sh export /tmp/oozie_db.zip
0 rows exported from BUNDLE_ACTIONS
0 rows exported from SLA_REGISTRATION
0 rows exported from SLA_SUMMARY
-</verbatim>
+```
-The database configuration is read from =oozie-site.xml=. After updating the configuration to point to the new database,
-the tables have to be created with ooziedb.sh in the [[AG_Install#Database_Configuration][Database configuration]]
+The database configuration is read from `oozie-site.xml`. After updating the configuration to point to the new database,
+the tables have to be created with ooziedb.sh in the [Database configuration](AG_Install.html#Database_Configuration)
section above.
Once the tables are created, they can be filled with data using the following command:
-<verbatim>
+
+```
$ bin/oozie-setup.sh import /tmp/oozie_db.zip
Loading to Oozie database version 3
50 rows imported to WF_JOBS
@@ -269,124 +277,128 @@ Loading to Oozie database version 3
0 rows imported to BUNDLE_ACTIONS
0 rows imported to SLA_REGISTRATION
0 rows imported to SLA_SUMMARY
-</verbatim>
+```
NOTE: The database version of the zip must match the version of the Oozie database it's imported to.
After starting the Oozie server, the history and the currently running workflows should be available.
-*IMPORTANT:* The tool was primarily developed to make the migration from embedded databases (e.g. Derby) to standalone databases
+**IMPORTANT:** The tool was primarily developed to make the migration from embedded databases (e.g. Derby) to standalone databases
(e.g. MySQL, PosgreSQL, Oracle, MS SQL Server), though it will work between any supported databases.
-It is *not* optimized to handle databases over 1 Gb. If the database size is larger, it should be purged before migration.
+It is **not** optimized to handle databases over 1 Gb. If the database size is larger, it should be purged before migration.
----++ Oozie Configuration
+## Oozie Configuration
-By default, Oozie configuration is read from Oozie's =conf/= directory
+By default, Oozie configuration is read from Oozie's `conf/` directory
The Oozie configuration is distributed in 3 different files:
- * =oozie-site.xml= : Oozie server configuration
- * =oozie-log4j.properties= : Oozie logging configuration
- * =adminusers.txt= : Oozie admin users list
+ * `oozie-site.xml` : Oozie server configuration
+ * `oozie-log4j.properties` : Oozie logging configuration
+ * `adminusers.txt` : Oozie admin users list
----+++ Oozie Configuration Properties
+### Oozie Configuration Properties
-All Oozie configuration properties and their default values are defined in the =oozie-default.xml= file.
+All Oozie configuration properties and their default values are defined in the `oozie-default.xml` file.
Oozie resolves configuration property values in the following order:
* If a Java System property is defined, it uses its value
- * Else, if the Oozie configuration file (=oozie-site.xml=) contains the property, it uses its value
- * Else, it uses the default value documented in the =oozie-default.xml= file
+ * Else, if the Oozie configuration file (`oozie-site.xml`) contains the property, it uses its value
+ * Else, it uses the default value documented in the `oozie-default.xml` file
-*NOTE:* The =oozie-default.xml= file found in Oozie's =conf/= directory is not used by Oozie, it is there
+**NOTE:** The `oozie-default.xml` file found in Oozie's `conf/` directory is not used by Oozie, it is there
for reference purposes only.
----+++ Precedence of Configuration Properties
+### Precedence of Configuration Properties
For compatibility reasons across Hadoop / Oozie versions, some configuration properties can be defined using multiple keys
in the launcher configuration. Beginning with Oozie 5.0.0, some of them can be overridden, some others will be prepended to default
configuration values.
----++++ Overriding Configuration Values
+#### Overriding Configuration Values
-Overriding happens for following configuration entries with =oozie.launcher= prefix, by switching =oozie.launcher.override=
+Overriding happens for following configuration entries with `oozie.launcher` prefix, by switching `oozie.launcher.override`
(on by default).
For those, following is the general approach:
+
* check whether a YARN compatible entry is present. If yes, use it to override default value
* check whether a MapReduce v2 compatible entry is present. If yes, use it to override default value
* check whether a MapReduce v1 compatible entry is present. If yes, use it to override default value
* use default value
Such properties are (legend: YARN / MapReduce v2 / MapReduce v1):
+
* max attempts of the MapReduce Application Master:
* N / A
- * =mapreduce.map.maxattempts=
- * =mapred.map.max.attempts=
+ * `mapreduce.map.maxattempts`
+ * `mapred.map.max.attempts`
* memory amount in MB of the MapReduce Application Master:
- * =yarn.app.mapreduce.am.resource.mb=
- * =mapreduce.map.memory.mb=
- * =mapred.job.map.memory.mb=
+ * `yarn.app.mapreduce.am.resource.mb`
+ * `mapreduce.map.memory.mb`
+ * `mapred.job.map.memory.mb`
* CPU vcore count of the MapReduce Application Master:
- * =yarn.app.mapreduce.am.resource.cpu-vcores=
- * =mapreduce.map.cpu.vcores=
+ * `yarn.app.mapreduce.am.resource.cpu-vcores`
+ * `mapreduce.map.cpu.vcores`
* N / A
* logging level of the MapReduce Application Master:
* N / A
- * =mapreduce.map.log.level=
- * =mapred.map.child.log.level=
+ * `mapreduce.map.log.level`
+ * `mapred.map.child.log.level`
* MapReduce Application Master JVM options:
- * =yarn.app.mapreduce.am.command-opts=
- * =mapreduce.map.java.opts=
- * =mapred.child.java.opts=
+ * `yarn.app.mapreduce.am.command-opts`
+ * `mapreduce.map.java.opts`
+ * `mapred.child.java.opts`
* MapReduce Application Master environment variable settings:
- * =yarn.app.mapreduce.am.env=
- * =mapreduce.map.env=
- * =mapred.child.env=
+ * `yarn.app.mapreduce.am.env`
+ * `mapreduce.map.env`
+ * `mapred.child.env`
* MapReduce Application Master job priority:
* N / A
- * =mapreduce.job.priority=
- * =mapred.job.priority=
+ * `mapreduce.job.priority`
+ * `mapred.job.priority`
* MapReduce Application Master job queue name:
* N / A
- * =mapreduce.job.queuename=
- * =mapred.job.queue.name=
+ * `mapreduce.job.queuename`
+ * `mapred.job.queue.name`
* MapReduce View ACL settings:
* N / A
- * =mapreduce.job.acl-view-job=
+ * `mapreduce.job.acl-view-job`
* N / A
* MapReduce Modify ACL settings:
* N / A
- * =mapreduce.job.acl-modify-job=
+ * `mapreduce.job.acl-modify-job`
* N / A
This list can be extended or modified by adding new configuration entries or updating existing values
-beginning with =oozie.launcher.override.= within =oozie-site.xml=. Examples can be found in =oozie-default.xml=.
+beginning with `oozie.launcher.override.` within `oozie-site.xml`. Examples can be found in `oozie-default.xml`.
----++++ Prepending Configuration Values
+#### Prepending Configuration Values
-Prepending happens for following configuration entries with =oozie.launcher= prefix, by switching =oozie.launcher.prepend=
+Prepending happens for following configuration entries with `oozie.launcher` prefix, by switching `oozie.launcher.prepend`
(on by default).
For those, following is the general approach:
+
* check whether a YARN compatible entry is present. If yes, use it to prepend to default value
* use default value
Such properties are (legend: YARN only):
- * MapReduce Application Master JVM options: =yarn.app.mapreduce.am.admin-command-opts=
- * MapReduce Application Master environment settings: =yarn.app.mapreduce.am.admin.user.env=
+
+ * MapReduce Application Master JVM options: `yarn.app.mapreduce.am.admin-command-opts`
+ * MapReduce Application Master environment settings: `yarn.app.mapreduce.am.admin.user.env`
This list can be extended or modified by adding new configuration entries or updating existing values
-beginning with =oozie.launcher.prepend.= within =oozie-site.xml=. Examples can be found in =oozie-default.xml=.
+beginning with `oozie.launcher.prepend.` within `oozie-site.xml`. Examples can be found in `oozie-default.xml`.
----+++ Logging Configuration
+### Logging Configuration
-By default, Oozie log configuration is defined in the =oozie-log4j.properties= configuration file.
+By default, Oozie log configuration is defined in the `oozie-log4j.properties` configuration file.
If the Oozie log configuration file changes, Oozie reloads the new settings automatically.
-By default, Oozie logs to Oozie's =logs/= directory.
+By default, Oozie logs to Oozie's `logs/` directory.
Oozie logs in 4 different files:
@@ -395,19 +407,19 @@ Oozie logs in 4 different files:
* oozie-instrumentation.log: instrumentation data, every 60 seconds (configurable)
* oozie-audit.log: audit messages, workflow jobs changes
-The embedded Jetty and embedded Derby log files are also written to Oozie's =logs/= directory.
+The embedded Jetty and embedded Derby log files are also written to Oozie's `logs/` directory.
----+++ Oozie User Authentication Configuration
+### Oozie User Authentication Configuration
Oozie supports Kerberos HTTP SPNEGO authentication, pseudo/simple authentication and anonymous access
for client connections.
-Anonymous access (*default*) does not require the user to authenticate and the user ID is obtained from
+Anonymous access (**default**) does not require the user to authenticate and the user ID is obtained from
the job properties on job submission operations, other operations are anonymous.
Pseudo/simple authentication requires the user to specify the user name on the request, this is done by
-the PseudoAuthenticator class by injecting the =user.name= parameter in the query string of all requests.
-The =user.name= parameter value is taken from the client process Java System property =user.name=.
+the PseudoAuthenticator class by injecting the `user.name` parameter in the query string of all requests.
+The `user.name` parameter value is taken from the client process Java System property `user.name`.
Kerberos HTTP SPNEGO authentication requires the user to perform a Kerberos HTTP SPNEGO authentication sequence.
@@ -419,7 +431,8 @@ This library can be extended to support other authentication mechanisms.
Oozie user authentication is configured using the following configuration properties (default values shown):
-<verbatim>
+
+```
oozie.authentication.type=simple
oozie.authentication.token.validity=36000
oozie.authentication.signature.secret=
@@ -427,63 +440,66 @@ Oozie user authentication is configured using the following configuration proper
oozie.authentication.simple.anonymous.allowed=true
oozie.authentication.kerberos.principal=HTTP/localhost@${local.realm}
oozie.authentication.kerberos.keytab=${oozie.service.HadoopAccessorService.keytab.file}
-</verbatim>
+```
-The =type= defines authentication used for Oozie HTTP endpoint, the supported values are:
+The `type` defines authentication used for Oozie HTTP endpoint, the supported values are:
simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME#.
-The =token.validity= indicates how long (in seconds) an authentication token is valid before it has
+The `token.validity` indicates how long (in seconds) an authentication token is valid before it has
to be renewed.
-The =signature.secret= is the signature secret for signing the authentication tokens. It is recommended to not set this, in which
+The `signature.secret` is the signature secret for signing the authentication tokens. It is recommended to not set this, in which
case Oozie will randomly generate one on startup.
-The =oozie.authentication.cookie.domain= The domain to use for the HTTP cookie that stores the
+The `oozie.authentication.cookie.domain` The domain to use for the HTTP cookie that stores the
authentication token. In order to authentication to work correctly across all Hadoop nodes web-consoles
the domain must be correctly set.
-The =simple.anonymous.allowed= indicates if anonymous requests are allowed. This setting is meaningful
+The `simple.anonymous.allowed` indicates if anonymous requests are allowed. This setting is meaningful
only when using 'simple' authentication.
-The =kerberos.principal= indicates the Kerberos principal to be used for HTTP endpoint.
+The `kerberos.principal` indicates the Kerberos principal to be used for HTTP endpoint.
The principal MUST start with 'HTTP/' as per Kerberos HTTP SPNEGO specification.
-The =kerberos.keytab= indicates the location of the keytab file with the credentials for the principal.
+The `kerberos.keytab` indicates the location of the keytab file with the credentials for the principal.
It should be the same keytab file Oozie uses for its Kerberos credentials for Hadoop.
----+++ Oozie Hadoop Authentication Configuration
+### Oozie Hadoop Authentication Configuration
Oozie works with Hadoop versions which support Kerberos authentication.
Oozie Hadoop authentication is configured using the following configuration properties (default values shown):
-<verbatim>
+
+```
oozie.service.HadoopAccessorService.kerberos.enabled=false
local.realm=LOCALHOST
oozie.service.HadoopAccessorService.keytab.file=${user.home}/oozie.keytab
oozie.service.HadoopAccessorService.kerberos.principal=${user.name}/localhost@{local.realm}
-</verbatim>
+```
The above default values are for a Hadoop 0.20 secure distribution (with support for Kerberos authentication).
To enable Kerberos authentication, the following property must be set:
-<verbatim>
+
+```
oozie.service.HadoopAccessorService.kerberos.enabled=true
-</verbatim>
+```
When using Kerberos authentication, the following properties must be set to the correct values (default values shown):
-<verbatim>
+
+```
local.realm=LOCALHOST
oozie.service.HadoopAccessorService.keytab.file=${user.home}/oozie.keytab
oozie.service.HadoopAccessorService.kerberos.principal=${user.name}/localhost@{local.realm}
-</verbatim>
+```
-*IMPORTANT:* When using Oozie with a Hadoop 20 with Security distribution, the Oozie user in Hadoop must be configured
+**IMPORTANT:** When using Oozie with a Hadoop 20 with Security distribution, the Oozie user in Hadoop must be configured
as a proxy user.
----+++ User ProxyUser Configuration
+### User ProxyUser Configuration
Oozie supports impersonation or proxyuser functionality (identical to Hadoop proxyuser capabilities and conceptually
similar to Unix 'sudo').
@@ -504,7 +520,7 @@ There are 2 configuration properties needed to set up a proxyuser:
Both properties support the '*' wildcard as value. Although this is recommended only for testing/development.
----+++ User Authorization Configuration
+### User Authorization Configuration
Oozie has a basic authorization model:
@@ -519,70 +535,74 @@ If security is disabled all users are admin users.
Oozie security is set via the following configuration property (default value shown):
-<verbatim>
+
+```
oozie.service.AuthorizationService.security.enabled=false
-</verbatim>
+```
NOTE: the old ACL model where a group was provided is still supported if the following property is set
-in =oozie-site.xml=:
+in `oozie-site.xml`:
-<verbatim>
+
+```
oozie.service.AuthorizationService.default.group.as.acl=true
-</verbatim>
+```
----++++ Defining Admin Users
+#### Defining Admin Users
Admin users are determined from the list of admin groups, specified in
- =oozie.service.AuthorizationService.admin.groups= property. Use commas to separate multiple groups, spaces, tabs
+ `oozie.service.AuthorizationService.admin.groups` property. Use commas to separate multiple groups, spaces, tabs
and ENTER characters are trimmed.
If the above property for admin groups is not set, then defining the admin users can happen in the following manners.
-The list of admin users can be in the =conf/adminusers.txt= file. The syntax of this file is:
+The list of admin users can be in the `conf/adminusers.txt` file. The syntax of this file is:
* One user name per line
* Empty lines and lines starting with '#' are ignored
Admin users can also be defined in
-=oozie.serviceAuthorizationService.admin.users= property. Use commas to separate multiple admin users, spaces, tabs
+`oozie.serviceAuthorizationService.admin.users` property. Use commas to separate multiple admin users, spaces, tabs
and ENTER characters are trimmed.
In case there are admin users defined using both methods, the effective list of admin users will be the union
-of the admin users found in the adminusers.txt and those specified with =oozie.serviceAuthorizationService.admin.users=.
+of the admin users found in the adminusers.txt and those specified with `oozie.serviceAuthorizationService.admin.users`.
----++++ Defining Access Control Lists
+#### Defining Access Control Lists
Access Control Lists are defined in the following ways:
- * workflow job submission over CLI: configuration property =group.name= of =job.properties=
- * workflow job submission over HTTP: configuration property =group.name= of the XML submitted over HTTP
- * workflow job re-run: configuration property =oozie.job.acl= (preferred) or configuration property =group.name= of
- =job.properties=
- * coordinator job submission over CLI: configuration property =oozie.job.acl= (preferred) or configuration property =group.name=
- of =job.properties=
- * bundle job submission over CLI: configuration property =oozie.job.acl= (preferred) or configuration property =group.name= of
- =job.properties=
+
+ * workflow job submission over CLI: configuration property `group.name` of `job.properties`
+ * workflow job submission over HTTP: configuration property `group.name` of the XML submitted over HTTP
+ * workflow job re-run: configuration property `oozie.job.acl` (preferred) or configuration property `group.name` of
+ `job.properties`
+ * coordinator job submission over CLI: configuration property `oozie.job.acl` (preferred) or configuration property `group.name`
+ of `job.properties`
+ * bundle job submission over CLI: configuration property `oozie.job.acl` (preferred) or configuration property `group.name` of
+ `job.properties`
For all other workflow, coordinator, or bundle actions the ACL set in beforehand will be used as basis.
Once the ACL for the job is defined, Oozie will check over HDFS whether the user trying to perform a specific action is part of the
-necessary group(s). For implementation details please check out =org.apache.hadoop.security.Groups#getGroups(String user)=.
+necessary group(s). For implementation details please check out `org.apache.hadoop.security.Groups#getGroups(String user)`.
Note that it's enough that the submitting user be part of at least one group of the ACL. Note also that the ACL can contain user
names as well. If there is an ACL defined and the submitting user isn't part of any group or user name present in the ACL, an
-=AuthorizationException= is thrown.
+`AuthorizationException` is thrown.
+
+**Example: A typical ACL setup**
-*Example: A typical ACL setup*
+Detail of `job.properties` on workflow job submission:
-Detail of =job.properties= on workflow job submission:
-<verbatim>
+```
user.name=joe
group.name=marketing,admin,qa,root
-<verbatim>
+```
-HDFS group membership of HDFS user =joe= is =qa=. That is, the check to =org.apache.hadoop.security.Groups#getGroups("joe")= returns
-=qa=. Hence, ACL check will pass inside =AuthorizationService=, because the =user.name= provided belongs to at least of the ACL list
-elements provided as =group.name=.
+HDFS group membership of HDFS user `joe` is `qa`. That is, the check to `org.apache.hadoop.security.Groups#getGroups("joe")` returns
+`qa`. Hence, ACL check will pass inside `AuthorizationService`, because the `user.name` provided belongs to at least of the ACL list
+elements provided as `group.name`.
----+++ Oozie System ID Configuration
+### Oozie System ID Configuration
Oozie has a system ID that is is used to generate the Oozie temporary runtime directory, the workflow job IDs, and the
workflow action IDs.
@@ -591,53 +611,57 @@ Two Oozie systems running with the same ID will not have any conflict but in cas
to identify resources created/used by the different Oozie systems if they have different system IDs (default value
shown):
-<verbatim>
+
+```
oozie.system.id=oozie-${user.name}
-</verbatim>
+```
----+++ Filesystem Configuration
+### Filesystem Configuration
Oozie lets you to configure the allowed Filesystems by using the following configuration property in oozie-site.xml:
-<verbatim>
+
+```
<property>
<name>oozie.service.HadoopAccessorService.supported.filesystems</name>
<value>hdfs</value>
</property>
-</verbatim>
+```
-The above value, =hdfs=, which is the default, means that Oozie will only allow HDFS filesystems to be used. Examples of other
+The above value, `hdfs`, which is the default, means that Oozie will only allow HDFS filesystems to be used. Examples of other
filesystems that Oozie is compatible with are: hdfs, hftp, webhdfs, and viewfs. Multiple filesystems can be specified as
comma-separated values. Putting a * will allow any filesystem type, effectively disabling this check.
----+++ HCatalog Configuration
+### HCatalog Configuration
-Refer to the [[DG_HCatalogIntegration][Oozie HCatalog Integration]] document for a overview of HCatalog and
+Refer to the [Oozie HCatalog Integration](DG_HCatalogIntegration.html) document for a overview of HCatalog and
integration of Oozie with HCatalog. This section explains the various settings to be configured in oozie-site.xml on
the Oozie server to enable Oozie to work with HCatalog.
-*Adding HCatalog jars to Oozie war:*
+**Adding HCatalog jars to Oozie war:**
For Oozie server to talk to HCatalog server, HCatalog and hive jars need to be in the server classpath.
hive-site.xml which has the configuration to talk to the HCatalog server also needs to be in the classpath or specified by the
following configuration property in oozie-site.xml:
-<verbatim>
+
+```
<property>
<name>oozie.service.HCatAccessorService.hcat.configuration</name>
<value>/local/filesystem/path/to/hive-site.xml</value>
</property>
-</verbatim>
+```
The hive-site.xml can also be placed in a location on HDFS and the above property can have a value
-of =hdfs://HOST:PORT/path/to/hive-site.xml= to point there instead of the local file system.
+of `hdfs://HOST:PORT/path/to/hive-site.xml` to point there instead of the local file system.
The oozie-[version]-hcataloglibs.tar.gz in the oozie distribution bundles the required hcatalog and hive jars that
needs to be placed in the Oozie server classpath. If using a version of HCatalog bundled in
Oozie hcataloglibs/, copy the corresponding HCatalog jars from hcataloglibs/ to the libext/ directory. If using a
different version of HCatalog, copy the required HCatalog jars from such version in the libext/ directory.
-This needs to be done before running the =oozie-setup.sh= script so that these jars get added for Oozie.
+This needs to be done before running the `oozie-setup.sh` script so that these jars get added for Oozie.
-*Configure HCatalog URI Handling:*
+**Configure HCatalog URI Handling:**
-<verbatim>
+
+```
<property>
<name>oozie.service.URIHandlerService.uri.handlers</name>
<value>org.apache.oozie.dependency.FSURIHandler,org.apache.oozie.dependency.HCatURIHandler</value>
@@ -645,16 +669,17 @@ This needs to be done before running the =oozie-setup.sh= script so that these j
Enlist the different uri handlers supported for data availability checks.
</description>
</property>
-</verbatim>
+```
The above configuration defines the different uri handlers which check for existence of data dependencies defined in a
-Coordinator. The default value is =org.apache.oozie.dependency.FSURIHandler=. FSURIHandler supports uris with
-schemes defined in the configuration =oozie.service.HadoopAccessorService.supported.filesystems= which are hdfs, hftp
+Coordinator. The default value is `org.apache.oozie.dependency.FSURIHandler`. FSURIHandler supports uris with
+schemes defined in the configuration `oozie.service.HadoopAccessorService.supported.filesystems` which are hdfs, hftp
and webhcat by default. HCatURIHandler supports uris with the scheme as hcat.
-*Configure HCatalog services:*
+**Configure HCatalog services:**
+
-<verbatim>
+```
<property>
<name>oozie.services.ext</name>
<value>
@@ -667,16 +692,17 @@ and webhcat by default. HCatURIHandler supports uris with the scheme as hcat.
Class names must be separated by commas.
</description>
</property>
-</verbatim>
+```
PartitionDependencyManagerService and HCatAccessorService are required to work with HCatalog and support Coordinators
having HCatalog uris as data dependency. If the HCatalog server is configured to publish partition availability
notifications to a JMS compliant messaging provider like ActiveMQ, then JMSAccessorService needs to be added
-to =oozie.services.ext= to handle those notifications.
+to `oozie.services.ext` to handle those notifications.
+
+**Configure JMS Provider JNDI connection mapping for HCatalog:**
-*Configure JMS Provider JNDI connection mapping for HCatalog:*
-<verbatim>
+```
<property>
<name>oozie.service.HCatAccessorService.jmsconnections</name>
<value>
@@ -692,21 +718,22 @@ to =oozie.services.ext= to handle those notifications.
hcat://${1}.${2}.com:8020=java.naming.factory.initial#Dummy.Factory;java.naming.provider.url#tcp://broker.${2}.com:61616
</description>
</property>
-</verbatim>
+```
Currently HCatalog does not provide APIs to get the connection details to connect to the JMS Provider it publishes
notifications to. It only has APIs which provide the topic name in the JMS Provider to which the notifications are
published for a given database table. So the JMS Provider's connection properties needs to be manually configured
-in Oozie using the above setting. You can either provide a =default= JNDI configuration which will be used as the
+in Oozie using the above setting. You can either provide a `default` JNDI configuration which will be used as the
JMS Provider for all HCatalog servers, or can specify a configuration per HCatalog server URL or provide a
configuration based on a rule matching multiple HCatalog server URLs. For example: With the configuration of
-hcat://${1}.${2}.com:8020=java.naming.factory.initial#Dummy.Factory;java.naming.provider.url#tcp://broker.${2}.com:61616,
-request URL of hcat://server1.colo1.com:8020 will map to tcp://broker.colo1.com:61616, hcat://server2.colo2.com:8020
-will map to tcp://broker.colo2.com:61616 and so on.
+`hcat://${1}.${2}.com:8020=java.naming.factory.initial#Dummy.Factory;java.naming.provider.url#tcp://broker.${2}.com:61616`,
+request URL of `hcat://server1.colo1.com:8020 `will map to`tcp://broker.colo1.com:61616`, `hcat://server2.colo2.com:8020`
+will map to`tcp://broker.colo2.com:61616` and so on.
-*Configure HCatalog Polling Frequency:*
+**Configure HCatalog Polling Frequency:**
-<verbatim>
+
+```
<property>
<name>oozie.service.coord.push.check.requeue.interval
</name>
@@ -714,29 +741,30 @@ will map to tcp://broker.colo2.com:61616 and so on.
<description>Command re-queue interval for push dependencies (in millisecond).
</description>
</property>
-</verbatim>
+```
If there is no JMS Provider configured for a HCatalog Server, then oozie polls HCatalog based on the frequency defined
-in =oozie.service.coord.input.check.requeue.interval=. This config also applies to HDFS polling.
+in `oozie.service.coord.input.check.requeue.interval`. This config also applies to HDFS polling.
If there is a JMS provider configured for a HCatalog Server, then oozie polls HCatalog based on the frequency defined
-in =oozie.service.coord.push.check.requeue.interval= as a fallback.
-The defaults for =oozie.service.coord.input.check.requeue.interval= and =oozie.service.coord.push.check.requeue.interval=
+in `oozie.service.coord.push.check.requeue.interval` as a fallback.
+The defaults for `oozie.service.coord.input.check.requeue.interval` and `oozie.service.coord.push.check.requeue.interval`
are 1 minute and 10 minutes respectively.
----+++ Notifications Configuration
+### Notifications Configuration
Oozie supports publishing notifications to a JMS Provider for job status changes and SLA met and miss events. For
-more information on the feature, refer [[DG_JMSNotifications][JMS Notifications]] documentation. Oozie can also send email
+more information on the feature, refer [JMS Notifications](DG_JMSNotifications.html) documentation. Oozie can also send email
notifications on SLA misses.
- * *Message Broker Installation*: <br/>
+ * **Message Broker Installation**: <br/>
For Oozie to send/receive messages, a JMS-compliant broker should be installed. Apache ActiveMQ is a popular JMS-compliant
-broker usable for this purpose. See [[http://activemq.apache.org/getting-started.html][here]] for instructions on
+broker usable for this purpose. See [here](http://activemq.apache.org/getting-started.html) for instructions on
installing and running ActiveMQ.
- * *Services*: <br/>
-Add/modify =oozie.services.ext= property in =oozie-site.xml= to include the following services.
- <verbatim>
+ * **Services**: <br/>
+Add/modify `oozie.services.ext` property in `oozie-site.xml` to include the following services.
+
+```
<property>
<name>oozie.services.ext</name>
<value>
@@ -746,10 +774,11 @@ Add/modify =oozie.services.ext= property in =oozie-site.xml= to include the foll
org.apache.oozie.sla.service.SLAService
</value>
</property>
- </verbatim>
+```
+
+ * **Event Handlers**: <br/>
- * *Event Handlers*: <br/>
- <verbatim>
+```
<property>
<name>oozie.service.EventHandlerService.event.listeners</name>
<value>
@@ -759,30 +788,33 @@ Add/modify =oozie.services.ext= property in =oozie-site.xml= to include the foll
org.apache.oozie.sla.listener.SLAEmailEventListener
</value>
</property>
- </verbatim>
- It is also recommended to increase =oozie.service.SchedulerService.threads= to 15 for faster event processing and sending notifications. The services and their functions are as follows: <br/>
+```
+It is also recommended to increase `oozie.service.SchedulerService.threads` to 15 for faster event processing and sending notifications. The services and their functions are as follows: <br/>
JMSJobEventListener - Sends JMS job notifications <br/>
JMSSLAEventListener - Sends JMS SLA notifications <br/>
SLAEmailEventListener - Sends Email SLA notifications <br/>
SLAJobEventListener - Processes job events and calculates SLA. Does not send any notifications
- * *JMS properties*: <br/>
-Add =oozie.jms.producer.connection.properties= property in =oozie-site.xml=. Its value corresponds to an
+
+ * **JMS properties**: <br/>
+Add `oozie.jms.producer.connection.properties` property in `oozie-site.xml`. Its value corresponds to an
identifier (e.g. default) assigned to a semi-colon separated key#value list of properties from your JMS broker's
-=jndi.properties= file. The important properties are =java.naming.factory.initial= and =java.naming.provider.url=.
+`jndi.properties` file. The important properties are `java.naming.factory.initial` and `java.naming.provider.url`.
+
+As an example, if using ActiveMQ in local env, the property can be set to
- As an example, if using ActiveMQ in local env, the property can be set to
- <verbatim>
+```
<property>
<name>oozie.jms.producer.connection.properties</name>
<value>
java.naming.factory.initial#org.apache.activemq.jndi.ActiveMQInitialContextFactory;java.naming.provider.url#tcp://localhost:61616;connectionFactoryNames#ConnectionFactory
</value>
</property>
- </verbatim>
- * *JMS Topic name*: <br/>
+```
+ * **JMS Topic name**: <br/>
JMS consumers listen on a particular "topic". Hence Oozie needs to define a topic variable with which to publish messages
about the various jobs.
- <verbatim>
+
+```
<property>
<name>oozie.service.JMSTopicService.topic.name</name>
<value>
@@ -802,10 +834,11 @@ about the various jobs.
For jobs with no defined topic, default topic will be ${username}
</description>
</property>
- </verbatim>
+```
- Another related property is the topic prefix.
- <verbatim>
+Another related property is the topic prefix.
+
+```
<property>
<name>oozie.service.JMSTopicService.topic.prefix</name>
<value></value>
@@ -813,17 +846,17 @@ about the various jobs.
This can be used to append a prefix to the topic in oozie.service.JMSTopicService.topic.name. For eg: oozie.
</description>
</property>
- </verbatim>
+```
----+++ Setting Up Oozie with HTTPS (SSL)
+### Setting Up Oozie with HTTPS (SSL)
-*IMPORTANT*:
+**IMPORTANT**:
The default HTTPS configuration will cause all Oozie URLs to use HTTPS except for the JobTracker callback URLs. This is to simplify
configuration (no changes needed outside of Oozie), but this is okay because Oozie doesn't inherently trust the callbacks anyway;
they are used as hints.
-The related environment variables are explained at [[AG_Install#Environment_Setup][Environment Setup]].
+The related environment variables are explained at [Environment Setup](AG_Install.html#Environment_Setup).
You can use either a certificate from a Certificate Authority or a Self-Signed Certificate. Using a self-signed certificate
requires some additional configuration on each Oozie client machine. If possible, a certificate from a Certificate Authority is
@@ -831,138 +864,147 @@ recommended because it's simpler to configure.
There's also some additional considerations when using Oozie HA with HTTPS.
----++++To use a Self-Signed Certificate
+#### To use a Self-Signed Certificate
There are many ways to create a Self-Signed Certificate, this is just one way. We will be using
-the [[http://docs.oracle.com/javase/6/docs/technotes/tools/solaris/keytool.html][keytool]] program, which is
+the [keytool](http://docs.oracle.com/javase/6/docs/technotes/tools/solaris/keytool.html) program, which is
included with your JRE. If it's not on your path, you should be able to find it in $JAVA_HOME/bin.
-1. Run the following command (as the Oozie user) to create the keystore file, which will be named =.keystore= and located in the
-Oozie user's home directory.
-<verbatim>
-keytool -genkeypair -alias jetty -keyalg RSA -dname "CN=hostname" -storepass password -keypass password
-</verbatim>
-The =hostname= should be the host name of the Oozie Server or a wildcard on the subdomain it belongs to. Make sure to include
-the "CN=" part. You can change =storepass= and =keypass= values, but they should be the same. If you do want to use something
-other than password, you'll also need to change the value of the =oozie.https.keystore.pass= property in =oozie-site.xml= to
-match; =password= is the default.
-
-For example, if your Oozie server was at oozie.int.example.com, then you would do this:
-<verbatim>
-keytool -genkeypair -alias jetty -keyalg RSA -dname "CN=oozie.int.example.com" -storepass password -keypass password
-</verbatim>
-If you're going to be using Oozie HA, it's simplest if you have a single certificate that all Oozie servers in the HA group can use.
-To do that, you'll need to use a wildcard on the subdomain it belongs to:
-<verbatim>
-keytool -genkeypair -alias jetty -keyalg RSA -dname "CN=*.int.example.com" -storepass password -keypass password
-</verbatim>
-The above would work on any server in the int.example.com domain.
+1. Run the following command (as the Oozie user) to create the keystore file, which will be named `.keystore` and located in the
+ Oozie user's home directory.
+
+ ```
+ keytool -genkeypair -alias jetty -keyalg RSA -dname "CN=hostname" -storepass password -keypass password
+ ```
+ The `hostname` should be the host name of the Oozie Server or a wildcard on the subdomain it belongs to. Make sure to include
+ the "CN=" part. You can change `storepass` and `keypass` values, but they should be the same. If you do want to use something
+ other than password, you'll also need to change the value of the `oozie.https.keystore.pass` property in `oozie-site.xml` to
+ match; `password` is the default.
+
+ For example, if your Oozie server was at oozie.int.example.com, then you would do this:
+
+ ```
+ keytool -genkeypair -alias jetty -keyalg RSA -dname "CN=oozie.int.example.com" -storepass password -keypass password
+ ```
+ If you're going to be using Oozie HA, it's simplest if you have a single certificate that all Oozie servers in the HA group can use.
+ To do that, you'll need to use a wildcard on the subdomain it belongs to:
+
+ ```
+ keytool -genkeypair -alias jetty -keyalg RSA -dname "CN=*.int.example.com" -storepass password -keypass password
+ ```
+ The above would work on any server in the int.example.com domain.
2. Run the following command (as the Oozie user) to export a certificate file from the keystore file:
-<verbatim>
-keytool -exportcert -alias jetty -file path/to/anywhere/certificate.cert -storepass password
-</verbatim>
+
+ ```
+ keytool -exportcert -alias jetty -file path/to/anywhere/certificate.cert -storepass password
+ ```
3. Run the following command (as any user) to create a truststore containing the certificate we just exported:
-<verbatim>
-keytool -import -alias jetty -file path/to/certificate.cert -keystore /path/to/anywhere/oozie.truststore -storepass password2
-</verbatim>
-You'll need the =oozie.truststore= later if you're using the Oozie client (or other Java-based client); otherwise, you can skip
-this step. The =storepass= value here is only used to verify or change the truststore and isn't typically required when only
-reading from it; so it does not have to be given to users only using the client.
----++++To use a Certificate from a Certificate Authority
+ ```
+ keytool -import -alias jetty -file path/to/certificate.cert -keystore /path/to/anywhere/oozie.truststore -storepass password2
+ ```
+ You'll need the `oozie.truststore` later if you're using the Oozie client (or other Java-based client); otherwise, you can skip
+ this step. The `storepass` value here is only used to verify or change the truststore and isn't typically required when only
+ reading from it; so it does not have to be given to users only using the client.
+
+#### To use a Certificate from a Certificate Authority
1. You will need to make a request to a Certificate Authority in order to obtain a proper Certificate; please consult a Certificate
-Authority on this procedure. If you're going to be using Oozie HA, it's simplest if you have a single certificate that all Oozie
-servers in the HA group can use. To do that, you'll need to use a wild on the subdomain it belongs to (e.g. "*.int.example.com").
+ Authority on this procedure. If you're going to be using Oozie HA, it's simplest if you have a single certificate that all Oozie
+ servers in the HA group can use. To do that, you'll need to use a wild on the subdomain it belongs to (e.g. "*.int.example.com").
2. Once you have your .cert file, run the following command (as the Oozie user) to create a keystore file from your certificate:
-<verbatim>
-keytool -import -alias jetty -file path/to/certificate.cert
-</verbatim>
-The keystore file will be named =.keystore= and located in the Oozie user's home directory.
----++++Configure the Oozie Server to use SSL (HTTPS)
+ ```
+ keytool -import -alias jetty -file path/to/certificate.cert
+ ```
+ The keystore file will be named `.keystore` and located in the Oozie user's home directory.
+
+#### Configure the Oozie Server to use SSL (HTTPS)
1. Make sure the Oozie server isn't running
-2. Configure settings necessary for enabling SSL/TLS support in =oozie-site.xml=.
+2. Configure settings necessary for enabling SSL/TLS support in `oozie-site.xml`.
+
+ 2a. Set `oozie.https.enabled` to `true`. To revert back to HTTP, set `oozie.https.enabled` to `false`.
-2a. Set =oozie.https.enabled= to =true=. To revert back to HTTP, set =oozie.https.enabled= to =false=.
-2b. Set location and password for the keystore and location for truststore by setting =oozie.https.keystore.file=,
-=oozie.https.keystore.pass=, =oozie.https.truststore.file=.
+ 2b. Set location and password for the keystore and location for truststore by setting `oozie.https.keystore.file`,
+ `oozie.https.keystore.pass`, `oozie.https.truststore.file`.
-*Note:* =oozie.https.truststore.file= can be overridden by setting =javax.net.ssl.trustStore= system property.
+ **Note:** `oozie.https.truststore.file` can be overridden by setting `javax.net.ssl.trustStore` system property.
-The default HTTPS port Oozie listens on for secure connections is 11443; it can be changed via =oozie.https.port=.
+ The default HTTPS port Oozie listens on for secure connections is 11443; it can be changed via `oozie.https.port`.
-It is possible to specify other HTTPS settings via =oozie-site.xml=:
-- To include / exclude cipher suites, set =oozie.https.include.cipher.suites= / =oozie.https.exclude.cipher.suites=.
-- To include / exclude TLS protocols, set =oozie.https.include.protocols= / =oozie.https.exclude.protocols=.
-*Note:* Exclude is always preferred over include (i.e. if you both include and exclude an entity, it will be excluded).
+ It is possible to specify other HTTPS settings via `oozie-site.xml`:
+ - To include / exclude cipher suites, set `oozie.https.include.cipher.suites` / `oozie.https.exclude.cipher.suites`.
+ - To include / exclude TLS protocols, set `oozie.https.include.protocols` / `oozie.https.exclude.protocols`.
+ **Note:** Exclude is always preferred over include (i.e. if you both include and exclude an entity, it will be excluded).
3. Start the Oozie server
-*Note:* If using Oozie HA, make sure that each Oozie server has a copy of the .keystore file.
+ **Note:** If using Oozie HA, make sure that each Oozie server has a copy of the .keystore file.
----++++Configure the Oozie Client to connect using SSL (HTTPS)
+#### Configure the Oozie Client to connect using SSL (HTTPS)
The first two steps are only necessary if you are using a Self-Signed Certificate; the third is required either way.
Also, these steps must be done on every machine where you intend to use the Oozie Client.
1. Copy or download the oozie.truststore file onto the client machine
-2. When using any Java-based program, you'll need to pass =-Djavax.net.ssl.trustStore= to the JVM. To
-do this for the Oozie client:
-<verbatim>
-export OOZIE_CLIENT_OPTS='-Djavax.net.ssl.trustStore=/path/to/oozie.truststore'
-</verbatim>
+2. When using any Java-based program, you'll need to pass `-Djavax.net.ssl.trustStore` to the JVM. To
+ do this for the Oozie client:
-3. When using the Oozie Client, you will need to use https://oozie.server.hostname:11443/oozie instead of
-http://oozie.server.hostname:11000/oozie -- Java will not automatically redirect from the http address to the https address.
+ ```
+ export OOZIE_CLIENT_OPTS='-Djavax.net.ssl.trustStore=/path/to/oozie.truststore'
+ ```
----++++Connect to the Oozie Web UI using SSL (HTTPS)
+3. When using the Oozie Client, you will need to use `https://oozie.server.hostname:11443/oozie` instead of
+ `http://oozie.server.hostname:11000/oozie` -- Java will not automatically redirect from the http address to the https address.
-1. Use https://oozie.server.hostname:11443/oozie
-though most browsers should automatically redirect you if you use http://oozie.server.hostname:11000/oozie
+#### Connect to the Oozie Web UI using SSL (HTTPS)
-*IMPORTANT*: If using a Self-Signed Certificate, your browser will warn you that it can't verify the certificate or something
-similar. You will probably have to add your certificate as an exception.
+1. Use `https://oozie.server.hostname:11443/oozie`
+ though most browsers should automatically redirect you if you use `http://oozie.server.hostname:11000/oozie`
----++++Additional considerations for Oozie HA with SSL
+ **IMPORTANT**: If using a Self-Signed Certificate, your browser will warn you that it can't verify the certificate or something
+ similar. You will probably have to add your certificate as an exception.
+
+#### Additional considerations for Oozie HA with SSL
You'll need to configure the load balancer to do SSL pass-through. This will allow the clients talking to Oozie to use the
SSL certificate provided by the Oozie servers (so the load balancer does not need one). Please consult your load balancer's
-documentation on how to configure this. Make sure to point the load balancer at the https://HOST:HTTPS_PORT addresses for your
-Oozie servers. Clients can then connect to the load balancer at https://LOAD_BALANCER_HOST:PORT.
+documentation on how to configure this. Make sure to point the load balancer at the `https://HOST:HTTPS_PORT` addresses for your
+Oozie servers. Clients can then connect to the load balancer at `https://LOAD_BALANCER_HOST:PORT`.
-*Important:* Callbacks from the ApplicationMaster are done via http or https depending on what you enter for the
-=OOZIE_BASE_URL= property. If you are using a Certificate from a Certificate Authority, you can simply put the https address here.
+**Important:** Callbacks from the ApplicationMaster are done via http or https depending on what you enter for the
+`OOZIE_BASE_URL` property. If you are using a Certificate from a Certificate Authority, you can simply put the https address here.
If you are using a self-signed certificate, you have to do one of the following options (Option 1 is recommended):
Option 1) You'll need to follow the steps in
-the [[AG_Install#Configure_the_Oozie_Client_to_connect_using_SSL_HTTPS][Configure the Oozie Client to connect using SSL (HTTPS)]]
-section, but on the host of the ApplicationMaster. You can then set =OOZIE_BASE_URL= to the load balancer https address.
+the [Configure the Oozie Client to connect using SSL (HTTPS)](AG_Install.html#Configure_the_Oozie_Client_to_connect_using_SSL_HTTPS)
+section, but on the host of the ApplicationMaster. You can then set `OOZIE_BASE_URL` to the load balancer https address.
This will allow the ApplicationMaster to contact the Oozie server with https (like the Oozie client, they are also Java
programs).
Option 2) You'll need setup another load balancer, or another "pool" on the existing load balancer, with the http addresses of the
-Oozie servers. You can then set =OOZIE_BASE_URL= to the load balancer http address. Clients should use the https load balancer
+Oozie servers. You can then set `OOZIE_BASE_URL` to the load balancer http address. Clients should use the https load balancer
address. This will allow clients to use https while the ApplicationMaster uses http for callbacks.
----+++ Fine Tuning an Oozie Server
+### Fine Tuning an Oozie Server
-Refer to the [[./oozie-default.xml][oozie-default.xml]] for details.
+Refer to the [oozie-default.xml](./oozie-default.xml) for details.
----+++ Using Instrumentation instead of Metrics
+### Using Instrumentation instead of Metrics
As of version 4.1.0, Oozie includes a replacement for the Instrumentation based on Codahale's Metrics library. It includes a
number of improvements over the original Instrumentation included in Oozie. They both report most of the same information, though
the formatting is slightly different and there's some additional information in the Metrics version; the format of the output to the
oozie-instrumentation log is also different.
-As of version 5.0.0, =MetricsInstrumentationService= is the default one, it's enlisted in =oozie.services=:
- <verbatim>
+As of version 5.0.0, `MetricsInstrumentationService` is the default one, it's enlisted in `oozie.services`:
+
+```
<property>
<name>oozie.services</name>
<value>
@@ -971,11 +1013,12 @@ As of version 5.0.0, =MetricsInstrumentationService= is the default one, it's en
...
</value>
</property>
- </verbatim>
+```
+
+The deprecated `InstrumentationService` can be enabled by adding `InstrumentationService` reference to the list of
+`oozie.services.ext`:
-The deprecated =InstrumentationService= can be enabled by adding =InstrumentationService= reference to the list of
-=oozie.services.ext=:
- <verbatim>
+```
<property>
<name>oozie.services.ext</name>
<value>
@@ -984,19 +1027,20 @@ The deprecated =InstrumentationService= can be enabled by adding =Instrumentatio
...
</value>
</property>
- </verbatim>
+```
-By default the =admin/instrumentation= REST endpoint is no longer be available and instead the =admin/metrics= endpoint can
-be used (see the [[WebServicesAPI#Oozie_Metrics][Web Services API]] documentation for more details); the Oozie Web UI also replaces
+By default the `admin/instrumentation` REST endpoint is no longer be available and instead the `admin/metrics` endpoint can
+be used (see the [Web Services API](WebServicesAPI.html#Oozie_Metrics) documentation for more details); the Oozie Web UI also replaces
the "Instrumentation" tab with a "Metrics" tab.
-If the deprecated =InstrumentationService= is used, the =admin/instrumentation= REST endpoint gets enabled, the =admin/metrics=
-REST endpoint is no longer available (see the [[WebServicesAPI#Oozie_Metrics][Web Services API]] documentation for more details);
+If the deprecated `InstrumentationService` is used, the `admin/instrumentation` REST endpoint gets enabled, the `admin/metrics`
+REST endpoint is no longer available (see the [Web Services API](WebServicesAPI.html#Oozie_Metrics) documentation for more details);
the Oozie Web UI also replaces the "Metrics" tab with the "Instrumentation" tab.
We can also publish the instrumentation metrics to the external server graphite or ganglia. For this the following
properties should be specified in oozie-site.xml :
- <verbatim>
+
+```
<property>
<name>oozie.external_monitoring.enable</name>
<value>false</value>
@@ -1029,11 +1073,12 @@ properties should be specified in oozie-site.xml :
<name>oozie.external_monitoring.reporterIntervalSecs</name>
<value>60</value>
</property>
- </verbatim>
+```
We can also publish the instrumentation metrics via JMX interface. For this the following property should be specified
in oozie-site.xml :
- <verbatim>
+
+```
<property>
<name>oozie.jmx_monitoring.enable</name>
<value>false</value>
@@ -1041,201 +1086,211 @@ in oozie-site.xml :
If the oozie functional metrics needs to be exposed via JMX interface, set it to true.
</description>
</property>>
- </verbatim>
+```
-#HA
----+++ High Availability (HA)
+<a name="HA"></a>
+### High Availability (HA)
Multiple Oozie Servers can be configured against the same database to provide High Availability (HA) of the Oozie service.
----++++ Pre-requisites
+#### Pre-requisites
1. A database that supports multiple concurrent connections. In order to have full HA, the database should also have HA support, or
-it becomes a single point of failure.
+ it becomes a single point of failure.
-*NOTE:* The default derby database does not support this
+ **NOTE:** The default derby database does not support this
2. A ZooKeeper ensemble.
-Apache ZooKeeper is a distributed, open-source coordination service for distributed applications; the Oozie servers use it for
-coordinating access to the database and communicating with each other. In order to have full HA, there should be at least 3
-ZooKeeper servers.
-More information on ZooKeeper can be found [[http://zookeeper.apache.org][here]].
+ Apache ZooKeeper is a distributed, open-source coordination service for distributed applications; the Oozie servers use it for
+ coordinating access to the database and communicating with each other. In order to have full HA, there should be at least 3
+ ZooKeeper servers.
+ More information on ZooKeeper can be found [here](http://zookeeper.apache.org).
3. Multiple Oozie servers.
-*IMPORTANT:* While not strictly required for all configuration properties, all of the servers should ideally have exactly the same
-configuration for consistency's sake.
+ **IMPORTANT:** While not strictly required for all configuration properties, all of the servers should ideally have exactly the same
+ configuration for consistency's sake.
4. A Loadbalancer, Virtual IP, or Round-Robin DNS.
-This is used to provide a single entry-point for users and for callbacks from the JobTracker/ResourceManager. The load balancer
-should be configured for round-robin between the Oozie servers to distribute the requests. Users (using either the Oozie client, a
-web browser, or the REST API) should connect through the load balancer. In order to have full HA, the load balancer should also
-have HA support, or it becomes a single point of failure.
+ This is used to provide a single entry-point for users and for callbacks from the JobTracker/ResourceManager. The load balancer
+ should be configured for round-robin between the Oozie servers to distribute the requests. Users (using either the Oozie client, a
+ web browser, or the REST API) should connect through the load balancer. In order to have full HA, the load balancer should also
+ have HA support, or it becomes a single point of failure.
----++++ Installation/Configuration Steps
+#### Installation/Configuration Steps
1. Install identically configured Oozie servers normally. Make sure they are all configured against the same database and make sure
-that you DO NOT start them yet.
+ that you DO NOT start them yet.
2. Add the following services to the extension services configuration property in oozie-site.xml in all Oozie servers. This will
-make Oozie use the ZooKeeper versions of these services instead of the default implementations.
+ make Oozie use the ZooKeeper versions of these services instead of the default implementations.
-<verbatim>
-<property>
- <name>oozie.services.ext</name>
- <value>
- org.apache.oozie.service.ZKLocksService,
- org.apache.oozie.service.ZKXLogStreamingService,
- org.apache.oozie.service.ZKJobsConcurrencyService,
- org.apache.oozie.service.ZKUUIDService
- </value>
-</property>
-</verbatim>
+
+ ```
+ <property>
+ <name>oozie.services.ext</name>
+ <value>
+ org.apache.oozie.service.ZKLocksService,
+ org.apache.oozie.service.ZKXLogStreamingService,
+ org.apache.oozie.service.ZKJobsConcurrencyService,
+ org.apache.oozie.service.ZKUUIDService
+ </value>
+ </property>
+ ```
3. Add the following property to oozie-site.xml in all Oozie servers. It should be a comma-separated list of host:port pairs of the
-ZooKeeper servers. The default value is shown below.
+ ZooKeeper servers. The default value is shown below.
-<verbatim>
-<property>
- <name>oozie.zookeeper.connection.string</name>
- <value>localhost:2181</value>
-</property>
-</verbatim>
+
+ ```
+ <property>
+ <name>oozie.zookeeper.connection.string</name>
+ <value>localhost:2181</value>
+ </property>
+ ```
4. (Optional) Add the following property to oozie-site.xml in all Oozie servers to specify the namespace to use. All of the Oozie
-Servers that are planning on talking to each other should have the same namespace. If there are multiple Oozie setups each doing
-their own HA, they should have their own namespace. The default value is shown below.
+ Servers that are planning on talking to each other should have the same namespace. If there are multiple Oozie setups each doing
+ their own HA, they should have their own namespace. The default value is shown below.
-<verbatim>
-<property>
- <name>oozie.zookeeper.namespace</name>
- <value>oozie</value>
-</property>
-</verbatim>
-5. Change the value of =OOZIE_BASE_URL= in oozie-site.xml to point to the loadbalancer or virtual IP, for example:
+ ```
+ <property>
+ <name>oozie.zookeeper.namespace</name>
+ <value>oozie</value>
+ </property>
+ ```
-<verbatim>
-<property>
- <name>oozie.base.url</name>
- <value>http://my.loadbalancer.hostname:11000/oozie</value>
-</property>
-</verbatim>
+5. Change the value of `OOZIE_BASE_URL` in oozie-site.xml to point to the loadbalancer or virtual IP, for example:
-6. (Optional) If using a secure cluster, see [[AG_Install#Security][Security]] below on configuring Kerberos with Oozie HA.
+
+ ```
+ <property>
+ <name>oozie.base.url</name>
+ <value>http://my.loadbalancer.hostname:11000/oozie</value>
+ </property>
+ ```
+
+6. (Optional) If using a secure cluster, see [Security](AG_Install.html#Security) below on configuring Kerberos with Oozie HA.
7. Start the ZooKeeper servers.
8. Start the Oozie servers.
-Note: If one of the Oozie servers becomes unavailable, querying Oozie for the logs from a job in the Web UI, REST API, or client may
-be missing information until that server comes back up.
+ Note: If one of the Oozie servers becomes unavailable, querying Oozie for the logs from a job in the Web UI, REST API, or client may
+ be missing information until that server comes back up.
----++++ Security
+#### Security
Oozie HA works with the existing Oozie security framework and settings. For HA features (log streaming, share lib, etc) to work
-properly in a secure setup, following property can be set on each server. If =oozie.server.authentication.type= is not set, then
-server-server authentication will fall back on =oozie.authentication.type=.
+properly in a secure setup, following property can be set on each server. If `oozie.server.authentication.type` is not set, then
+server-server authentication will fall back on `oozie.authentication.type`.
-<verbatim>
+
+```
<property>
<name>oozie.server.authentication.type</name>
<value>kerberos</value>
</property>
-</verbatim>
+```
Below are some additional steps and information specific to Oozie HA:
1. (Optional) To prevent unauthorized users or programs from interacting with or reading the znodes used by Oozie in ZooKeeper,
-you can tell Oozie to use Kerberos-backed ACLs. To enforce this for all of the Oozie-related znodes, simply add the following
-property to oozie-site.xml in all Oozie servers and set it to =true=. The default is =false=.
+ you can tell Oozie to use Kerberos-backed ACLs. To enforce this for all of the Oozie-related znodes, simply add the following
+ property to oozie-site.xml in all Oozie servers and set it to `true`. The default is `false`.
-<verbatim>
-<property>
- <name>oozie.zookeeper.secure</name>
- <value>true</value>
-</property>
-</verbatim>
-Note: The Kerberos principals of each of the Oozie servers should have the same primary name (i.e. in =primary/instance@REALM=, each
-server should have the same value for =primary=).
+ ```
+ <property>
+ <name>oozie.zookeeper.secure</name>
+ <value>true</value>
+ </property>
+ ```
+
+ Note: The Kerberos principals of each of the Oozie servers should have the same primary name (i.e. in `primary/instance@REALM`, each
+ server should have the same value for `primary`).
+
+ **Important:** Once this property is set to `true`, it will set the ACLs on all existing Oozie-related znodes to only allow Kerberos
+ authenticated users with a principal that has the same primary as described above (also for any subsequently created new znodes).
+ This means that if you ever want to turn this feature off, you will have to manually connect to ZooKeeper using a Kerberos principal
+ with the same primary and either delete all znodes under and including the namespace (i.e. if `oozie.zookeeper.namespace` ` `oozie=
+ then that would be `/oozie`); alternatively, instead of deleting them all, you can manually set all of their ACLs to `world:anyone`.
+ In either case, make sure that no Oozie servers are running while this is being done.
-*Important:* Once this property is set to =true=, it will set the ACLs on all existing Oozie-related znodes to only allow Kerberos
-authenticated users with a principal that has the same primary as described above (also for any subsequently created new znodes).
-This means that if you ever want to turn this feature off, you will have to manually connect to ZooKeeper using a Kerberos principal
-with the same primary and either delete all znodes under and including the namespace (i.e. if =oozie.zookeeper.namespace= = =oozie=
-then that would be =/oozie=); alternatively, instead of deleting them all, you can manually set all of their ACLs to =world:anyone=.
-In either case, make sure that no Oozie servers are running while this is being done.
+ Also, in your zoo.cfg for ZooKeeper, make sure to set the following properties:
-Also, in your zoo.cfg for ZooKeeper, make sure to set the following properties:
-<verbatim>
-authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
-kerberos.removeHostFromPrincipal=true
-kerberos.removeRealmFromPrincipal=true
-</verbatim>
+ ```
+ authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
+ kerberos.removeHostFromPrincipal=true
+ kerberos.removeRealmFromPrincipal=true
+ ```
2. Until Hadoop 2.5.0 and later, there is a known limitation where each Oozie server can only use one HTTP principal. However,
-for Oozie HA, we need to use two HTTP principals: =HTTP/oozie-server-host@realm= and =HTTP/load-balancer-host@realm=. This
-allows access to each Oozie server directly and through the load balancer. While users should always go through the load balancer,
-certain features (e.g. log streaming) require the Oozie servers to talk to each other directly; it can also be helpful for an
-administrator to talk directly to an Oozie server. So, if using a Hadoop version prior to 2.5.0, you will have to choose which
-HTTP principal to use as you cannot use both; it is recommended to choose =HTTP/load-balancer-host@realm= so users can connect
-through the load balancer. This will prevent Oozie servers from talking to each other directly, which will effectively disable
-log streaming.
+ for Oozie HA, we need to use two HTTP principals: `HTTP/oozie-server-host@realm` and `HTTP/load-balancer-host@realm`. This
+ allows access to each Oozie server directly and through the load balancer. While users should always go through the load balancer,
+ certain features (e.g. log streaming) require the Oozie servers to talk to each other directly; it can also be helpful for an
+ administrator to talk directly to an Oozie server. So, if using a Hadoop version prior to 2.5.0, you will have to choose which
+ HTTP principal to use as you cannot use both; it is recommended to choose `HTTP/load-balancer-host@realm` so users can connect
+ through the load balancer. This will prevent Oozie servers from talking to each other directly, which will effectively disable
+ log streaming.
-For Hadoop 2.5.0 and later:
+ For Hadoop 2.5.0 and later:
-2a. When creating the keytab used by Oozie, make sure to include Oozie's principal and the two HTTP principals mentioned above.
+ 2a. When creating the keytab used by Oozie, make sure to include Oozie's principal and the two HTTP principals mentioned above.
-2b. Set =oozie.authentication.kerberos.principal= to * (that is, an asterisks) so it will use both HTTP principals.
-For earlier versions of Hadoop:
+ 2b. Set `oozie.authentication.kerberos.principal` to * (that is, an asterisks) so it will use both HTTP principals.
-2a. When creating the keytab used by Oozie, make sure to include Oozie's principal and the load balancer HTTP principal
+ For earlier versions of Hadoop:
-2b. Set =oozie.authentication.kerberos.principal= to =HTTP/load-balancer-host@realm=.
+ 2a. When creating the keytab used by Oozie, make sure to include Oozie's principal and the load balancer HTTP principal
+
+
+ 2b. Set `oozie.authentication.kerberos.principal` to `HTTP/load-balancer-host@realm`.
3. With Hadoop 2.6.0 and later, a rolling random secret that is synchronized across all Oozie servers will be used for signing the
-Oozie auth tokens. This is done automatically when HA is enabled; no additional configuration is needed.
+ Oozie auth tokens. This is done automatically when HA is enabled; no additional configuration is needed.
-For earlier versions of Hadoop, each server will have a different random secret. This will still work but will likely result in
-additional calls to the KDC to authenticate users to the Oozie server (because the auth tokens will not be accepted by other
-servers, which will cause a fallback to Kerberos).
+ For earlier versions of Hadoop, each server will have a different random secret. This will still work but will likely result in
+ additional calls to the KDC to authenticate users to the Oozie server (because the auth tokens will not be accepted by other
+ servers, which will cause a fallback to Kerberos).
4. If you'd like to use HTTPS (SSL) with Oozie HA, there's some additional considerations that need to be made.
-See the [[AG_Install#Setting_Up_Oozie_with_HTTPS_SSL][Setting Up Oozie with HTTPS (SSL)]] section for more information.
+ See the [Setting Up Oozie with HTTPS (SSL)](AG_Install.html#Setting_Up_Oozie_with_HTTPS_SSL) section for more information.
----++++ JobId sequence
+#### JobId sequence
Oozie in HA mode, uses ZK to generate job id sequence. Job Ids are of following format.
-<Id sequence>-<yyMMddHHmmss(server start time)>-<system_id>-<W/C/B>
+`<Id sequence>-<yyMMddHHmmss(server start time)>-<system_id>-<W/C/B>`
-Where, <systemId> is configured as =oozie.system.id= (default is "oozie-" + "user.name")
+Where, `<systemId>` is configured as `oozie.system.id` (default is "oozie-" + "user.name")
W/C/B is suffix to job id indicating that generated job is a type of workflow or coordinator or bundle.
Maximum allowed character for job id sequence is 40. "Id sequence" is stored in ZK and reset to 0 once maximum job id sequence is
-reached. Maximum job id sequence is configured as =oozie.service.ZKUUIDService.jobid.sequence.max=, default value is 99999999990.
+reached. Maximum job id sequence is configured as `oozie.service.ZKUUIDService.jobid.sequence.max`, default value is 99999999990.
+
-<verbatim>
+```
<property>
<name>oozie.service.ZKUUIDService.jobid.sequence.max</name>
<value>99999999990</value>
</property>
-</verbatim>
+```
----++ Starting and Stopping Oozie
+## Starting and Stopping Oozie
Use the standard commands to start and stop Oozie.
----++ Oozie Command Line Installation
+## Oozie Command Line Installation
-Copy and expand the =oozie-client= TAR.GZ file bundled with the distribution. Add the =bin/= directory to the =PATH=.
+Copy and expand the `oozie-client` TAR.GZ file bundled with the distribution. Add the `bin/` directory to the `PATH`.
-Refer to the [[DG_CommandLineTool][Command Line Interface Utilities]] document for a full reference of the =oozie=
+Refer to the [Command Line Interface Utilities](DG_CommandLineTool.html) document for a full reference of the `oozie`
command line tool.
----++ Oozie Share Lib
+## Oozie Share Lib
The Oozie sharelib TAR.GZ file bundled with the distribution contains the necessary files to run Oozie map-reduce streaming, pig,
hive, sqooop, and distcp actions. There is also a sharelib for HCatalog. The sharelib is required for these actions to work; any
@@ -1246,21 +1301,23 @@ classes necessary for the launcher job. If false, Oozie will not do this, and i
respective sharelib jars or the "oozie" sharelib instead. When false, the sharelib is required for ALL actions; when true, the
sharelib is only required for actions that need additional jars (the original list from above).
-<verbatim>
+
+```
<property>
<name>oozie.action.ship.launcher.jar</name>
<value>true</value>
</property>
-</verbatim>
+```
-Using sharelib CLI, sharelib files are copied to new lib_<timestamped> directory. At start, server picks the sharelib from latest
+Using sharelib CLI, sharelib files are copied to new lib_`<timestamped>` directory. At start, server picks the sharelib from latest
time-stamp directory. While starting, server also purges sharelib directory which are older than sharelib retention days
(defined as oozie.service.ShareLibService.temp.sharelib.retention.days and 7 days is default).
Sharelib mapping file can be also configured. Configured file is a key value mapping, where key will be the sharelib name for the
action and value is a comma separated list of DFS or local filesystem directories or jar files. Local filesystem refers to the local
filesystem of the node where the Oozie launcher is running. This can be configured in oozie-site.xml as :
- <verbatim>
+
+```
<!-- OOZIE -->
<property>
<name>oozie.service.ShareLibService.mapping.file</name>
@@ -1277,11 +1334,12 @@ filesystem of the node where the Oozie launcher is running. This can be configur
oozie.hive=file:///usr/local/oozie/share/lib/hive/
</description>
</property>
- </verbatim>
+```
Example mapping file with local filesystem resources:
- <verbatim>
+
+```
<property>
<name>oozie.service.ShareLibService.mapping.file</name>
<value>
@@ -1296,7 +1354,7 @@ Example mapping file with local filesystem resources:
oozie.sqoop=file:///usr/localoozie/share/lib/sqoop
</value>
</property>
- </verbatim>
+```
If you are using local filesystem resources in the mapping file, make sure corresponding jars are already deployed to
all the nodes where Oozie launcher jobs will be executed, and the files are readable by the launchers. To do this, you
@@ -1307,79 +1365,83 @@ with Apache Spark version. Therefore, to run pySpark using Spark Action, user ne
files. These files can be added either to workflow's lib/ directory, to the sharelib or in sharelib mapping file.
----++ Oozie Coordinators/Bundles Processing Timezone
+## Oozie Coordinators/Bundles Processing Timezone
-By default Oozie runs coordinator and bundle jobs using =UTC= timezone for datetime values specified in the application
+By default Oozie runs coordinator and bundle jobs using `UTC` timezone for datetime values specified in the application
XML and in the job parameter properties. This includes coordinator applications start and end times of jobs, coordinator
datasets initial-instance, and bundle applications kickoff times. In addition, coordinator dataset instance URI templates
will be resolved using datetime values of the Oozie processing timezone.
It is possible to set the Oozie processing timezone to a timezone that is an offset of UTC, alternate timezones must
-expressed in using a GMT offset ( =GMT+/-####= ). For example: =GMT+0530= (India timezone).
+expressed in using a GMT offset ( `GMT+/-#### ` ). For example: `GMT+0530` (India timezone).
-To change the default =UTC= timezone, use the =oozie.processing.timezone= property in the =oozie-site.xml=. For example:
+To change the default `UTC` timezone, use the `oozie.processing.timezone` property in the `oozie-site.xml`. For example:
-<verbatim>
+
+```
<configuration>
<property>
<name>oozie.processing.timezone</name>
<value>GMT+0530</value>
</property>
</configuration>
-</verbatim>
+```
-*IMPORTANT:* If using a processing timezone other than =UTC=, all datetime values in coordinator and bundle jobs must
-be expressed in the corresponding timezone, for example =2012-08-08T12:42+0530=.
+**IMPORTANT:** If using a processing timezone other than `UTC`, all datetime values in coord
<TRUNCATED>
[07/11] oozie git commit: OOZIE-2734 [docs] Switch from TWiki to
Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_ActionAuthentication.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_ActionAuthentication.twiki b/docs/src/site/twiki/DG_ActionAuthentication.twiki
index bbf2d57..d280baf 100644
--- a/docs/src/site/twiki/DG_ActionAuthentication.twiki
+++ b/docs/src/site/twiki/DG_ActionAuthentication.twiki
@@ -1,12 +1,12 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
----+!! Action Authentication
+[::Go back to Oozie Documentation Index::](index.html)
-%TOC%
+# Action Authentication
----++ Background
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Background
A secure cluster requires that actions have been authenticated (typically via Kerberos). However, due to the way that Oozie runs
actions, Kerberos credentials are not easily made available to actions launched by Oozie. For many action types, this is not a
@@ -20,17 +20,18 @@ For these situations, Oozie will have to use its Kerberos credentials to obtain
behalf of the user from the service in question. The details of what this means is beyond the scope of this documentation, but
basically, Oozie needs some extra configuration in the workflow so that it can obtain this delegation token.
----++ Oozie Server Configuration
+## Oozie Server Configuration
The code to obtain delegation tokens is pluggable so that it is easy to add support for different services by simply subclassing
org.apache.oozie.action.hadoop.Credentials to retrieve a delegation token from the service and add it to the Configuration.
Out of the box, Oozie already comes with support for some credential types
-(see [[DG_ActionAuthentication#Built-in_Credentials_Implementations][Built-in Credentials Implementations]]).
+(see [Built-in Credentials Implementations](DG_ActionAuthentication.html#Built-in_Credentials_Implementations)).
The credential classes that Oozie should load are specified by the following property in oozie-site.xml. The left hand side of the
equals sign is the type for the credential type, while the right hand side is the class.
-<verbatim>
+
+```
<property>
<name>oozie.credentials.credentialclasses</name>
<value>
@@ -39,19 +40,20 @@ equals sign is the type for the credential type, while the right hand side is th
hive2=org.apache.oozie.action.hadoop.Hive2Credentials
</value>
</property>
-</verbatim>
+```
----++ Workflow Changes
+## Workflow Changes
-The user should add a =credentials= section to the top of their workflow that contains 1 or more =credential= sections. Each of
-these =credential= sections contains a name for the credential, the type for the credential, and any configuration properties
-needed by that type of credential for obtaining a delegation token. The =credentials= section is available in workflow schema
+The user should add a `credentials` section to the top of their workflow that contains 1 or more `credential` sections. Each of
+these `credential` sections contains a name for the credential, the type for the credential, and any configuration properties
+needed by that type of credential for obtaining a delegation token. The `credentials` section is available in workflow schema
version 0.3 and later.
For example, the following workflow is configured to obtain an HCatalog delegation token, which is given to a Pig action so that the
Pig action can talk to a secure HCatalog:
-<verbatim>
+
+```
<workflow-app xmlns='uri:oozie:workflow:0.4' name='pig-wf'>
<credentials>
<credential name='my-hcat-creds' type='hcat'>
@@ -80,41 +82,41 @@ Pig action can talk to a secure HCatalog:
</action>
...
</workflow-app>
-</verbatim>
+```
-The type of the =credential= is "hcat", which is the type name we gave for the HCatCredentials class in oozie-site.xml. We gave
-the =credential= a name, "my-hcat-creds", which can be whatever you want; we then specify cred='my-hcat-creds' in the Pig action,
+The type of the `credential` is "hcat", which is the type name we gave for the HCatCredentials class in oozie-site.xml. We gave
+the `credential` a name, "my-hcat-creds", which can be whatever you want; we then specify cred='my-hcat-creds' in the Pig action,
so that Oozie will include these credentials with the action. You can include multiple credentials with an action by specifying
-a comma-separated list of =credential= names. And finally, the HCatCredentials required two properties (the metastore URI and
+a comma-separated list of `credential` names. And finally, the HCatCredentials required two properties (the metastore URI and
principal), which we also specified.
-Adding the =credentials= section to a workflow and referencing it in an action will make Oozie always try to obtain that delegation
+Adding the `credentials` section to a workflow and referencing it in an action will make Oozie always try to obtain that delegation
token. Ordinarily, this would mean that you cannot re-use this workflow in a non-secure cluster without editing it because trying
-to obtain the delegation token will likely fail. However, you can tell Oozie to ignore the =credentials= for a workflow by setting
-the job-level property =oozie.credentials.skip= to =true=; this will allow you to use the same workflow.xml in a secure and
-non-secure cluster by simply changing the job-level property at runtime. If omitted or set to =false=, Oozie will handle
-the =credentials= section normally. In addition, you can also set this property at the action-level or server-level to skip getting
+to obtain the delegation token will likely fail. However, you can tell Oozie to ignore the `credentials` for a workflow by setting
+the job-level property `oozie.credentials.skip` to `true`; this will allow you to use the same workflow.xml in a secure and
+non-secure cluster by simply changing the job-level property at runtime. If omitted or set to `false`, Oozie will handle
+the `credentials` section normally. In addition, you can also set this property at the action-level or server-level to skip getting
credentials for just that action or for all workflows, respectively. The order of priority is this:
- 1. =oozie.credentials.skip= in the =configuration= section of an action, if set
- 1. =oozie.credentials.skip= in the job.properties for a workflow, if set
- 1. =oozie.credentials.skip= in oozie-site.xml for all workflows, if set
+ 1. `oozie.credentials.skip` in the `configuration` section of an action, if set
+ 1. `oozie.credentials.skip` in the job.properties for a workflow, if set
+ 1. `oozie.credentials.skip` in oozie-site.xml for all workflows, if set
1. (don't skip)
----++ Built-in Credentials Implementations
+## Built-in Credentials Implementations
Oozie currently comes with the following Credentials implementations:
- 1. HCatalog and Hive Metastore: =org.apache.oozie.action.hadoop.HCatCredentials=
- 1. HBase: =org.apache.oozie.action.hadoop.HBaseCredentials=
- 1. Hive Server 2: =org.apache.oozie.action.hadoop.Hive2Credentials=
+ 1. HCatalog and Hive Metastore: `org.apache.oozie.action.hadoop.HCatCredentials`
+ 1. HBase: `org.apache.oozie.action.hadoop.HBaseCredentials`
+ 1. Hive Server 2: `org.apache.oozie.action.hadoop.Hive2Credentials`
HCatCredentials requires these two properties:
- 1. =hcat.metastore.principal= or hive.metastore.kerberos.principal
- 1. =hcat.metastore.uri= or hive.metastore.uris
+ 1. `hcat.metastore.principal` or hive.metastore.kerberos.principal
+ 1. `hcat.metastore.uri` or hive.metastore.uris
-*Note:* The HCatalog Metastore and Hive Metastore are one and the same and so the "hcat" type credential can also be used to talk
+**Note:** The HCatalog Metastore and Hive Metastore are one and the same and so the "hcat" type credential can also be used to talk
to a secure Hive Metastore, though the property names would still start with "hcat.".
HBase does not require any additional properties since the hbase-site.xml on the Oozie server provides necessary information
@@ -122,9 +124,9 @@ to obtain a delegation token; though properties can be overwritten here if desir
Hive2Credentials requires these two properties:
- 1. =hive2.server.principal=
- 1. =hive2.jdbc.url=
+ 1. `hive2.server.principal`
+ 1. `hive2.jdbc.url`
+
+[::Go back to Oozie Documentation Index::](index.html)
-[[index][::Go back to Oozie Documentation Index::]]
-</noautolink>
[06/11] oozie git commit: OOZIE-2734 [docs] Switch from TWiki to
Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/DG_CommandLineTool.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/DG_CommandLineTool.twiki b/docs/src/site/twiki/DG_CommandLineTool.twiki
index 8d33c50..51382c7 100644
--- a/docs/src/site/twiki/DG_CommandLineTool.twiki
+++ b/docs/src/site/twiki/DG_CommandLineTool.twiki
@@ -1,39 +1,42 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
----+!! Command Line Interface Utilities
+[::Go back to Oozie Documentation Index::](index.html)
-%TOC%
+# Command Line Interface Utilities
----++ Introduction
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
-Oozie provides a command line utility, =oozie=, to perform job and admin tasks. All operations are done via
-sub-commands of the =oozie= CLI.
+## Introduction
-The =oozie= CLI interacts with Oozie via its WS API.
+Oozie provides a command line utility, `oozie`, to perform job and admin tasks. All operations are done via
+sub-commands of the `oozie` CLI.
----++ Oozie Command Line Usage
+The `oozie` CLI interacts with Oozie via its WS API.
-<verbatim>
+## Oozie Command Line Usage
+
+
+```
usage:
the env variable 'OOZIE_URL' is used as default value for the '-oozie' option
the env variable 'OOZIE_TIMEZONE' is used as default value for the '-timezone' option
the env variable 'OOZIE_AUTH' is used as default value for the '-auth' option
custom headers for Oozie web services can be specified using '-Dheader:NAME=VALUE'
-</verbatim>
+```
----+++ Oozie basic commands
-<verbatim>
+### Oozie basic commands
+
+```
oozie help : display usage
oozie version : show client version
-</verbatim>
+```
+
+### Oozie job operation commands
----+++ Oozie job operation commands
-<verbatim>
+```
oozie job <OPTIONS> : job operations
-action <arg> coordinator rerun/kill on action ids (requires -rerun/-kill);
@@ -117,10 +120,11 @@ oozie job <OPTIONS> : job operations
prefix defined in
=oozie-site.xml#oozie.client.jobs.application.generated.path= is used. Output is the workflow ID
-</verbatim>
+```
----+++ Oozie jobs operation commands
-<verbatim>
+### Oozie jobs operation commands
+
+```
oozie jobs <OPTIONS> : jobs status
-auth <arg> select authentication type [SIMPLE|KERBEROS]
-doas <arg> doAs user, impersonates as the specified user.
@@ -142,10 +146,11 @@ oozie jobs <OPTIONS> : jobs status
other options, it will resume all the first 50 workflow jobs. Command will fail if one or more
of the jobs is in wrong state.
-verbose verbose mode
-</verbatim>
+```
+
+### Oozie admin operation commands
----+++ Oozie admin operation commands
-<verbatim>
+```
oozie admin <OPTIONS> : admin operations
-auth <arg> select authentication type [SIMPLE|KERBEROS]
-configuration show Oozie system configuration
@@ -165,27 +170,30 @@ oozie admin <OPTIONS> : admin operations
-version show Oozie server build version
-purge <arg> purge old oozie workflow, coordinator and bundle records from DB (parameter unit: day)
wf=<N>\;coord=<N>\;bundle=<N>\;limit=<N>\;oldcoordaction=<true/false>
-</verbatim>
+```
+
+### Oozie validate command
----+++ Oozie validate command
-<verbatim>
+```
oozie validate <OPTIONS> <ARGS> : validate a workflow, coordinator, bundle XML file
-auth <arg> select authentication type [SIMPLE|KERBEROS]
-oozie <arg> Oozie URL
-</verbatim>
+```
----+++ Oozie SLA operation commands
-<verbatim>
+### Oozie SLA operation commands
+
+```
oozie sla <OPTIONS> : sla operations (Deprecated as of Oozie 4.0)
-auth <arg> select authentication type [SIMPLE|KERBEROS]
-len <arg> number of results (default '100', max limited by oozie server setting which defaults to '1000')
-offset <arg> start offset (default '0')
-oozie <arg> Oozie URL
-filter <arg> jobid=<JobID/ActionID>\;appname=<Application Name>
-</verbatim>
+```
+
+### Oozie Pig submit command
----+++ Oozie Pig submit command
-<verbatim>
+```
oozie pig <OPTIONS> -X <ARGS> : submit a pig job, everything after '-X' are pass-through parameters to pig, any '-D' arguments
after '-X' are put in <configuration>
-auth <arg> select authentication type [SIMPLE|KERBEROS]
@@ -195,10 +203,11 @@ oozie pig <OPTIONS> -X <ARGS> : submit a pig job, everything after '-X' are pass
-file <arg> Pig script
-oozie <arg> Oozie URL
-P <property=value> set parameters for script
-</verbatim>
+```
+
+### Oozie Hive submit command
----+++ Oozie Hive submit command
-<verbatim>
+```
oozie hive <OPTIONS> -X<ARGS> : submit a hive job, everything after '-X' are pass-through parameters to hive, any '-D' arguments
after '-X' are put in <configuration>
-auth <arg> select authentication type [SIMPLE|KERBEROS]
@@ -208,10 +217,11 @@ oozie hive <OPTIONS> -X<ARGS> : submit a hive job, everything after '-X' are pa
-file <arg> hive script
-oozie <arg> Oozie URL
-P <property=value> set parameters for script
-</verbatim>
+```
----+++ Oozie Sqoop submit command
-<verbatim>
+### Oozie Sqoop submit command
+
+```
oozie sqoop <OPTIONS> -X<ARGS> : submit a sqoop job, any '-D' arguments after '-X' are put in <configuration>
-auth <arg> select authentication type [SIMPLE|KERBEROS]
-config <arg> job configuration file '.properties'
@@ -219,246 +229,259 @@ oozie sqoop <OPTIONS> -X<ARGS> : submit a sqoop job, any '-D' arguments after '-
-doas <arg> doAs user, impersonates as the specified user
-command <arg> sqoop command
-oozie <arg> Oozie URL
-</verbatim>
+```
+
+### Oozie info command
----+++ Oozie info command
-<verbatim>
+```
oozie info <OPTIONS> : get more detailed info about specific topics
-timezones display a list of available time zones
-</verbatim>
+```
+
+### Oozie MapReduce job command
----+++ Oozie MapReduce job command
-<verbatim>
+```
oozie mapreduce <OPTIONS> : submit a mapreduce job
-auth <arg> select authentication type [SIMPLE|KERBEROS]
-config <arg> job configuration file '.properties'
-D <property=value> set/override value for given property
-doas <arg> doAs user, impersonates as the specified user
-oozie <arg> Oozie URL
-</verbatim>
+```
----++ Common CLI Options
+## Common CLI Options
----+++ Authentication
+### Authentication
-The =oozie= CLI automatically perform authentication if the Oozie server requests it. By default it supports both
+The `oozie` CLI automatically perform authentication if the Oozie server requests it. By default it supports both
pseudo/simple authentication and Kerberos HTTP SPNEGO authentication.
-To perform a specific authentication, the =auth= option with authentication type requests Oozie client to run the
-specified authentication mechanism only. Oozie client provides two types =simple= and =kerberos= which supports =pseudo/simple= and =Kerberos=.
+To perform a specific authentication, the `auth` option with authentication type requests Oozie client to run the
+specified authentication mechanism only. Oozie client provides two types `simple` and `kerberos` which supports `pseudo/simple` and `Kerberos`.
-For pseudo/simple authentication the =oozie= CLI uses the user name of the current OS user.
+For pseudo/simple authentication the `oozie` CLI uses the user name of the current OS user.
-For Kerberos HTTP SPNEGO authentication the =oozie= CLI uses the default principal for the OS Kerberos cache
-(normally the principal that did =kinit=).
+For Kerberos HTTP SPNEGO authentication the `oozie` CLI uses the default principal for the OS Kerberos cache
+(normally the principal that did `kinit`).
Oozie uses Apache Hadoop-Auth (Java HTTP SPNEGO) library for authentication.
This library can be extended to support other authentication mechanisms.
Once authentication is performed successfully the received authentication token is cached in the user home directory
-in the =.oozie-auth-token= file with owner-only permissions. Subsequent requests reuse the cached token while valid.
+in the `.oozie-auth-token` file with owner-only permissions. Subsequent requests reuse the cached token while valid.
-The use of the cache file can be disabled by invoking the =oozie= CLI with the =-Doozie.auth.token.cache=false=
+The use of the cache file can be disabled by invoking the `oozie` CLI with the `-Doozie.auth.token.cache`false=
option.
-To use an custom authentication mechanism, a Hadoop-Auth =Authenticator= implementation must be specified with the
- =-Dauthenticator.class= = =CLASS= option.
+To use an custom authentication mechanism, a Hadoop-Auth `Authenticator` implementation must be specified with the
+ `-Dauthenticator.class=CLASS` option.
----+++ Impersonation, doAs
+### Impersonation, doAs
-The <code>-doas</code> option allows the current user to impersonate other users when interacting with the Oozie
+The `-doas` option allows the current user to impersonate other users when interacting with the Oozie
system. The current user must be configured as a proxyuser in the Oozie system. The proxyuser configuration may
restrict from which hosts a user may impersonate users, as well as users of which groups can be impersonated.
----+++ Oozie URL
+### Oozie URL
-All =oozie= CLI sub-commands expect the <code>-oozie OOZIE_URL</code> option indicating the URL of the Oozie system
-to run the command against. If the OOZIE_URL environment variable has not been set, =oozie= will use the default
-URL specified in oozie-client-env.sh (equivalent to =!http://$(hostname -f):11000/oozie=).
+All `oozie` CLI sub-commands expect the `-oozie OOZIE_URL` option indicating the URL of the Oozie system
+to run the command against. If the OOZIE_URL environment variable has not been set, `oozie` will use the default
+URL specified in oozie-client-env.sh (equivalent to `!http://$(hostname -f):11000/oozie`).
-If the <code>-oozie</code> option is not specified, the =oozie= CLI will look for the =OOZIE_URL= environment variable
+If the `-oozie` option is not specified, the `oozie` CLI will look for the `OOZIE_URL` environment variable
and uses it if set.
-If the option is not provided and the environment variable is not set, the =oozie= CLI will fail.
+If the option is not provided and the environment variable is not set, the `oozie` CLI will fail.
----+++ Time zone
+### Time zone
-The <code>-timezone TIME_ZONE_ID</code> option in the =job= and =jobs= sub-commands allows you to specify the time zone to use in
-the output of those sub-commands. The <code>TIME_ZONE_ID</code> should be one of the standard Java Time Zone IDs. You can get a
-list of the available time zones with the command =oozie info -timezones=.
+The `-timezone TIME_ZONE_ID` option in the `job` and `jobs` sub-commands allows you to specify the time zone to use in
+the output of those sub-commands. The `TIME_ZONE_ID` should be one of the standard Java Time Zone IDs. You can get a
+list of the available time zones with the command `oozie info -timezones`.
-If the <code>-localtime</code> option is used, it will cause Oozie to use whatever the time zone is of the machine. If
-both <code>-localtime</code> and <code>-timezone TIME_ZONE_ID</code> are used, the <code>-localtime</code> option will override
-the <code>-timezone TIME_ZONE_ID</code> option. If neither option is given, Oozie will look for the =OOZIE_TIMEZONE= environment
+If the `-localtime` option is used, it will cause Oozie to use whatever the time zone is of the machine. If
+both `-localtime` and `-timezone TIME_ZONE_ID` are used, the `-localtime` option will override
+the `-timezone TIME_ZONE_ID` option. If neither option is given, Oozie will look for the `OOZIE_TIMEZONE` environment
variable and uses it if set. If neither option is given and the environment variable is not set, or if Oozie is given an invalid
time zone, it will use GMT.
----+++ Debug Mode
+### Debug Mode
-If you export <code>OOZIE_DEBUG=1</code> then the Oozie CLI will output the Web Services API details used by any commands you
+If you export `OOZIE_DEBUG=1` then the Oozie CLI will output the Web Services API details used by any commands you
execute. This is useful for debugging purposes to or see how the Oozie CLI works with the WS API.
----+++ CLI retry
+### CLI retry
Oozie CLI retries connection to Oozie servers for transparent high availability failover when one of the Oozie servers go down.
-=Oozie= CLI command will retry for all commands in case of ConnectException.
-In case of SocketException, all commands except =PUT= and =POST= will have retry logic.
-All job submit are POST call, examples of PUT and POST commands can be find out from [[WebServicesAPI][WebServicesAPI]].
-Retry count can be configured with system property =oozie.connection.retry.count=. Default count is 4.
+`Oozie` CLI command will retry for all commands in case of ConnectException.
+In case of SocketException, all commands except `PUT` and `POST` will have retry logic.
+All job submit are POST call, examples of PUT and POST commands can be find out from [WebServicesAPI](WebServicesAPI.html).
+Retry count can be configured with system property `oozie.connection.retry.count`. Default count is 4.
----++ Job Operations
+## Job Operations
----+++ Submitting a Workflow, Coordinator or Bundle Job
+### Submitting a Workflow, Coordinator or Bundle Job
* Submitting bundle feature is only supported in Oozie 3.0 or later. Similarly, all bundle operation features below are only
supported in Oozie 3.0 or later.
Example:
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -config job.properties -submit
.
job: 14-20090525161321-oozie-joe
-</verbatim>
+```
The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML
-Configuration file (.xml). This file must be specified with the <code>-config</code> option.
+Configuration file (.xml). This file must be specified with the `-config` option.
-The workflow application path must be specified in the file with the =oozie.wf.application.path= property. The
-coordinator application path must be specified in the file with the =oozie.coord.application.path= property.The
-bundle application path must be specified in the file with the =oozie.bundle.application.path= property.
+The workflow application path must be specified in the file with the `oozie.wf.application.path` property. The
+coordinator application path must be specified in the file with the `oozie.coord.application.path` property.The
+bundle application path must be specified in the file with the `oozie.bundle.application.path` property.
Specified path must be an HDFS path.
-The job will be created, but it will not be started, it will be in =PREP= status.
+The job will be created, but it will not be started, it will be in `PREP` status.
----+++ Starting a Workflow, Coordinator or Bundle Job
+### Starting a Workflow, Coordinator or Bundle Job
Example:
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -start 14-20090525161321-oozie-joe
-</verbatim>
+```
-The =start= option starts a previously submitted workflow job or bundle job that is in =PREP= status.
+The `start` option starts a previously submitted workflow job or bundle job that is in `PREP` status.
-After the command is executed the workflow job will be in =RUNNING= status and bundle job will be in =RUNNING=status.
+After the command is executed the workflow job will be in `RUNNING` status and bundle job will be in `RUNNING`status.
-A coordinator job does not support the =start= action. It will show the following error message when trying to start it
+A coordinator job does not support the `start` action. It will show the following error message when trying to start it
via the CLI:
-<verbatim>
+
+```
Error: E0303 : E0303: Invalid parameter value, [action] = [start]
-</verbatim>
+```
----+++ Running a Workflow, Coordinator or Bundle Job
+### Running a Workflow, Coordinator or Bundle Job
Example:
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -config job.properties -run
.
job: 15-20090525161321-oozie-joe
-</verbatim>
+```
-The =run= option creates and starts a workflow job, coordinator job or bundle job.
+The `run` option creates and starts a workflow job, coordinator job or bundle job.
The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML
-Configuration file (.xml). This file must be specified with the <code>-config</code> option.
+Configuration file (.xml). This file must be specified with the `-config` option.
-The workflow application path must be specified in the file with the =oozie.wf.application.path= property. The
-coordinator application path must be specified in the file with the =oozie.coord.application.path= property. The
-bundle application path must be specified in the file with the =oozie.bundle.application.path= property.The
+The workflow application path must be specified in the file with the `oozie.wf.application.path` property. The
+coordinator application path must be specified in the file with the `oozie.coord.application.path` property. The
+bundle application path must be specified in the file with the `oozie.bundle.application.path` property.The
specified path must be an HDFS path.
-The job will be created and it will started, the job will be in =RUNNING= status.
+The job will be created and it will started, the job will be in `RUNNING` status.
----+++ Suspending a Workflow, Coordinator or Bundle Job
+### Suspending a Workflow, Coordinator or Bundle Job
Example:
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -suspend 14-20090525161321-oozie-joe
-</verbatim>
+```
-The =suspend= option suspends a workflow job in =RUNNING= status.
-After the command is executed the workflow job will be in =SUSPENDED= status.
+The `suspend` option suspends a workflow job in `RUNNING` status.
+After the command is executed the workflow job will be in `SUSPENDED` status.
-The =suspend= option suspends a coordinator/bundle job in =RUNNING=, =RUNNINGWITHERROR= or =PREP= status.
-When the coordinator job is suspended, running coordinator actions will stay in running and the workflows will be suspended. If the coordinator job is in =RUNNING=status, it will transit to =SUSPENDED=status; if it is in =RUNNINGWITHERROR=status, it will transit to =SUSPENDEDWITHERROR=; if it is in =PREP=status, it will transit to =PREPSUSPENDED=status.
+The `suspend` option suspends a coordinator/bundle job in `RUNNING`, `RUNNINGWITHERROR` or `PREP` status.
+When the coordinator job is suspended, running coordinator actions will stay in running and the workflows will be suspended. If the coordinator job is in `RUNNING`status, it will transit to `SUSPENDED`status; if it is in `RUNNINGWITHERROR`status, it will transit to `SUSPENDEDWITHERROR`; if it is in `PREP`status, it will transit to `PREPSUSPENDED`status.
-When the bundle job is suspended, running coordinators will be suspended. If the bundle job is in =RUNNING=status, it will transit to =SUSPENDED=status; if it is in =RUNNINGWITHERROR=status, it will transit to =SUSPENDEDWITHERROR=; if it is in =PREP=status, it will transit to =PREPSUSPENDED=status.
+When the bundle job is suspended, running coordinators will be suspended. If the bundle job is in `RUNNING`status, it will transit to `SUSPENDED`status; if it is in `RUNNINGWITHERROR`status, it will transit to `SUSPENDEDWITHERROR`; if it is in `PREP`status, it will transit to `PREPSUSPENDED`status.
----+++ Resuming a Workflow, Coordinator or Bundle Job
+### Resuming a Workflow, Coordinator or Bundle Job
Example:
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -resume 14-20090525161321-oozie-joe
-</verbatim>
+```
-The =resume= option resumes a workflow job in =SUSPENDED= status.
+The `resume` option resumes a workflow job in `SUSPENDED` status.
-After the command is executed the workflow job will be in =RUNNING= status.
+After the command is executed the workflow job will be in `RUNNING` status.
-The =suspend= option suspends a coordinator/bundle job in =SUSPENDED=, =SUSPENDEDWITHERROR= or =PREPSUSPENDED= status.
-If the coordinator job is in =SUSPENDED=status, it will transit to =RUNNING=status; if it is in =SUSPENDEDWITHERROR=status, it will transit to =RUNNINGWITHERROR=; if it is in =PREPSUSPENDED=status, it will transit to =PREP=status.
+The `suspend` option suspends a coordinator/bundle job in `SUSPENDED`, `SUSPENDEDWITHERROR` or `PREPSUSPENDED` status.
+If the coordinator job is in `SUSPENDED`status, it will transit to `RUNNING`status; if it is in `SUSPENDEDWITHERROR`status, it will transit to `RUNNINGWITHERROR`; if it is in `PREPSUSPENDED`status, it will transit to `PREP`status.
When the coordinator job is resumed it will create all the coordinator actions that should have been created during the time
it was suspended, actions will not be lost, they will delayed.
-When the bundle job is resumed, suspended coordinators will resume running. If the bundle job is in =SUSPENDED=status, it will transit to =RUNNING=status; if it is in =SUSPENDEDWITHERROR=status, it will transit to =RUNNINGWITHERROR=; if it is in =PREPSUSPENDED=status, it will transit to =PREP=status.
+When the bundle job is resumed, suspended coordinators will resume running. If the bundle job is in `SUSPENDED`status, it will transit to `RUNNING`status; if it is in `SUSPENDEDWITHERROR`status, it will transit to `RUNNINGWITHERROR`; if it is in `PREPSUSPENDED`status, it will transit to `PREP`status.
----+++ Killing a Workflow, Coordinator or Bundle Job
+### Killing a Workflow, Coordinator or Bundle Job
Example:
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -kill 14-20090525161321-oozie-joe
-</verbatim>
+```
-The =kill= option kills a workflow job in =PREP=, =SUSPENDED= or =RUNNING= status and a coordinator/bundle job in
-=PREP=, =RUNNING=, =PREPSUSPENDED=, =SUSPENDED=, =PREPPAUSED=, or =PAUSED= status.
+The `kill` option kills a workflow job in `PREP`, `SUSPENDED` or `RUNNING` status and a coordinator/bundle job in
+`PREP`, `RUNNING`, `PREPSUSPENDED`, `SUSPENDED`, `PREPPAUSED`, or `PAUSED` status.
-After the command is executed the job will be in =KILLED= status.
+After the command is executed the job will be in `KILLED` status.
----+++ Killing a Coordinator Action or Multiple Actions
+### Killing a Coordinator Action or Multiple Actions
Example:
-<verbatim>
+
+```
$oozie job -kill <coord_Job_id> [-action 1, 3-4, 7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z]
-</verbatim>
+```
- * The =kill= option here for a range of coordinator actions kills a non-terminal (=RUNNING=, =WAITING=, =READY=, =SUSPENDED=) coordinator action when coordinator job is not in =FAILED= or =KILLED= state.
+ * The `kill` option here for a range of coordinator actions kills a non-terminal (`RUNNING`, `WAITING`, `READY`, `SUSPENDED`) coordinator action when coordinator job is not in `FAILED` or `KILLED` state.
* Either -action or -date should be given.
* If neither -action nor -date is given, the exception will be thrown. Also if BOTH -action and -date are given, an error will be thrown.
* Multiple ranges can be used in -action or -date. See the above example.
* If one of the actions in the given list of -action is already in terminal state, the output of this command will only include the other actions.
* The dates specified in -date must be UTC.
* Single date specified in -date must be able to find an action with matched nominal time to be effective.
- * After the command is executed the killed coordinator action will have =KILLED= status.
+ * After the command is executed the killed coordinator action will have `KILLED` status.
----+++ Changing endtime/concurrency/pausetime/status of a Coordinator Job
+### Changing endtime/concurrency/pausetime/status of a Coordinator Job
Example:
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -change 14-20090525161321-oozie-joe -value endtime=2011-12-01T05:00Z\;concurrency=100\;2011-10-01T05:00Z
$ oozie job -oozie http://localhost:11000/oozie -change 0000001-140321155112907-oozie-puru-C -value status=RUNNING
-</verbatim>
+```
-The =endtime/concurrency/pausetime= option changes a coordinator job that is not in =KILLED= status.
+The `endtime/concurrency/pausetime` option changes a coordinator job that is not in `KILLED` status.
Valid value names are:
+
* endtime: the end time of the coordinator job.
* concurrency: the concurrency of the coordinator job.
* pausetime: the pause time of the coordinator job.
* status: new status for coordinator job.
Conditions and usage:
+
* Repeated value names are not allowed.
* New end time should not be before job's start time and last action time.
* If end time is before job start time and if the job has not materialized any actions, then job status is changed to SUCCEEDED.
@@ -470,85 +493,91 @@ Conditions and usage:
After the command is executed the job's end time, concurrency or pause time should be changed. If an already-succeeded job changes its end time, its status will become running.
----+++ Changing endtime/pausetime of a Bundle Job
+### Changing endtime/pausetime of a Bundle Job
Example:
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -change 14-20090525161321-oozie-joe -value pausetime=2011-12-01T05:00Z
-</verbatim>
+```
-The =change= option changes a bundle job that is not in =KILLED= status.
+The `change` option changes a bundle job that is not in `KILLED` status.
Valid value names are:
+
* pausetime: the pause time of the bundle job.
* endtime: the end time of the bundle job.
Repeated value names are not allowed. An empty string "" can be used to reset pause time to none. New end time should not be before job's kickoff time.
-Bundle will execute pause/end date change command on each coordinator job. Refer conditions and usage section of coordinator change command for more details [[DG_CommandLineTool#Changing_endtimeconcurrencypausetimestatus_of_a_Coordinator_Job][Coordinator job change command]].
+Bundle will execute pause/end date change command on each coordinator job. Refer conditions and usage section of coordinator change command for more details [Coordinator job change command](DG_CommandLineTool.html#Changing_endtimeconcurrencypausetimestatus_of_a_Coordinator_Job).
----+++ Rerunning a Workflow Job
+### Rerunning a Workflow Job
Example:
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -config job.properties -rerun 14-20090525161321-oozie-joe
-</verbatim>
+```
-The =rerun= option reruns a completed ( =SUCCEEDED=, =FAILED= or =KILLED= ) job skipping the specified nodes.
+The `rerun` option reruns a completed ( `SUCCEEDED`, `FAILED` or `KILLED` ) job skipping the specified nodes.
The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML
-Configuration file (.xml). This file must be specified with the <code>-config</code> option.
+Configuration file (.xml). This file must be specified with the `-config` option.
-The workflow application path must be specified in the file with the =oozie.wf.application.path= property. The
+The workflow application path must be specified in the file with the `oozie.wf.application.path` property. The
specified path must be an HDFS path.
-The list of nodes to skipped must be provided in the =oozie.wf.rerun.skip.nodes= property separated by commas.
+The list of nodes to skipped must be provided in the `oozie.wf.rerun.skip.nodes` property separated by commas.
-After the command is executed the job will be in =RUNNING= status.
+After the command is executed the job will be in `RUNNING` status.
-Refer to the [[DG_WorkflowReRun][Rerunning Workflow Jobs]] for details on rerun.
+Refer to the [Rerunning Workflow Jobs](DG_WorkflowReRun.html) for details on rerun.
----+++ Rerunning a Coordinator Action or Multiple Actions
+### Rerunning a Coordinator Action or Multiple Actions
Example:
-<verbatim>
+
+```
$oozie job -rerun <coord_Job_id> [-nocleanup] [-refresh] [-failed] [-config <arg>]
[-action 1, 3-4, 7-40] (-action or -date is required to rerun.)
[-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z]
(if neither -action nor -date is given, the exception will be thrown.)
-</verbatim>
+```
-The =rerun= option reruns a terminated (=TIMEDOUT=, =SUCCEEDED=, =KILLED=, =FAILED=, =IGNORED=) coordinator action when coordinator job
-is not in =FAILED= or =KILLED= state.
+The `rerun` option reruns a terminated (`TIMEDOUT`, `SUCCEEDED`, `KILLED`, `FAILED`, `IGNORED`) coordinator action when coordinator job
+is not in `FAILED` or `KILLED` state.
-After the command is executed the rerun coordinator action will be in =WAITING= status.
+After the command is executed the rerun coordinator action will be in `WAITING` status.
-Refer to the [[DG_CoordinatorRerun][Rerunning Coordinator Actions]] for details on rerun.
+Refer to the [Rerunning Coordinator Actions](DG_CoordinatorRerun.html) for details on rerun.
----+++ Rerunning a Bundle Job
+### Rerunning a Bundle Job
Example:
-<verbatim>
+
+```
$oozie job -rerun <bundle_Job_id> [-nocleanup] [-refresh]
[-coordinator c1, c3, c4] (-coordinator or -date is required to rerun.)
[-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z]
(if neither -coordinator nor -date is given, the exception will be thrown.)
-</verbatim>
+```
-The =rerun= option reruns coordinator actions belonging to specified coordinators within the specified date range.
+The `rerun` option reruns coordinator actions belonging to specified coordinators within the specified date range.
-After the command is executed the rerun coordinator action will be in =WAITING= status.
+After the command is executed the rerun coordinator action will be in `WAITING` status.
----+++ Checking the Information and Status of a Workflow, Coordinator or Bundle Job or a Coordinator Action
+### Checking the Information and Status of a Workflow, Coordinator or Bundle Job or a Coordinator Action
Example:
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -info 14-20090525161321-oozie-joe
.
.----------------------------------------------------------------------------------------------------------------------------------------------------------------
@@ -567,36 +596,37 @@ Action Name Type Status Transition External Id
.----------------------------------------------------------------------------------------------------------------------------------------------------------------
hadoop1 map-reduce OK end job_200904281535_0254 SUCCEEDED - 2009-05-26 05:01 +0000 2009-05-26 05:01 +0000
.----------------------------------------------------------------------------------------------------------------------------------------------------------------
-</verbatim>
+```
-The =info= option can display information about a workflow job or coordinator job or coordinator action.
-The =info= option for a Coordinator job will retrieve the Coordinator actions ordered by nominal time. However, the =info= command may timeout if the number of Coordinator actions are very high. In that case, =info= should be used with =offset= and =len= option.
+The `info` option can display information about a workflow job or coordinator job or coordinator action.
+The `info` option for a Coordinator job will retrieve the Coordinator actions ordered by nominal time. However, the `info` command may timeout if the number of Coordinator actions are very high. In that case, `info` should be used with `offset` and `len` option.
-The =offset= and =len= option should be used for pagination. offset determines the start offset of the action
+The `offset` and `len` option should be used for pagination. offset determines the start offset of the action
returned among all the actions that matched the filter criteria. len determines number of actions to be returned.
-The =localtime= option displays times in local time, if not specified times are displayed in GMT.
+The `localtime` option displays times in local time, if not specified times are displayed in GMT.
-The =filter= option can be used to filter coordinator actions based on some criteria.
-The filter option syntax is: <code><key><comparator><value>[;<key><comparator><value>]*</code>.
-(Note escape <code>\</code> needed before semicolon to specify multiple names for filter in shell)
+The `filter` option can be used to filter coordinator actions based on some criteria.
+The filter option syntax is: `<key><comparator><value>[;<key><comparator><value>]*`.
+(Note escape `\` needed before semicolon to specify multiple names for filter in shell)
key: status or nominalTime
-comparator: =, !=, <, <=, >, >=
-value: valid status like SUCCEEDED, KILLED, RUNNING etc. Only = and != apply for status
+comparator: `, !`, <, <`, >, >`
+value: valid status like SUCCEEDED, KILLED, RUNNING etc. Only ` and !` apply for status
value for nominalTime is valid date of the format yyyy-MM-dd'T'HH:mm'Z' (like 2014-06-01T00:00Z)
Multiple values must be specified as different name value pairs. The query is formed by doing AND of all conditions,
with the exception of = which uses OR if there are multiple values for the same key. For example,
-filter 'status=RUNNING;status=WAITING;nominalTime>=2014-06-01T00:00Z' maps to query (status = RUNNING OR status =
-WAITING) AND nominalTime >= 2014-06-01T00:00Z which returns all waiting or running actions with nominalTime >=
+filter 'status`RUNNING;status`WAITING;nominalTime>`2014-06-01T00:00Z' maps to query (status ` RUNNING OR status =
+WAITING) AND nominalTime >` 2014-06-01T00:00Z which returns all waiting or running actions with nominalTime >`
2014-06-01T00:00Z.
-Currently, the filter option can be used only with an =info= option on Coordinator job.
+Currently, the filter option can be used only with an `info` option on Coordinator job.
-The =verbose= option gives more detailed information for all the actions, if checking for workflow job or coordinator job.
-An example below shows how the =verbose= option can be used to gather action statistics information for a job:
+The `verbose` option gives more detailed information for all the actions, if checking for workflow job or coordinator job.
+An example below shows how the `verbose` option can be used to gather action statistics information for a job:
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -info 0000001-111219170928042-oozie-para-W@mr-node -verbose
ID : 0000001-111219170928042-oozie-para-W@mr-node
------------------------------------------------------------------------------------------------------------------------------------
@@ -615,23 +645,24 @@ Ended : 2011-12-20 01:12
External Stats : {"org.apache.hadoop.mapred.JobInProgress$Counter":{"TOTAL_LAUNCHED_REDUCES":1,"TOTAL_LAUNCHED_MAPS":1,"DATA_LOCAL_MAPS":1},"ACTION_TYPE":"MAP_REDUCE","FileSystemCounters":{"FILE_BYTES_READ":1746,"HDFS_BYTES_READ":1409,"FILE_BYTES_WRITTEN":3524,"HDFS_BYTES_WRITTEN":1547},"org.apache.hadoop.mapred.Task$Counter":{"REDUCE_INPUT_GROUPS":33,"COMBINE_OUTPUT_RECORDS":0,"MAP_INPUT_RECORDS":33,"REDUCE_SHUFFLE_BYTES":0,"REDUCE_OUTPUT_RECORDS":33,"SPILLED_RECORDS":66,"MAP_OUTPUT_BYTES":1674,"MAP_INPUT_BYTES":1409,"MAP_OUTPUT_RECORDS":33,"COMBINE_INPUT_RECORDS":0,"REDUCE_INPUT_RECORDS":33}}
External ChildIDs : null
------------------------------------------------------------------------------------------------------------------------------------
-</verbatim>
+```
The two fields External Stats and External ChildIDs display the action statistics information (that includes counter information in case of MR action and PigStats information in case of a pig action) and child ids of the given job.
Note that the user can turn on/off External Stats by specifying the property _oozie.action.external.stats.write_ as _true_ or _false_ in workflow.xml. By default, it is set to false (not to collect External Stats). External ChildIDs will always be stored.
----+++ Listing all the Workflows for a Coordinator Action
+### Listing all the Workflows for a Coordinator Action
A coordinator action kicks off different workflows for its original run and all subsequent reruns.
Getting a list of those workflow ids is a useful tool to keep track of your actions' runs and
to go debug the workflow job logs if required. Along with ids, it also lists their statuses,
and start and end times for quick reference.
-This is achieved by using the Coordinator Action info command and specifying a flag *=allruns=*
-along with the =info= command.
+This is achieved by using the Coordinator Action info command and specifying a flag **`allruns`**
+along with the `info` command.
+
-<verbatim>
+```
$ oozie job -info 0000001-111219170928042-oozie-joe-C@1 -allruns -oozie http://localhost:11000/oozie
.
Job ID Status Started Ended
@@ -642,13 +673,14 @@ Job ID Status Started Ended
.----------------------------------------------------------------------------------------------------
0000001-140324164318985-oozie-joe-W SUCCEEDED 2014-03-24 23:44 GMT 2014-03-24 23:44 GMT
.----------------------------------------------------------------------------------------------------
-</verbatim>
+```
+
+### Listing all retry attempts of a workflow action
----+++ Listing all retry attempts of a workflow action
+When retry-max is specified for an action in the workflow definition, and there is a failure, it will be retried till it succeeds or retry-max attempts are exhausted. To get information on all the retry attempts, `-retries` command can be used.
-When retry-max is specified for an action in the workflow definition, and there is a failure, it will be retried till it succeeds or retry-max attempts are exhausted. To get information on all the retry attempts, =-retries= command can be used.
-<verbatim>
+```
$ oozie job -retries 0000000-161212175234862-oozie-puru-W@pig-node -oozie http://localhost:11000/oozie
ID : 0000000-161212175234862-oozie-puru-W@pig-node
@@ -669,14 +701,15 @@ End Time : Tue, 13 Dec 2016 01:56:27 GMT
Console URL : http://localhost:50030/jobdetails.jsp?jobid=job_201612051339_2650
------------------------------------------------------------------------------------------------------------------------------------
$
-</verbatim>
+```
----+++ Checking the xml definition of a Workflow, Coordinator or Bundle Job
+### Checking the xml definition of a Workflow, Coordinator or Bundle Job
Example:
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -definition 14-20090525161321-oozie-joe
.
<workflow-app xmlns="uri:oozie:workflow:0.2" name="sm3-segment-2413">
@@ -686,23 +719,25 @@ $ oozie job -oozie http://localhost:11000/oozie -definition 14-20090525161321-oo
<end name="end"/>
</workflow-app>
-</verbatim>
+```
----+++ Checking the server logs of a Workflow, Coordinator or Bundle Job
+### Checking the server logs of a Workflow, Coordinator or Bundle Job
Example:
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -log 14-20090525161321-oozie-joe
-</verbatim>
+```
----+++ Checking the server error logs of a Workflow, Coordinator or Bundle Job
+### Checking the server error logs of a Workflow, Coordinator or Bundle Job
Example:
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -errorlog 0000000-150121110331712-oozie-puru-B
2015-01-21 11:33:29,090 WARN CoordSubmitXCommand:523 - SERVER[-] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-150121110331712-oozie-puru-B] ACTION[] SAXException :
org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 22; cvc-complex-type.2.4.a: Invalid content was found starting with element 'concurrency'. One of '{"uri:oozie:coordinator:0.2":controls, "uri:oozie:coordinator:0.2":datasets, "uri:oozie:coordinator:0.2":input-events, "uri:oozie:coordinator:0.2":output-events, "uri:oozie:coordinator:0.2":action}' is expected.
@@ -722,39 +757,42 @@ org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 22; cvc-complex-typ
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.jaxp.validation.StreamValidatorHelper.validate(Unknown Source)
at org.apache.xerces.jaxp.validation.ValidatorImpl.validate(Unknown Source)
-</verbatim>
+```
----+++ Checking the audit logs of a Workflow, Coordinator or Bundle Job
+### Checking the audit logs of a Workflow, Coordinator or Bundle Job
Example:
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -auditlog 0000000-150322000230582-oozie-puru-C
2015-03-22 00:04:35,494 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [start], PARAMETER [null], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
2015-03-22 00:05:13,823 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
2015-03-22 00:06:59,561 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
2015-03-22 23:22:20,012 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
2015-03-22 23:28:48,218 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [resume], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
-</verbatim>
+```
----+++ Checking the server logs for particular actions of a Coordinator Job
+### Checking the server logs for particular actions of a Coordinator Job
Example:
-<verbatim>
+
+```
$ oozie job -log <coord_job_id> [-action 1, 3-4, 7-40] (-action is optional.)
-</verbatim>
+```
----+++ Filtering the server logs with logfilter options
+### Filtering the server logs with logfilter options
User can provide multiple option to filter logs using -logfilter opt1=val1;opt2=val1;opt3=val1. This can be used to fetch only just logs of interest faster as fetching Oozie server logs is slow due to the overhead of pattern matching.
Available options are:
- * recent: Specify recent hours/min of logs to scan. The recent offset specified is taken relative to the =end= time specified, job end time or the current system time if the job is still running in that order of precedence. For eg: recent=3h or recent=30m will fetch logs starting 3 hours/30 minutes before the end time and up to the end time. H/h is used to denote hour and M/m is used to denote minutes. If no suffix is specified default is hours.
- * start: Start time for scanning logs. Default is start time of the job. User can provide a valid date or offset similar to =recent= option. Valid date formats are "yyyy-MM-dd'T'HH:mm'Z'" and "yyyy-MM-dd HH:mm:ss,SSS". When an offset is specified, it is calculated relative to the start time of the job. For eg: start=2h will fetch logs starting 2 hours after the job was started.
- * end: End time for scanning logs. Default is end time of the job or current system time if the job is still running. User can provide a valid date or offset similar to =start= option. When an offset is specified, it is calculated relative to start time i.e job start time . For eg: end=2h will fetch logs from start time and start time plus 2 hours.
+
+ * recent: Specify recent hours/min of logs to scan. The recent offset specified is taken relative to the `end` time specified, job end time or the current system time if the job is still running in that order of precedence. For eg: recent=3h or recent=30m will fetch logs starting 3 hours/30 minutes before the end time and up to the end time. H/h is used to denote hour and M/m is used to denote minutes. If no suffix is specified default is hours.
+ * start: Start time for scanning logs. Default is start time of the job. User can provide a valid date or offset similar to `recent` option. Valid date formats are "yyyy-MM-dd'T'HH:mm'Z'" and "yyyy-MM-dd HH:mm:ss,SSS". When an offset is specified, it is calculated relative to the start time of the job. For eg: start=2h will fetch logs starting 2 hours after the job was started.
+ * end: End time for scanning logs. Default is end time of the job or current system time if the job is still running. User can provide a valid date or offset similar to `start` option. When an offset is specified, it is calculated relative to start time i.e job start time . For eg: end=2h will fetch logs from start time and start time plus 2 hours.
* loglevel : Multiple log levels separated by "|" can be specified. Supported log levels are ALL, DEBUG, ERROR, INFO, TRACE, WARN, FATAL.
* text: String to search in logs.
* limit : Limit number of line to be searched. Log search will end when when n lines(excluding stack-trace) have been matched.
@@ -763,7 +801,8 @@ Available options are:
Examples.
Searching log with log level ERROR or WARN will only give log with Error and Warning (with stack-trace) only.
This will be useful if job has failed and user want to find error logs with exception.
-<verbatim>
+
+```
$ ./oozie job -log 0000006-140319184715726-oozie-puru-W -logfilter loglevel=WARN\;limit=3 -oozie http://localhost:11000/oozie/
2014-03-20 10:01:52,977 WARN ActionStartXCommand:542 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000006-140319184715726-oozie-puru-W] ACTION[0000006-140319184715726-oozie-puru-W@:start:] [***0000006-140319184715726-oozie-puru-W@:start:***]Action status=DONE
@@ -787,20 +826,22 @@ Caused by: java.io.IOException: Queue "aaadefault" does not exist
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3613)
... 12 more
$
-</verbatim>
+```
Search with specific text and recent option.
-<verbatim>
+
+```
$ ./oozie job -log 0000003-140319184715726-oozie-puru-C -logfilter text=Missing\;limit=4\;recent=1h -oozie http://localhost:11000/oozie/
2014-03-20 09:59:50,329 INFO CoordActionInputCheckXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@1] [0000003-140319184715726-oozie-puru-C@1]::CoordActionInputCheck:: Missing deps:hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/
2014-03-20 09:59:50,330 INFO CoordActionInputCheckXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@1] [0000003-140319184715726-oozie-puru-C@1]::ActionInputCheck:: In checkListOfPaths: hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/ is Missing.
2014-03-20 10:02:19,087 INFO CoordActionInputCheckXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@2] [0000003-140319184715726-oozie-puru-C@2]::CoordActionInputCheck:: Missing deps:hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/
2014-03-20 10:02:19,088 INFO CoordActionInputCheckXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@2] [0000003-140319184715726-oozie-puru-C@2]::ActionInputCheck:: In checkListOfPaths: hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/ is Missing.
$
-</verbatim>
+```
Search example with specific date range.
-<verbatim>
+
+```
$ ./oozie job -log 0000003-140319184715726-oozie-puru-C -logfilter "start=2014-03-20 10:00:57,063;end=2014-03-20 10:10:57,063" -oozie http://localhost:11000/oozie/
2014-03-20 10:00:57,063 INFO CoordActionUpdateXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000003-140319184715726-oozie-puru-C] ACTION[0000003-140319184715726-oozie-puru-C@1] Updating Coordinator action id :0000003-140319184715726-oozie-puru-C@1 status to KILLED, pending = 0
2014-03-20 10:02:18,967 INFO CoordMaterializeTransitionXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[aggregator-coord] JOB[0000003-140319184715726-oozie-puru-C] ACTION[-] materialize actions for tz=Coordinated Universal Time,
@@ -811,15 +852,16 @@ $ ./oozie job -log 0000003-140319184715726-oozie-puru-C -logfilter "start=2014-
2014-03-20 10:02:18,971 WARN CoordELFunctions:542 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[aggregator-coord] JOB[0000003-140319184715726-oozie-puru-C] ACTION[-] If the initial instance of the dataset is later than the current-instance specified, such as coord:current(-200) in this case, an empty string is returned. This means that no data is available at the current-instance specified by the user and the user could try modifying his initial-instance to an earlier time.
2014-03-20 10:02:18,975 INFO CoordMaterializeTransitionXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[aggregator-coord] JOB[0000003-140319184715726-oozie-puru-C] ACTION[-] [0000003-140319184715726-oozie-puru-C]: all actions have been materialized, set pending to true
2014-03-20 10:02:18,975 INFO CoordMaterializeTransitionXCommand:539 - SERVER[ ] USER[-] GROUP[-] TOKEN[] APP[aggregator-coord] JOB[0000003-140319184715726-oozie-puru-C] ACTION[-] Coord Job status updated to = RUNNING
-</verbatim>
+```
----+++ Dryrun of Coordinator Job
+### Dryrun of Coordinator Job
* This feature is only supported in Oozie 2.0 or later.
Example:
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -dryrun -config job.properties
***coordJob after parsing: ***
@@ -856,47 +898,49 @@ start="2009-03-06T010:00Z" end="2009-03-20T11:00Z" timezone="America/Los_Angeles
</coordinator-app>
------------------------------------------------------------------------------------------------------------------------------------
-</verbatim>
+```
-The =dryrun= option tests running a coordinator job with given job properties and does not create the job.
+The `dryrun` option tests running a coordinator job with given job properties and does not create the job.
The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML
-Configuration file (.xml). This file must be specified with the <code>-config</code> option.
+Configuration file (.xml). This file must be specified with the `-config` option.
-The coordinator application path must be specified in the file with the =oozie.coord.application.path= property. The
+The coordinator application path must be specified in the file with the `oozie.coord.application.path` property. The
specified path must be an HDFS path.
----+++ Dryrun of Workflow Job
+### Dryrun of Workflow Job
* This feature is only supported in Oozie 3.3.2 or later.
Example:
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -dryrun -config job.properties
OK
-</verbatim>
+```
-The =dryrun= option tests running a workflow job with given job properties and does not create the job.
+The `dryrun` option tests running a workflow job with given job properties and does not create the job.
The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML
-Configuration file (.xml). This file must be specified with the <code>-config</code> option.
+Configuration file (.xml). This file must be specified with the `-config` option.
-The workflow application path must be specified in the file with the =oozie.wf.application.path= property. The
+The workflow application path must be specified in the file with the `oozie.wf.application.path` property. The
specified path must be an HDFS path.
-If the workflow is accepted (i.e. Oozie is able to successfully read and parse it), it will return ="OK"=; otherwise, it will return
+If the workflow is accepted (i.e. Oozie is able to successfully read and parse it), it will return `"OK"`; otherwise, it will return
an error message describing why it was rejected.
----+++ Dryrun of Bundle Job
+### Dryrun of Bundle Job
* This feature is only supported in Oozie 5.1 or later.
Example:
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -dryrun -config job.properties
***Bundle job after parsing: ***
@@ -929,17 +973,17 @@ $ oozie job -oozie http://localhost:11000/oozie -dryrun -config job.properties
</coordinator>
</bundle-app>
-</verbatim>
+```
-The =dryrun= option tests running a bundle job with given job properties and does not create the job.
+The `dryrun` option tests running a bundle job with given job properties and does not create the job.
The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML
-Configuration file (.xml). This file must be specified with the <code>-config</code> option.
+Configuration file (.xml). This file must be specified with the `-config` option.
If the bundle job is accepted (i.e. Oozie is able to successfully read and parse it), it will return the parsed bundle job in xml;
otherwise, it will return an error message describing why it was rejected.
----+++ Updating coordinator definition and properties
+### Updating coordinator definition and properties
Existing coordinator definition will be replaced by new definition. The refreshed coordinator would keep the same coordinator ID, state, and coordinator actions.
All created coord action(including in WAITING) will use old configuration.
One can rerun actions with -refresh option, -refresh option will use new configuration to rerun coord action
@@ -948,13 +992,14 @@ Update command also verifies coordinator definition like submit command, if ther
Update command with -dryrun will show coordinator definition and properties differences.
Config option is optional, if not specified existing coordinator property is used to find coordinator path.
-Update command doesn't allow update of coordinator name, frequency, start time, end time and timezone and will fail on an attempt to change any of them. To change end time of coordinator use the =-change= command.
+Update command doesn't allow update of coordinator name, frequency, start time, end time and timezone and will fail on an attempt to change any of them. To change end time of coordinator use the `-change` command.
To change the entire XML for a running coordinator, hdfs path for the new XML can be specified
-as =oozie.coord.application.path= in job.properties. Then, use <code>-config job.properties</code> in the update command.
+as `oozie.coord.application.path` in job.properties. Then, use `-config job.properties` in the update command.
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -config job.properties -update 0000005-140402104721140-oozie-puru-C -dryrun
.
**********Job definition changes**********
@@ -994,62 +1039,66 @@ $ oozie job -oozie http://localhost:11000/oozie -config job.properties -update 0
<value>localhost:9001</value>
</property>
**********************************
-</verbatim>
+```
----+++ Ignore a Coordinator Job
+### Ignore a Coordinator Job
Example:
-<verbatim>
+
+```
$oozie job -ignore <coord_Job_id>
-</verbatim>
+```
-The =ignore= option changes a coordinator job in =KILLED=, =FAILED= to =IGNORED= state.
-When a coordinator job in a bundle is in =IGNORED= state, the coordinator job doesn't impact the state of the bundle job.
-For example, when a coordinator job in a bundle failed and afterwards ignored, the bundle job becomes =SUCCEEDED= instead of =DONEWITHERROR= as long as other coordinator jobs in the bundle succeeded.
- A ignored coordinator job can be changed to =RUNNING= using -change command.
- Refer to the [[DG_CommandLineTool#Changing_endtimeconcurrencypausetimestatus_of_a_Coordinator_Job][Coordinator job change command]] for details.
+The `ignore` option changes a coordinator job in `KILLED`, `FAILED` to `IGNORED` state.
+When a coordinator job in a bundle is in `IGNORED` state, the coordinator job doesn't impact the state of the bundle job.
+For example, when a coordinator job in a bundle failed and afterwards ignored, the bundle job becomes `SUCCEEDED` instead of `DONEWITHERROR` as long as other coordinator jobs in the bundle succeeded.
+ A ignored coordinator job can be changed to `RUNNING` using -change command.
+ Refer to the [Coordinator job change command](DG_CommandLineTool.html#Changing_endtimeconcurrencypausetimestatus_of_a_Coordinator_Job) for details.
----+++ Ignore a Coordinator Action or Multiple Coordinator Actions
+### Ignore a Coordinator Action or Multiple Coordinator Actions
Example:
-<verbatim>
+
+```
$oozie job -ignore <coord_Job_id> -action 1,3-4,7-40
-</verbatim>
-The =ignore= option changes a coordinator action(s) in terminal state (=KILLED=, =FAILED=, =TIMEDOUT=) to =IGNORED= state, while not changing the state of the coordinator job.
-When a coordinator action is in =IGNORED= state, the action doesn't impact the state of a coordinator job.
-For example, when a coordinator action failed and afterwards ignored, a coordinator job becomes =SUCCEEDED= instead of =DONEWITHERROR= as long
+```
+The `ignore` option changes a coordinator action(s) in terminal state (`KILLED`, `FAILED`, `TIMEDOUT`) to `IGNORED` state, while not changing the state of the coordinator job.
+When a coordinator action is in `IGNORED` state, the action doesn't impact the state of a coordinator job.
+For example, when a coordinator action failed and afterwards ignored, a coordinator job becomes `SUCCEEDED` instead of `DONEWITHERROR` as long
as other coordinator actions succeeded.
A ignored coordinator action can be rerun using -rerun command.
-Refer to the [[DG_CoordinatorRerun][Rerunning Coordinator Actions]] for details.
-When a workflow job of a ignored coordinator action is rerun, the coordinator action becomes =RUNNING= state.
+Refer to the [Rerunning Coordinator Actions](DG_CoordinatorRerun.html) for details.
+When a workflow job of a ignored coordinator action is rerun, the coordinator action becomes `RUNNING` state.
----+++ Polling an Oozie job
+### Polling an Oozie job
-This command allows polling the Oozie server for an Oozie job until it reaches a completed status (e.g. =SUCCEEDED=, =KILLED=, etc).
+This command allows polling the Oozie server for an Oozie job until it reaches a completed status (e.g. `SUCCEEDED`, `KILLED`, etc).
Example:
-<verbatim>
+
+```
$ oozie job -poll <job_id> -interval 10 -timeout 60 -verbose
.
RUNNING
RUNNING
RUNNING
SUCCEEDED
-</verbatim>
+```
-The =-poll= argument takes a valid Workflow Job ID, Coordinator Job ID, Coordinator Action ID, or Bundle Job ID.
+The `-poll` argument takes a valid Workflow Job ID, Coordinator Job ID, Coordinator Action ID, or Bundle Job ID.
All other arguments are optional:
- * =verbose= will cause the job status to be printed at each poll; otherwise, there will be no output
- * =interval= allows specifying the polling interval in minutes (default is 5)
- * =timeout= allows specifying the timeout in minutes (default is 30 minutes); negative values indicate no timeout
----+++ Changing job SLA definition and alerting
+ * `verbose` will cause the job status to be printed at each poll; otherwise, there will be no output
+ * `interval` allows specifying the polling interval in minutes (default is 5)
+ * `timeout` allows specifying the timeout in minutes (default is 30 minutes); negative values indicate no timeout
+
+### Changing job SLA definition and alerting
* slaenable command can be used to enable job sla alerts.
* sladisable command can be used to disable job sla alerts.
* slachange command can be used to change sla job definition.
@@ -1059,20 +1108,22 @@ All other arguments are optional:
* Sla commands with -action or -date parameter will be applied to only non terminated actions.
Eg.
- <verbatim>
+
+```
$oozie job -slaenable <coord_Job_id> [-action 1,3-4,7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z,2009-11-10T01:00Z::2009-12-31T22:00Z]
$oozie job -sladisable <coord_Job_id> [-action 1,3-4,7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z,2009-11-10T01:00Z::2009-12-31T22:00Z]
- $oozie job -slachange <coord_Job_id> [-action 1,3-4,7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z,2009-11-10T01:00Z::2009-12-31T22:00Z] -value 'sla-max-duration=${10 * MINUTES};sla-should-end=${30 * MINUTES};sla-max-duration=${30 * MINUTES}'
+ $oozie job -slachange <coord_Job_id> [-action 1,3-4,7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z,2009-11-10T01:00Z::2009-12-31T22:00Z] -value 'sla-max-duration=${10 ** MINUTES};sla-should-end=${30 ** MINUTES};sla-max-duration=${30 * MINUTES}'
$oozie job -slaenable <bundle_job_id> [-action 1,3-4,7-40] [-date 2009-01-01T01:00Z::2009-05-31T23:59Z,2009-11-10T01:00Z::2009-12-31T22:00Z] [-coordinator <List_of_coord_names/ids]
- </verbatim>
+```
----+++ Getting missing dependencies of coordinator action(s)
+### Getting missing dependencies of coordinator action(s)
* Coordination action id can be specified directly for getting missing dependencies of a single action.
* To get information on multiple actions, either -action or -date option can be specified with the coordinator job id.
* missingdeps command doesn't recompute dependencies. It list missing dependencies which were last computed.
- * Oozie checks missing dependencies sequentially, and it will stop on first missing dependency. =Blocked On= is the first missing dependency for action. So, there could be a chance that Oozie will report some missing dependencies, but it might be present. To resolve the waiting issue, one should fix the blockedOn missing dependency.
+ * Oozie checks missing dependencies sequentially, and it will stop on first missing dependency. `Blocked On` is the first missing dependency for action. So, there could be a chance that Oozie will report some missing dependencies, but it might be present. To resolve the waiting issue, one should fix the blockedOn missing dependency.
* For input logic, missingdeps command doesn't compute input-logic expression. It will report everything which is missing or not computed.
-<verbatim>
+
+```
$oozie job -oozie http://localhost:11000/oozie -missingdeps 0000000-170104141851590-oozie-puru-C -action 1
$oozie job -oozie http://localhost:11000/oozie -missingdeps 0000000-170104141851590-oozie-puru-C@1
.
@@ -1090,34 +1141,37 @@ Pending Dependencies :
hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/00/20/_SUCCESS
hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/2010/01/01/00/00/_SUCCESS
$
-</verbatim>
+```
----+++ Checking a workflow definition generated by a Fluent Job API jar file
+### Checking a workflow definition generated by a Fluent Job API jar file
Since Oozie 5.1.0.
Generate a workflow definition given the Fluent Job API jar file supplied at command line, and check for its correctness.
-*Preconditions:*
+**Preconditions:**
+
* the Fluent Job API jar file has to be present and readable by the current user at the local path provided
* the folder containing the Fluent Job API jar file provided has to be writable by the current user, since the generated workflow
definition is written there
-If the =-verbose= option is provided, a lot more debugging output, including the generated workflow definition, is given.
+If the `-verbose` option is provided, a lot more debugging output, including the generated workflow definition, is given.
For more information what an Fluent Job API jar file is, how to build it etc.,
-refer to [[DG_FluentJobAPI#AE.A_Appendix_A_API_JAR_format][Fluent Job API - API JAR format]].
+refer to [Fluent Job API - API JAR format](DG_FluentJobAPI.html#AE.A_Appendix_A_API_JAR_format).
-*Example:*
+**Example:**
-<verbatim>
+
+```
$ oozie job -oozie http://localhost:11000/oozie -validatejar /tmp/workflow-api-jar.jar
Valid workflow-app
-</verbatim>
+```
+
+**Example (verbose):**
-*Example (verbose):*
-<verbatim>
+```
$ oozie job -oozie http://localhost:11000/oozie -validatejar /tmp/workflow-api-jar.jar -verbose
Checking API jar:/tmp/workflow-api-jar.jar
Loading API jar /tmp/workflow-api-jar.jar
@@ -1131,39 +1185,42 @@ API jar is written to /tmp/workflow1876390751841950810.xml
Servlet response is:
Valid workflow-app
API jar is valid.
-</verbatim>
+```
----+++ Submitting a workflow definition generated by a Fluent Job API jar file
+### Submitting a workflow definition generated by a Fluent Job API jar file
Since Oozie 5.1.0.
Generate a workflow definition given the Fluent Job API jar file supplied at command line, write it to a provided or generated HDFS
location, and submit.
-*Preconditions:*
+**Preconditions:**
+
* all the parameters that are present in the workflow definition have to be supplied either as command line parameters or via
- =job.properties= passed along with the =-config= option
+ `job.properties` passed along with the `-config` option
* the Fluent Job API jar file has to be present and readable by the current user at the local path provided
* the folder containing the Fluent Job API jar file provided has to be writable by the current user, since the generated workflow
definition is written there
- * the HDFS folder either given by =-Doozie.wf.application.path= command line parameter, or in its absence contained by
- =oozie-site.xml#oozie.client.jobs.application.generated.path= has to be writable by the current user
+ * the HDFS folder either given by `-Doozie.wf.application.path` command line parameter, or in its absence contained by
+ `oozie-site.xml#oozie.client.jobs.application.generated.path` has to be writable by the current user
-If the =-verbose= option is provided, a lot more debugging output, including the generated workflow definition, is given.
+If the `-verbose` option is provided, a lot more debugging output, including the generated workflow definition, is given.
For more information what an Fluent Job API jar file is, how to build it etc., refer to
-[[DG_FluentJobAPI#AE.A_Appendix_A_API_JAR_format][Fluent Job API - API JAR format]].
+[Fluent Job API - API JAR format](DG_FluentJobAPI.html#AE.A_Appendix_A_API_JAR_format).
+
+**Example:**
-*Example:*
-<verbatim>
+```
$ oozie job -oozie http://localhost:11000/oozie -submitjar /tmp/workflow-api-jar.jar -config /tmp/job.properties
job: 0000009-180107110323219-oozie-oozi-W
-</verbatim>
+```
+
+**Example (verbose):**
-*Example (verbose):*
-<verbatim>
+```
$ oozie job -oozie http://localhost:11000/oozie -submitjar /tmp/workflow-api-jar.jar -config /tmp/job.properties -verbose
Submitting a job based on API jar: /tmp/workflow-api-jar.jar
Loading API jar /tmp/workflow-api-jar.jar
@@ -1175,39 +1232,42 @@ Workflow job definition generated from API jar:
.
job: 0000010-180107110323219-oozie-oozi-W
Job based on API jar submitted successfully.
-</verbatim>
+```
----+++ Running a workflow definition generated by a Fluent Job API jar file
+### Running a workflow definition generated by a Fluent Job API jar file
Since Oozie 5.1.0.
Generate a workflow definition given the Fluent Job API jar file supplied at command line, write it to a provided or generated HDFS
location, submit and run.
-*Preconditions:*
+**Preconditions:**
+
* all the parameters that are present in the workflow definition have to be supplied either as command line parameters or via
- =job.properties= passed along with the =-config= option
+ `job.properties` passed along with the `-config` option
* the Fluent Job API jar file has to be present and readable by the current user at the local path provided
* the folder containing the Fluent Job API jar file provided has to be writable by the current user, since the generated workflow
definition is written there
- * the HDFS folder either given by =-Doozie.wf.application.path= command line parameter, or in its absence contained by
- =oozie-site.xml#oozie.client.jobs.application.generated.path= has to be writable by the current user
+ * the HDFS folder either given by `-Doozie.wf.application.path` command line parameter, or in its absence contained by
+ `oozie-site.xml#oozie.client.jobs.application.generated.path` has to be writable by the current user
-If the =-verbose= option is provided, a lot more debugging output, including the generated workflow definition, is given.
+If the `-verbose` option is provided, a lot more debugging output, including the generated workflow definition, is given.
For more information what an Fluent Job API jar file is, how to build it etc., refer to
-[[DG_FluentJobAPI#AE.A_Appendix_A_API_JAR_format][Fluent Job API - API JAR format]].
+[Fluent Job API - API JAR format](DG_FluentJobAPI.html#AE.A_Appendix_A_API_JAR_format).
+
+**Example:**
-*Example:*
-<verbatim>
+```
$ oozie job -oozie http://localhost:11000/oozie -runjar /tmp/workflow-api-jar.jar -config /tmp/job.properties
job: 0000011-180107110323219-oozie-oozi-W
-</verbatim>
+```
+
+**Example (verbose):**
-*Example (verbose):*
-<verbatim>
+```
$ oozie job -oozie http://localhost:11000/oozie -runjar /tmp/workflow-api-jar.jar -config /tmp/job.properties -verbose
Submitting a job based on API jar: /tmp/workflow-api-jar.jar
Loading API jar /tmp/workflow-api-jar.jar
@@ -1219,15 +1279,16 @@ Workflow job definition generated from API jar:
.
job: 0000010-180107110323219-oozie-oozi-W
Job based on API jar run successfully.
-</verbatim>
+```
----++ Jobs Operations
+## Jobs Operations
----+++ Checking the Status of multiple Workflow Jobs
+### Checking the Status of multiple Workflow Jobs
Example:
-<verbatim>
+
+```
$ oozie jobs -oozie http://localhost:11000/oozie -localtime -len 2 -filter status=RUNNING
.
Job Id Workflow Name Status Run User Group Created Started Ended
@@ -1235,20 +1296,20 @@ Job Id Workflow Name Status Run User
4-20090527151008-oozie-joe hadoopel-wf RUNNING 0 joe other 2009-05-27 15:34 +0530 2009-05-27 15:34 +0530 -
0-20090527151008-oozie-joe hadoopel-wf RUNNING 0 joe other 2009-05-27 15:15 +0530 2009-05-27 15:15 +0530 -
.----------------------------------------------------------------------------------------------------------------------------------------------------------------
-</verbatim>
+```
-The =jobs= sub-command will display information about multiple jobs.
+The `jobs` sub-command will display information about multiple jobs.
-The =offset= and =len= option specified the offset and number of jobs to display, default values are =1= and =100=
+The `offset` and `len` option specified the offset and number of jobs to display, default values are `1` and `100`
respectively.
-The =localtime= option displays times in local time, if not specified times are displayed in GMT.
+The `localtime` option displays times in local time, if not specified times are displayed in GMT.
-The =verbose= option gives more detailed information for each job.
+The `verbose` option gives more detailed information for each job.
A filter can be specified after all options.
-The =filter=option syntax is: <code>[NAME=VALUE][;NAME=VALUE]*</code>.
+The `filter`option syntax is: `[NAME=VALUE][;NAME=VALUE]*`.
Valid filter names are:
@@ -1259,20 +1320,21 @@ Valid filter names are:
* status: the status of the job.
* startcreatedtime: the start of time window in specifying createdtime range filter.
* endcreatedtime: the end of time window in specifying createdtime range filter
- * sortby: order the results. Supported values for =sortby= are: =createdTime= and =lastModifiedTime=
+ * sortby: order the results. Supported values for `sortby` are: `createdTime` and `lastModifiedTime`
The query will do an AND among all the filter names. The query will do an OR among all the filter values for the same
name. Multiple values must be specified as different name value pairs.
-startCreatedTime and endCreatedTime should be specified either in *ISO8601 (UTC)* format (*yyyy-MM-dd'T'HH:mm'Z'*) or a offset value in days or hours from the current time. For example, -2d means the current time - 2 days. -3h means the current time - 3 hours, -5m means the current time - 5 minutes
+startCreatedTime and endCreatedTime should be specified either in **ISO8601 (UTC)** format (**yyyy-MM-dd'T'HH:mm'Z'**) or a offset value in days or hours from the current time. For example, -2d means the current time - 2 days. -3h means the current time - 3 hours, -5m means the current time - 5 minutes
----+++ Checking the Status of multiple Coordinator Jobs
+### Checking the Status of multiple Coordinator Jobs
* This feature is only supported in Oozie 2.0 or later.
Example:
-<verbatim>
+
+```
$ oozie jobs -oozie http://localhost:11000/oozie -jobtype coordinator
.
Job ID App Name Status Freq Unit Started Next Materialized
@@ -1281,9 +1343,9 @@ Job ID
.----------------------------------------------------------------------------------------------------------------------------------------------------------------
0003823-100531045722929-oozie-wrkf-C coordcal2880minutescurrent SUCCEEDED 2880 MINUTE 2010-02-01 16:30 2010-02-05 16:30
.----------------------------------------------------------------------------------------------------------------------------------------------------------------
-</verbatim>
+```
-The =jobtype= option specified the job type to display, default value is 'wf'. To see the coordinator jobs, value is 'coordinator'.
+The `jobtype` option specified the job type to display, default value is 'wf'. To see the coordinator jobs, value is 'coordinator'.
Valid filter names are:
@@ -1293,15 +1355,16 @@ Valid filter names are:
* status: the status of the job.
* frequency: the frequency of the Coordinator job.
* unit: the time unit. It can take one of the following four values: months, days, hours or minutes. Time unit should be added only when frequency is specified.
- * sortby: order the results. Supported values for =sortby= are: =createdTime= and =lastModifiedTime=
+ * sortby: order the results. Supported values for `sortby` are: `createdTime` and `lastModifiedTime`
----+++ Checking the Status of multiple Bundle Jobs
+### Checking the Status of multiple Bundle Jobs
* This feature is only supported in Oozie 3.0 or later.
Example:
-<verbatim>
+
+```
$ oozie jobs -oozie http://localhost:11000/oozie -jobtype bundle
Job ID Bundle Name Status Kickoff Created User Group
.------------------------------------------------------------------------------------------------------------------------------------
@@ -1311,15 +1374,16 @@ Job ID Bundle Name Status Kickoff
.------------------------------------------------------------------------------------------------------------------------------------
0000000-110322105610515-oozie-chao-B BUNDLE-TEST DONEWITHERROR2012-01-15 00:24 2011-03-22 17:58 joe users
.------------------------------------------------------------------------------------------------------------------------------------
-</verbatim>
+```
-The =jobtype= option specified the job type to display, default value is 'wf'. To see the bundle jobs, value is 'bundle'.
+The `jobtype` option specified the job type to display, default value is 'wf'. To see the bundle jobs, value is 'bundle'.
----+++ Bulk kill, suspend or resume multiple jobs
+### Bulk kill, suspend or resume multiple jobs
Example:
-<verbatim>
+
+```
$ oozie jobs -oozie http://localhost:11000/oozie -kill|-suspend|-resume -filter name=cron-coord -jobtype coordinator
The following jobs have been killed|suspended|resumed
Job ID App Name Status Freq Unit Started Next Materialized
@@ -1330,21 +1394,21 @@ Job ID App Name Status Freq Unit
.------------------------------------------------------------------------------------------------------------------------------------
0000000-150224141553231-oozie-bzha-C cron-coord KILLED 10 MINUTE 2015-02-25 22:00 GMT -
.------------------------------------------------------------------------------------------------------------------------------------
-</verbatim>
+```
The above command will kill, suspend, or resume all the coordinator jobs with name of "cron-coord" starting with offset 1
to 50.
-The =jobs= sub-command will bulk modify all the jobs that satisfy the filter, len, offset, and jobtype options when adding
+The `jobs` sub-command will bulk modify all the jobs that satisfy the filter, len, offset, and jobtype options when adding
a -kill|-suspend|-resume option. Another way to think about is the subcommand works to modify all the jobs that will be
displayed if the write option(kill|suspend|resume) is not there.
-The =offset= and =len= option specified the offset and number of jobs to be modified, default values are =1= and =50=
+The `offset` and `len` option specified the offset and number of jobs to be modified, default values are `1` and `50`
respectively.
-The =jobtype= option specifies the type of jobs to be modified, be it "wf", "coordinator" or "bundle". default value is "wf".
+The `jobtype` option specifies the type of jobs to be modified, be it "wf", "coordinator" or "bundle". default value is "wf".
A filter can be specified after all options.
-The =filter=option syntax is: <code>[NAME=VALUE][;NAME=VALUE]*</code>.
+The `filter`option syntax is: `[NAME=VALUE][;NAME=VALUE]*`.
Valid filter names are:
@@ -1354,21 +1418,22 @@ Valid filter names are:
* status: the status of the job.
* frequency: the frequency of the Coordinator job.
* unit: the time unit. It can take one of the following four values: months, days, hours or minutes. Time unit should be added only when frequency is specified.
- * sortby: order the results. Supported values for =sortby= are: =createdTime= and =lastModifiedTime=
+ * sortby: order the results. Supported values for `sortby` are: `createdTime` and `lastModifiedTime`
The query will do an AND among all the filter names. The query will do an OR among all the filter values for the same
name. Multiple values must be specified as different name value pairs.
The following example shows how to suspend the first 20 bundle jobs whose name is "bundle-app":
-<verbatim>
+
+```
$ oozie jobs -oozie http://localhost:11000/oozie -suspend -filter name=bundle-app -jobtype bundle -len 20
-</verbatim>
+```
----+++ Bulk monitoring for jobs launched via Bundles
+### Bulk monitoring for jobs launched via Bundles
* This command-line query helps to directly query for a bulk of jobs using a set of rich filters.
-The jobs need to have a parent *Bundle*, and this performs a deep query to provide results about all the workflows that satisfy your filters.
+The jobs need to have a parent **Bundle**, and this performs a deep query to provide results about all the workflows that satisfy your filters.
These results display relevant job-ids, app-names, and error message (if any) and are most helpful when you need to monitor a bulk of jobs and get a gist,
while avoiding typing multiple queries.
@@ -1376,20 +1441,22 @@ This feature is only supported in Oozie 3.3 or later.
Example 1:
-<verbatim>
+
+```
$ oozie jobs -oozie http://localhost:11000/oozie -bulk bundle=bundle-app-1
.
Bundle Name Bundle ID Coord Name Coord Action ID External ID Status Created Time Error Message
.-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
bundle-app-1 0000000-130408151805946-oozie-chit-B coord-1 0000001-130408151805946-oozie-chit-C@1 0000002-130408151805946-oozie-chit-W KILLED 2013-04-08 22:20 GMT null
.-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-</verbatim>
+```
Example 2: This example illustrates giving multiple arguments and -verbose option.
_NOTE: The filter string after -bulk should be enclosed within quotes_
-<verbatim>
+
+```
.
$ oozie jobs -oozie http://localhost:11000/oozie -bulk 'bundle=bundle-app-2;actionstatus=SUCCEEDED' -verbose
.
@@ -1404,7 +1471,7 @@ Created Time : 2013-04-23 01:43 GMT
User : user_xyz
Error Message : -
.------------------------------------------------------------------------------------------------------------------------------------
-</verbatim>
+```
You can type 'help' to view usage for the CLI options and filters available. Namely:
@@ -1414,7 +1481,8 @@ You can type 'help' to view usage for the CLI options and filters available. Nam
* startcreatedtime/endcreatedtime: specify boundaries for the created-time. Only jobs created within this window are included.
* startscheduledtime/endscheduledtime: specify boundaries for the nominal-time. Only jobs scheduled to run within this window are included.
-<verbatim>
+
+```
$ oozie jobs <OPTIONS> : jobs status
-bulk <arg> key-value pairs to filter bulk jobs response. e.g.
bundle=<B>\;coordinators=<C>\;actionstatus=<S>\;
@@ -1423,57 +1491,61 @@ $ oozie jobs <OPTIONS> : jobs status
coordinators and actionstatus can be multiple comma
separated values. "bundle" and "coordinators" are 'names' of those jobs.
Bundle name is mandatory, other params are optional
-</verbatim>
+```
Similar to the usual jobs filter, different filter arguments here should be separated by semicolon (;).
----++ Admin Operations
+## Admin Operations
----+++ Checking the Status of the Oozie System
+### Checking the Status of the Oozie System
Example:
-<verbatim>
+
+```
$ oozie admin -oozie http://localhost:11000/oozie -status
.
Safemode: OFF
-</verbatim>
+```
It returns the current status of the Oozie system.
----+++ Changing the Status of the Oozie System
+### Changing the Status of the Oozie System
* This feature is only supported in Oozie 2.0 or later.
Example:
-<verbatim>
+
+```
$ oozie admin -oozie http://localhost:11000/oozie -systemmode [NORMAL|NOWEBSERVICE|SAFEMODE]
.
Safemode: ON
-</verbatim>
+```
It returns the current status of the Oozie system.
----+++ Displaying the Build Version of the Oozie System
+### Displaying the Build Version of the Oozie System
Example:
-<verbatim>
+
+```
$ oozie admin -oozie http://localhost:11000/oozie -version
.
Oozie server build version: 2.0.2.1-0.20.1.3092118008--
-</verbatim>
+```
It returns the Oozie server build version.
----+++ Displaying the queue dump of the Oozie System
+### Displaying the queue dump of the Oozie System
* This feature is for administrator debugging purpose
Example:
-<verbatim>
+
+```
$ oozie admin -oozie http://localhost:11000/oozie -queuedump
.
[Server Queue Dump]:
@@ -1481,29 +1553,31 @@ $ oozie admin -oozie http://localhost:11000/oozie -queuedump
(coord_action_ready,1)
(action.check,2)
-</verbatim>
+```
It returns the Oozie server current queued commands.
----+++ Displaying the list of available Oozie Servers
+### Displaying the list of available Oozie Servers
Example:
-<verbatim>
+
+```
$ oozie admin -oozie http://hostA:11000/oozie -servers
hostA : http://hostA:11000/oozie
hostB : http://hostB:11000/oozie
hostC : http://hostC:11000/oozie
-</verbatim>
+```
-It returns a list of available Oozie Servers. This is useful when Oozie is configured for [[AG_Install#HA][High Availability]]; if
+It returns a list of available Oozie Servers. This is useful when Oozie is configured for [High Availability](AG_Install.html#HA); if
not, it will simply return the one Oozie Server.
----+++ Displaying the Oozie server configuration
+### Displaying the Oozie server configuration
Example:
-<verbatim>
+
+```
$ oozie admin -oozie http://localhost:11000/oozie -configuration
local.realm : LOCALHOST
oozie.JobCommand.job.console.url : http://localhost:11000/oozie?job=
@@ -1515,19 +1589,20 @@ oozie.action.mapreduce.uber.jar.enable : false
oozie.action.max.output.data : 2048
oozie.action.retries.max : 3
...
-</verbatim>
+```
It returns a list of the configuration properties and values from oozie-site.xml and oozie-default.xml being used by the Oozie
server.
----+++ Displaying the Oozie server OS environment
+### Displaying the Oozie server OS environment
Example:
-<verbatim>
+
+```
$ oozie admin -oozie http://localhost:11000/oozie -osenv
...
-JETTY_OPTS : -Doozie.home.dir=/Users/asasvari/dev/oozie -Doozie.config.dir=/Users/asasvari/dev/oozie/conf -Doozie.log.dir=/Users/asasvari/dev/oozie/logs -Doozie.data.dir=/Users/asasvari/dev/oozie/data -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Djava.library.path= -cp /Users/asasvari/dev/oozie/embedded-oozie-server/*:/Users/asasvari/dev/oozie/embedded-oozie-server/dependency/*:/Users/asasvari/dev/oozie/lib/*:/Users/asasvari/dev/oozie/libtools/*:/Users/asasvari/dev/oozie/embedded-oozie-server
+JETTY_OPTS : -Doozie.home.dir=/Users/asasvari/dev/oozie -Doozie.config.dir=/Users/asasvari/dev/oozie/conf -Doozie.log.dir=/Users/asasvari/dev/oozie/logs -Doozie.data.dir=/Users/asasvari/dev/oozie/data -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Djava.library.path= -cp /Users/asasvari/dev/oozie/embedded-oozie-server/**:/Users/asasvari/dev/oozie/embedded-oozie-server/dependency/**:/Users/asasvari/dev/oozie/lib/**:/Users/asasvari/dev/oozie/libtools/**:/Users/asasvari/dev/oozie/embedded-oozie-server
JETTY_OUT : /Users/asasvari/dev/oozie/logs/jetty.out
JETTY_PID_FILE : /Users/asasvari/dev/oozie/embedded-oozie-server/oozie.pid
OOZIE_CONFIG : /Users/asasvari/dev/oozie/conf
@@ -1538,15 +1613,16 @@ OOZIE_LOG : /Users/asasvari/dev/oozie/logs
OOZIE_LOG4J_FILE : oozie-log4j.properties
OOZIE_LOG4J_RELOAD : 10
...
-</verbatim>
+```
It returns a list of OS environment variables in the Oozie server.
----+++ Displaying the Oozie server Java system properties
+### Displaying the Oozie server Java system properties
Example:
-<verbatim>
+
+```
$ oozie admin -oozie http://localhost:11000/oozie -javasysprops
...
oozie.config.dir : /Users/asasvari/dev/oozie/conf
@@ -1557,17 +1633,18 @@ oozie.log.dir : /Users/asasvari/dev/oozie/logs
oozie.log4j.file : oozie-log4j.properties
oozie.log4j.reload : 10
...
-</verbatim>
+```
It returns a list of java system properties in the Oozie server.
----+++ Displaying the Oozie server Instrumentation
+### Displaying the Oozie server Instrumentation
Deprecated and by default disabled since 5.0.0.
Example:
-<verbatim>
+
+```
$
<TRUNCATED>
[02/11] oozie git commit: OOZIE-2734 [docs] Switch from TWiki to
Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/WorkflowFunctionalSpec.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/WorkflowFunctionalSpec.twiki b/docs/src/site/twiki/WorkflowFunctionalSpec.twiki
index 46a454c..463635b 100644
--- a/docs/src/site/twiki/WorkflowFunctionalSpec.twiki
+++ b/docs/src/site/twiki/WorkflowFunctionalSpec.twiki
@@ -1,135 +1,157 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
+
+[::Go back to Oozie Documentation Index::](index.html)
-----
----+!! Oozie Specification, a Hadoop Workflow System
-*<center>(v5.0)</center>*
+# Oozie Specification, a Hadoop Workflow System
+**<center>(v5.0)</center>**
The goal of this document is to define a workflow engine system specialized in coordinating the execution of Hadoop
Map/Reduce and Pig jobs.
-%TOC%
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Changelog
+
+
+**2016FEB19**
+
+ * 3.2.7 Updated notes on System.exit(int n) behavior
+
+**2015APR29**
+
+ * 3.2.1.4 Added notes about Java action retries
+ * 3.2.7 Added notes about Java action retries
----++ Changelog
+**2014MAY08**
----+++!! 2016FEB19
+ * 3.2.2.4 Added support for fully qualified job-xml path
- * #3.2.7 Updated notes on System.exit(int n) behavior
----+++!! 2015APR29
+**2013JUL03**
- * #3.2.1.4 Added notes about Java action retries
- * #3.2.7 Added notes about Java action retries
----+++!! 2014MAY08
+ * Appendix A, Added new workflow schema 0.5 and SLA schema 0.2
- * #3.2.2.4 Added support for fully qualified job-xml path
----+++!! 2013JUL03
+**2012AUG30**
- * #Appendix A, Added new workflow schema 0.5 and SLA schema 0.2
----+++!! 2012AUG30
+ * 4.2.2 Added two EL functions (replaceAll and appendAll)
- * #4.2.2 Added two EL functions (replaceAll and appendAll)
----+++!! 2012JUL26
+**2012JUL26**
- * #Appendix A, updated XML schema 0.4 to include =parameters= element
- * #4.1 Updated to mention about =parameters= element as of schema 0.4
----+++!! 2012JUL23
+ * Appendix A, updated XML schema 0.4 to include `parameters` element
+ * 4.1 Updated to mention about `parameters` element as of schema 0.4
- * #Appendix A, updated XML schema 0.4 (Fs action)
- * #3.2.4 Updated to mention that a =name-node=, a =job-xml=, and a =configuration= element are allowed in the Fs action as of
+**2012JUL23**
+
+ * Appendix A, updated XML schema 0.4 (Fs action)
+ * 3.2.4 Updated to mention that a `name-node`, a `job-xml`, and a `configuration` element are allowed in the Fs action as of
schema 0.4
----+++!! 2012JUN19
- * #Appendix A, added XML schema 0.4
- * #3.2.2.4 Updated to mention that multiple =job-xml= elements are allowed as of schema 0.4
- * #3.2.3 Updated to mention that multiple =job-xml= elements are allowed as of schema 0.4
----+++!! 2011AUG17
+**2012JUN19**
+
+ * Appendix A, added XML schema 0.4
+ * 3.2.2.4 Updated to mention that multiple `job-xml` elements are allowed as of schema 0.4
+ * 3.2.3 Updated to mention that multiple `job-xml` elements are allowed as of schema 0.4
+
+**2011AUG17**
+
+ * 3.2.4 fs 'chmod' xml closing element typo in Example corrected
+
+**2011AUG12**
+
+ * 3.2.4 fs 'move' action characteristics updated, to allow for consistent source and target paths and existing target path only if directory
+ * 18, Update the doc for user-retry of workflow action.
+
+**2011FEB19**
- * #3.2.4 fs 'chmod' xml closing element typo in Example corrected
----+++!! 2011AUG12
+ * 10, Update the doc to rerun from the failed node.
- * #3.2.4 fs 'move' action characteristics updated, to allow for consistent source and target paths and existing target path only if directory
- * #18, Update the doc for user-retry of workflow action.
----+++!! 2011FEB19
+**2010OCT31**
- * #10, Update the doc to rerun from the failed node.
----+++!! 2010OCT31
+ * 17, Added new section on Shared Libraries
- * #17, Added new section on Shared Libraries
----+++!! 2010APR27
+**2010APR27**
- * #3.2.3 Added new "arguments" tag to PIG actions
- * #3.2.5 SSH actions are deprecated in Oozie schema 0.1 and removed in Oozie schema 0.2
- * #Appendix A, Added schema version 0.2
-
----+++!! 2009OCT20
+ * 3.2.3 Added new "arguments" tag to PIG actions
+ * 3.2.5 SSH actions are deprecated in Oozie schema 0.1 and removed in Oozie schema 0.2
+ * Appendix A, Added schema version 0.2
- * #Appendix A, updated XML schema
----+++!! 2009SEP15
+**2009OCT20**
- * #3.2.6 Removing support for sub-workflow in a different Oozie instance (removing the 'oozie' element)
+ * Appendix A, updated XML schema
----+++!! 2009SEP07
- * #3.2.2.3 Added Map Reduce Pipes specifications.
- * #3.2.2.4 Map-Reduce Examples. Previously was 3.2.2.3.
+**2009SEP15**
----+++!! 2009SEP02
+ * 3.2.6 Removing support for sub-workflow in a different Oozie instance (removing the 'oozie' element)
- * #10 Added missing skip nodes property name.
- * #3.2.1.4 Reworded action recovery explanation.
----+++!! 2009AUG26
+**2009SEP07**
- * #3.2.9 Added =java= action type
- * #3.1.4 Example uses EL constant to refer to counter group/name
+ * 3.2.2.3 Added Map Reduce Pipes specifications.
+ * 3.2.2.4 Map-Reduce Examples. Previously was 3.2.2.3.
----+++!! 2009JUN09
- * #12.2.4 Added build version resource to admin end-point
- * #3.2.6 Added flag to propagate workflow configuration to sub-workflows
- * #10 Added behavior for workflow job parameters given in the rerun
- * #11.3.4 workflows info returns pagination information
+**2009SEP02**
----+++!! 2009MAY18
+ * 10 Added missing skip nodes property name.
+ * 3.2.1.4 Reworded action recovery explanation.
- * #3.1.4 decision node, 'default' element, 'name' attribute changed to 'to'
- * #3.1.5 fork node, 'transition' element changed to 'start', 'to' attribute change to 'path'
- * #3.1.5 join node, 'transition' element remove, added 'to' attribute to 'join' element
- * #3.2.1.4 Rewording on action recovery section
- * #3.2.2 map-reduce action, added 'job-tracker', 'name-node' actions, 'file', 'file' and 'archive' elements
- * #3.2.2.1 map-reduce action, remove from 'streaming' element 'file', 'file' and 'archive' elements
- * #3.2.2.2 map-reduce action, reorganized streaming section
- * #3.2.3 pig action, removed information about implementation (SSH), changed elements names
- * #3.2.4 fs action, removed 'fs-uri' and 'user-name' elements, file system URI is now specified in path, user is propagated
- * #3.2.6 sub-workflow action, renamed elements 'oozie-url' to 'oozie' and 'workflow-app' to 'app-path'
- * #4 Properties that are valid Java identifiers can be used as ${NAME}
- * #4.1 Renamed default properties file from 'configuration.xml' to 'default-configuration.xml'
- * #4.2 Changes in EL Constants and Functions
- * #5 Updated notification behavior and tokens
- * #6 Changed user propagation behavior
- * #7 Changed application packaging from ZIP to HDFS directory
+
+**2009AUG26**
+
+ * 3.2.9 Added `java` action type
+ * 3.1.4 Example uses EL constant to refer to counter group/name
+
+
+**2009JUN09**
+
+ * 12.2.4 Added build version resource to admin end-point
+ * 3.2.6 Added flag to propagate workflow configuration to sub-workflows
+ * 10 Added behavior for workflow job parameters given in the rerun
+ * 11.3.4 workflows info returns pagination information
+
+
+**2009MAY18**
+
+ * 3.1.4 decision node, 'default' element, 'name' attribute changed to 'to'
+ * 3.1.5 fork node, 'transition' element changed to 'start', 'to' attribute change to 'path'
+ * 3.1.5 join node, 'transition' element remove, added 'to' attribute to 'join' element
+ * 3.2.1.4 Rewording on action recovery section
+ * 3.2.2 map-reduce action, added 'job-tracker', 'name-node' actions, 'file', 'file' and 'archive' elements
+ * 3.2.2.1 map-reduce action, remove from 'streaming' element 'file', 'file' and 'archive' elements
+ * 3.2.2.2 map-reduce action, reorganized streaming section
+ * 3.2.3 pig action, removed information about implementation (SSH), changed elements names
+ * 3.2.4 fs action, removed 'fs-uri' and 'user-name' elements, file system URI is now specified in path, user is propagated
+ * 3.2.6 sub-workflow action, renamed elements 'oozie-url' to 'oozie' and 'workflow-app' to 'app-path'
+ * 4 Properties that are valid Java identifiers can be used as ${NAME}
+ * 4.1 Renamed default properties file from 'configuration.xml' to 'default-configuration.xml'
+ * 4.2 Changes in EL Constants and Functions
+ * 5 Updated notification behavior and tokens
+ * 6 Changed user propagation behavior
+ * 7 Changed application packaging from ZIP to HDFS directory
* Removed application lifecycle and self containment model sections
- * #10 Changed workflow job recovery, simplified recovery behavior
- * #11 Detailed Web Services API
- * #12 Updated Client API section
- * #15 Updated Action Executor API section
- * #Appendix A XML namespace updated to 'uri:oozie:workflow:0.1'
- * #Appendix A Updated XML schema to changes in map-reduce/pig/fs/ssh actions
- * #Appendix B Updated workflow example to schema changes
+ * 10 Changed workflow job recovery, simplified recovery behavior
+ * 11 Detailed Web Services API
+ * 12 Updated Client API section
+ * 15 Updated Action Executor API section
+ * Appendix A XML namespace updated to 'uri:oozie:workflow:0.1'
+ * Appendix A Updated XML schema to changes in map-reduce/pig/fs/ssh actions
+ * Appendix B Updated workflow example to schema changes
+
----+++!! 2009MAR25
+**2009MAR25**
* Changing all references of HWS to Oozie (project name)
* Typos, XML Formatting
* XML Schema URI correction
----+++!! 2009MAR09
- * Changed =CREATED= job state to =PREP= to have same states as Hadoop
+**2009MAR09**
+
+ * Changed `CREATED` job state to `PREP` to have same states as Hadoop
* Renamed 'hadoop-workflow' element to 'workflow-app'
* Decision syntax changed to be 'switch/case' with no transition indirection
* Action nodes common root element 'action', with the action type as sub-element (using a single built-in XML schema)
@@ -142,32 +164,34 @@ schema 0.4
* Added Oozie XML schema to Appendix A
* Added workflow example to Appendix B
----+++!! 2009FEB22
- * Opened [[https://issues.apache.org/jira/browse/HADOOP-5303][JIRA HADOOP-5303]]
+**2009FEB22**
----+++!! 27/DEC/2012:
+ * Opened [JIRA HADOOP-5303](https://issues.apache.org/jira/browse/HADOOP-5303)
+
+
+**27/DEC/2012:**
* Added information on dropping hcatalog table partitions in prepare block
* Added hcatalog EL functions section
----++ 0 Definitions
+## 0 Definitions
-*Action:* An execution/computation task (Map-Reduce job, Pig job, a shell command). It can also be referred as task or
+**Action:** An execution/computation task (Map-Reduce job, Pig job, a shell command). It can also be referred as task or
'action node'.
-*Workflow:* A collection of actions arranged in a control dependency DAG (Directed Acyclic Graph). "control dependency"
+**Workflow:** A collection of actions arranged in a control dependency DAG (Directed Acyclic Graph). "control dependency"
from one action to another means that the second action can't run until the first action has completed.
-*Workflow Definition:* A programmatic description of a workflow that can be executed.
+**Workflow Definition:** A programmatic description of a workflow that can be executed.
-*Workflow Definition Language:* The language used to define a Workflow Definition.
+**Workflow Definition Language:** The language used to define a Workflow Definition.
-*Workflow Job:* An executable instance of a workflow definition.
+**Workflow Job:** An executable instance of a workflow definition.
-*Workflow Engine:* A system that executes workflows jobs. It can also be referred as a DAG engine.
+**Workflow Engine:** A system that executes workflows jobs. It can also be referred as a DAG engine.
----++ 1 Specification Highlights
+## 1 Specification Highlights
A Workflow application is DAG that coordinates the following types of actions: Hadoop, Pig, and
sub-workflows.
@@ -176,7 +200,7 @@ Flow control operations within the workflow applications can be done using decis
workflows are not supported.
Actions and decisions can be parameterized with job properties, actions output (i.e. Hadoop counters) and file information (file exists, file size, etc). Formal parameters are expressed in the workflow
-definition as =${VAR}= variables.
+definition as `${VAR}` variables.
A Workflow application is a ZIP file that contains the workflow definition (an XML file), all the necessary files to
run all the actions: JAR files for Map/Reduce jobs, shells for streaming Map/Reduce jobs, native libraries, Pig
@@ -191,7 +215,7 @@ Monitoring the system and workflow jobs can be done via a web console, command l
When submitting a workflow job, a set of properties resolving all the formal parameters in the workflow definitions
must be provided. This set of properties is a Hadoop configuration.
-Possible states for a workflow jobs are: =PREP=, =RUNNING=, =SUSPENDED=, =SUCCEEDED=, =KILLED= and =FAILED=.
+Possible states for a workflow jobs are: `PREP`, `RUNNING`, `SUSPENDED`, `SUCCEEDED`, `KILLED` and `FAILED`.
In the case of a action start failure in a workflow job, depending on the type of failure, Oozie will attempt automatic
retries, it will request a manual retry or it will fail the workflow job.
@@ -202,119 +226,124 @@ In the case of workflow job failure, the workflow job can be resubmitted skippin
Before doing a resubmission the workflow application could be updated with a patch to fix a problem in the workflow
application code.
-#WorkflowDefinition
----++ 2 Workflow Definition
+<a name="WorkflowDefinition"></a>
+## 2 Workflow Definition
A workflow definition is a DAG with control flow nodes (start, end, decision, fork, join, kill) or action nodes
(map-reduce, pig, etc.), nodes are connected by transitions arrows.
The workflow definition language is XML based and it is called hPDL (Hadoop Process Definition Language).
-Refer to the Appendix A for the[[WorkflowFunctionalSpec#OozieWFSchema][Oozie Workflow Definition XML Schema]]. Appendix
-B has [[WorkflowFunctionalSpec#OozieWFExamples][Workflow Definition Examples]].
+Refer to the Appendix A for the[Oozie Workflow Definition XML Schema](WorkflowFunctionalSpec.html#OozieWFSchema). Appendix
+B has [Workflow Definition Examples](WorkflowFunctionalSpec.html#OozieWFExamples).
----+++ 2.1 Cycles in Workflow Definitions
+### 2.1 Cycles in Workflow Definitions
Oozie does not support cycles in workflow definitions, workflow definitions must be a strict DAG.
At workflow application deployment time, if Oozie detects a cycle in the workflow definition it must fail the
deployment.
----++ 3 Workflow Nodes
+## 3 Workflow Nodes
Workflow nodes are classified in control flow nodes and action nodes:
- * *Control flow nodes:* nodes that control the start and end of the workflow and workflow job execution path.
- * *Action nodes:* nodes that trigger the execution of a computation/processing task.
+ * **Control flow nodes:** nodes that control the start and end of the workflow and workflow job execution path.
+ * **Action nodes:** nodes that trigger the execution of a computation/processing task.
-Node names and transitions must be conform to the following pattern =[a-zA-Z][\-_a-zA-Z0-0]*=, of up to 20 characters
+Node names and transitions must be conform to the following pattern `[a-zA-Z][\-_a-zA-Z0-0]*`, of up to 20 characters
long.
----+++ 3.1 Control Flow Nodes
+### 3.1 Control Flow Nodes
-Control flow nodes define the beginning and the end of a workflow (the =start=, =end= and =kill= nodes) and provide a
-mechanism to control the workflow execution path (the =decision=, =fork= and =join= nodes).
+Control flow nodes define the beginning and the end of a workflow (the `start`, `end` and `kill` nodes) and provide a
+mechanism to control the workflow execution path (the `decision`, `fork` and `join` nodes).
-#StartNode
----++++ 3.1.1 Start Control Node
+<a name="StartNode"></a>
+#### 3.1.1 Start Control Node
-The =start= node is the entry point for a workflow job, it indicates the first workflow node the workflow job must
+The `start` node is the entry point for a workflow job, it indicates the first workflow node the workflow job must
transition to.
-When a workflow is started, it automatically transitions to the node specified in the =start=.
+When a workflow is started, it automatically transitions to the node specified in the `start`.
+
+A workflow definition must have one `start` node.
-A workflow definition must have one =start= node.
+**Syntax:**
-*Syntax:*
-<verbatim>
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<start to="[NODE-NAME]"/>
...
</workflow-app>
-</verbatim>
+```
-The =to= attribute is the name of first workflow node to execute.
+The `to` attribute is the name of first workflow node to execute.
-*Example:*
+**Example:**
-<verbatim>
+
+```
<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0">
...
<start to="firstHadoopJob"/>
...
</workflow-app>
-</verbatim>
+```
-#EndNode
----++++ 3.1.2 End Control Node
+<a name="EndNode"></a>
+#### 3.1.2 End Control Node
-The =end= node is the end for a workflow job, it indicates that the workflow job has completed successfully.
+The `end` node is the end for a workflow job, it indicates that the workflow job has completed successfully.
-When a workflow job reaches the =end= it finishes successfully (SUCCEEDED).
+When a workflow job reaches the `end` it finishes successfully (SUCCEEDED).
-If one or more actions started by the workflow job are executing when the =end= node is reached, the actions will be
+If one or more actions started by the workflow job are executing when the `end` node is reached, the actions will be
killed. In this scenario the workflow job is still considered as successfully run.
-A workflow definition must have one =end= node.
+A workflow definition must have one `end` node.
+
+**Syntax:**
-*Syntax:*
-<verbatim>
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<end name="[NODE-NAME]"/>
...
</workflow-app>
-</verbatim>
+```
-The =name= attribute is the name of the transition to do to end the workflow job.
+The `name` attribute is the name of the transition to do to end the workflow job.
-*Example:*
+**Example:**
-<verbatim>
+
+```
<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0">
...
<end name="end"/>
</workflow-app>
-</verbatim>
+```
-#KillNode
----++++ 3.1.3 Kill Control Node
+<a name="KillNode"></a>
+#### 3.1.3 Kill Control Node
-The =kill= node allows a workflow job to kill itself.
+The `kill` node allows a workflow job to kill itself.
-When a workflow job reaches the =kill= it finishes in error (KILLED).
+When a workflow job reaches the `kill` it finishes in error (KILLED).
-If one or more actions started by the workflow job are executing when the =kill= node is reached, the actions will be
+If one or more actions started by the workflow job are executing when the `kill` node is reached, the actions will be
killed.
-A workflow definition may have zero or more =kill= nodes.
+A workflow definition may have zero or more `kill` nodes.
+
+**Syntax:**
-*Syntax:*
-<verbatim>
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<kill name="[NODE-NAME]">
@@ -322,17 +351,18 @@ A workflow definition may have zero or more =kill= nodes.
</kill>
...
</workflow-app>
-</verbatim>
+```
-The =name= attribute in the =kill= node is the name of the Kill action node.
+The `name` attribute in the `kill` node is the name of the Kill action node.
-The content of the =message= element will be logged as the kill reason for the workflow job.
+The content of the `message` element will be logged as the kill reason for the workflow job.
-A =kill= node does not have transition elements because it ends the workflow job, as =KILLED=.
+A `kill` node does not have transition elements because it ends the workflow job, as `KILLED`.
-*Example:*
+**Example:**
-<verbatim>
+
+```
<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0">
...
<kill name="killBecauseNoInput">
@@ -340,29 +370,31 @@ A =kill= node does not have transition elements because it ends the workflow job
</kill>
...
</workflow-app>
-</verbatim>
+```
-#DecisionNode
----++++ 3.1.4 Decision Control Node
+<a name="DecisionNode"></a>
+#### 3.1.4 Decision Control Node
-A =decision= node enables a workflow to make a selection on the execution path to follow.
+A `decision` node enables a workflow to make a selection on the execution path to follow.
-The behavior of a =decision= node can be seen as a switch-case statement.
+The behavior of a `decision` node can be seen as a switch-case statement.
-A =decision= node consists of a list of predicates-transition pairs plus a default transition. Predicates are evaluated
-in order or appearance until one of them evaluates to =true= and the corresponding transition is taken. If none of the
-predicates evaluates to =true= the =default= transition is taken.
+A `decision` node consists of a list of predicates-transition pairs plus a default transition. Predicates are evaluated
+in order or appearance until one of them evaluates to `true` and the corresponding transition is taken. If none of the
+predicates evaluates to `true` the `default` transition is taken.
Predicates are JSP Expression Language (EL) expressions (refer to section 4.2 of this document) that resolve into a
-boolean value, =true= or =false=. For example:
+boolean value, `true` or `false`. For example:
+
-<verbatim>
+```
${fs:fileSize('/usr/foo/myinputdir') gt 10 * GB}
-</verbatim>
+```
+
+**Syntax:**
-*Syntax:*
-<verbatim>
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<decision name="[NODE-NAME]">
@@ -375,22 +407,23 @@ boolean value, =true= or =false=. For example:
</decision>
...
</workflow-app>
-</verbatim>
+```
-The =name= attribute in the =decision= node is the name of the decision node.
+The `name` attribute in the `decision` node is the name of the decision node.
-Each =case= elements contains a predicate and a transition name. The predicate ELs are evaluated
-in order until one returns =true= and the corresponding transition is taken.
+Each `case` elements contains a predicate and a transition name. The predicate ELs are evaluated
+in order until one returns `true` and the corresponding transition is taken.
-The =default= element indicates the transition to take if none of the predicates evaluates
-to =true=.
+The `default` element indicates the transition to take if none of the predicates evaluates
+to `true`.
-All decision nodes must have a =default= element to avoid bringing the workflow into an error
+All decision nodes must have a `default` element to avoid bringing the workflow into an error
state if none of the predicates evaluates to true.
-*Example:*
+**Example:**
-<verbatim>
+
+```
<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0">
...
<decision name="mydecision">
@@ -408,21 +441,22 @@ state if none of the predicates evaluates to true.
</decision>
...
</workflow-app>
-</verbatim>
+```
+
+<a name="ForkJoinNodes"></a>
+#### 3.1.5 Fork and Join Control Nodes
-#ForkJoinNodes
----++++ 3.1.5 Fork and Join Control Nodes
+A `fork` node splits one path of execution into multiple concurrent paths of execution.
-A =fork= node splits one path of execution into multiple concurrent paths of execution.
+A `join` node waits until every concurrent execution path of a previous `fork` node arrives to it.
-A =join= node waits until every concurrent execution path of a previous =fork= node arrives to it.
+The `fork` and `join` nodes must be used in pairs. The `join` node assumes concurrent execution paths are children of
+the same `fork` node.
-The =fork= and =join= nodes must be used in pairs. The =join= node assumes concurrent execution paths are children of
-the same =fork= node.
+**Syntax:**
-*Syntax:*
-<verbatim>
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<fork name="[FORK-NODE-NAME]">
@@ -434,18 +468,19 @@ the same =fork= node.
<join name="[JOIN-NODE-NAME]" to="[NODE-NAME]" />
...
</workflow-app>
-</verbatim>
+```
-The =name= attribute in the =fork= node is the name of the workflow fork node. The =start= attribute in the =path=
-elements in the =fork= node indicate the name of the workflow node that will be part of the concurrent execution paths.
+The `name` attribute in the `fork` node is the name of the workflow fork node. The `start` attribute in the `path`
+elements in the `fork` node indicate the name of the workflow node that will be part of the concurrent execution paths.
-The =name= attribute in the =join= node is the name of the workflow join node. The =to= attribute in the =join= node
+The `name` attribute in the `join` node is the name of the workflow join node. The `to` attribute in the `join` node
indicates the name of the workflow node that will executed after all concurrent execution paths of the corresponding
fork arrive to the join node.
-*Example:*
+**Example:**
+
-<verbatim>
+```
<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
...
<fork name="forking">
@@ -473,37 +508,37 @@ fork arrive to the join node.
<join name="joining" to="nextaction"/>
...
</workflow-app>
-</verbatim>
+```
By default, Oozie performs some validation that any forking in a workflow is valid and won't lead to any incorrect behavior or
instability. However, if Oozie is preventing a workflow from being submitted and you are very certain that it should work, you can
disable forkjoin validation so that Oozie will accept the workflow. To disable this validation just for a specific workflow, simply
-set =oozie.wf.validate.ForkJoin= to =false= in the job.properties file. To disable this validation for all workflows, simply set
-=oozie.validate.ForkJoin= to =false= in the oozie-site.xml file. Disabling this validation is determined by the AND of both of
+set `oozie.wf.validate.ForkJoin` to `false` in the job.properties file. To disable this validation for all workflows, simply set
+`oozie.validate.ForkJoin` to `false` in the oozie-site.xml file. Disabling this validation is determined by the AND of both of
these properties, so it will be disabled if either or both are set to false and only enabled if both are set to true (or not
specified).
-#ActionNodes
----+++ 3.2 Workflow Action Nodes
+<a name="ActionNodes"></a>
+### 3.2 Workflow Action Nodes
Action nodes are the mechanism by which a workflow triggers the execution of a computation/processing task.
----++++ 3.2.1 Action Basis
+#### 3.2.1 Action Basis
The following sub-sections define common behavior and capabilities for all action types.
----+++++ 3.2.1.1 Action Computation/Processing Is Always Remote
+##### 3.2.1.1 Action Computation/Processing Is Always Remote
All computation/processing tasks triggered by an action node are remote to Oozie. No workflow application specific
computation/processing task is executed within Oozie.
----+++++ 3.2.1.2 Actions Are Asynchronous
+##### 3.2.1.2 Actions Are Asynchronous
All computation/processing tasks triggered by an action node are executed asynchronously by Oozie. For most types of
computation/processing tasks triggered by workflow action, the workflow job has to wait until the
computation/processing task completes before transitioning to the following node in the workflow.
-The exception is the =fs= action that is handled as a synchronous action.
+The exception is the `fs` action that is handled as a synchronous action.
Oozie can detect completion of computation/processing tasks by two different means, callbacks and polling.
@@ -514,19 +549,19 @@ For cases that the task failed to invoke the callback URL for any reason (i.e. a
the type of task cannot invoke the callback URL upon completion, Oozie has a mechanism to poll computation/processing
tasks for completion.
----+++++ 3.2.1.3 Actions Have 2 Transitions, =ok= and =error=
+##### 3.2.1.3 Actions Have 2 Transitions, `ok` and `error`
-If a computation/processing task -triggered by a workflow- completes successfully, it transitions to =ok=.
+If a computation/processing task -triggered by a workflow- completes successfully, it transitions to `ok`.
-If a computation/processing task -triggered by a workflow- fails to complete successfully, its transitions to =error=.
+If a computation/processing task -triggered by a workflow- fails to complete successfully, its transitions to `error`.
-If a computation/processing task exits in error, there computation/processing task must provide =error-code= and
- =error-message= information to Oozie. This information can be used from =decision= nodes to implement a fine grain
+If a computation/processing task exits in error, there computation/processing task must provide `error-code` and
+ `error-message` information to Oozie. This information can be used from `decision` nodes to implement a fine grain
error handling at workflow application level.
Each action type must clearly define all the error codes it can produce.
----+++++ 3.2.1.4 Action Recovery
+##### 3.2.1.4 Action Recovery
Oozie provides recovery capabilities when starting or ending actions.
@@ -554,13 +589,13 @@ administrator or an external managing system to perform any necessary cleanup be
If the failure is an error and a retry will not resolve the problem, Oozie will perform the error transition for the
action.
-#MapReduceAction
----++++ 3.2.2 Map-Reduce Action
+<a name="MapReduceAction"></a>
+#### 3.2.2 Map-Reduce Action
-The =map-reduce= action starts a Hadoop map/reduce job from a workflow. Hadoop jobs can be Java Map/Reduce jobs or
+The `map-reduce` action starts a Hadoop map/reduce job from a workflow. Hadoop jobs can be Java Map/Reduce jobs or
streaming jobs.
-A =map-reduce= action can be configured to perform file system cleanup and directory creation before starting the
+A `map-reduce` action can be configured to perform file system cleanup and directory creation before starting the
map reduce job. This capability enables Oozie to retry a Hadoop job in the situation of a transient failure (Hadoop
checks the non-existence of the job output directory and then creates it when the Hadoop job is starting, thus a retry
without cleanup of the job output directory would fail).
@@ -568,36 +603,37 @@ without cleanup of the job output directory would fail).
The workflow job will wait until the Hadoop map/reduce job completes before continuing to the next action in the
workflow execution path.
-The counters of the Hadoop job and job exit status (=FAILED=, =KILLED= or =SUCCEEDED=) must be available to the
+The counters of the Hadoop job and job exit status (`FAILED`, `KILLED` or `SUCCEEDED`) must be available to the
workflow job after the Hadoop jobs ends. This information can be used from within decision nodes and other actions
configurations.
-The =map-reduce= action has to be configured with all the necessary Hadoop JobConf properties to run the Hadoop
+The `map-reduce` action has to be configured with all the necessary Hadoop JobConf properties to run the Hadoop
map/reduce job.
-Hadoop JobConf properties can be specified as part of
- * the =config-default.xml= or
+Hadoop JobConf properties can be specified as part of
+
+ * the `config-default.xml` or
* JobConf XML file bundled with the workflow application or
- * <global> tag in workflow definition or
- * Inline =map-reduce= action configuration or
- * An implementation of OozieActionConfigurator specified by the <config-class> tag in workflow definition.
+ * \<global\> tag in workflow definition or
+ * Inline `map-reduce` action configuration or
+ * An implementation of OozieActionConfigurator specified by the \<config-class\> tag in workflow definition.
-The configuration properties are loaded in the following above order i.e. =streaming=, =job-xml=, =configuration=,
-and =config-class=, and the precedence order is later values override earlier values.
+The configuration properties are loaded in the following above order i.e. `streaming`, `job-xml`, `configuration`,
+and `config-class`, and the precedence order is later values override earlier values.
Streaming and inline property values can be parameterized (templatized) using EL expressions.
-The Hadoop =mapred.job.tracker= and =fs.default.name= properties must not be present in the job-xml and inline
+The Hadoop `mapred.job.tracker` and `fs.default.name` properties must not be present in the job-xml and inline
configuration.
-#FilesArchives
----+++++ 3.2.2.1 Adding Files and Archives for the Job
+<a name="FilesArchives"></a>
+##### 3.2.2.1 Adding Files and Archives for the Job
-The =file=, =archive= elements make available, to map-reduce jobs, files and archives. If the specified path is
+The `file`, `archive` elements make available, to map-reduce jobs, files and archives. If the specified path is
relative, it is assumed the file or archiver are within the application directory, in the corresponding sub-path.
If the path is absolute, the file or archive it is expected in the given absolute path.
-Files specified with the =file= element, will be symbolic links in the home directory of the task.
+Files specified with the `file` element, will be symbolic links in the home directory of the task.
If a file is a native library (an '.so' or a '.so.#' file), it will be symlinked as and '.so' file in the task running
directory, thus available to the task JVM.
@@ -607,27 +643,29 @@ To force a symlink for a file on the task running directory, use a '#' followed
Refer to Hadoop distributed cache documentation for details more details on files and archives.
----+++++ 3.2.2.2 Configuring the MapReduce action with Java code
+##### 3.2.2.2 Configuring the MapReduce action with Java code
Java code can be used to further configure the MapReduce action. This can be useful if you already have "driver" code for your
MapReduce action, if you're more familiar with MapReduce's Java API, if there's some configuration that requires logic, or some
configuration that's difficult to do in straight XML (e.g. Avro).
Create a class that implements the org.apache.oozie.action.hadoop.OozieActionConfigurator interface from the "oozie-sharelib-oozie"
-artifact. It contains a single method that receives a =JobConf= as an argument. Any configuration properties set on this =JobConf=
+artifact. It contains a single method that receives a `JobConf` as an argument. Any configuration properties set on this `JobConf`
will be used by the MapReduce action.
The OozieActionConfigurator has this signature:
-<verbatim>
+
+```
public interface OozieActionConfigurator {
public void configure(JobConf actionConf) throws OozieActionConfiguratorException;
}
-</verbatim>
-where =actionConf= is the =JobConf= you can update. If you need to throw an Exception, you can wrap it in
-an =OozieActionConfiguratorException=, also in the "oozie-sharelib-oozie" artifact.
+```
+where `actionConf` is the `JobConf` you can update. If you need to throw an Exception, you can wrap it in
+an `OozieActionConfiguratorException`, also in the "oozie-sharelib-oozie" artifact.
For example:
-<verbatim>
+
+```
package com.example;
import org.apache.hadoop.fs.Path;
@@ -653,11 +691,12 @@ public class MyConfigClass implements OozieActionConfigurator {
...
}
}
-</verbatim>
+```
To use your config class in your MapReduce action, simply compile it into a jar, make the jar available to your action, and specify
-the class name in the =config-class= element (this requires at least schema 0.5):
-<verbatim>
+the class name in the `config-class` element (this requires at least schema 0.5):
+
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<action name="[NODE-NAME]">
@@ -679,51 +718,52 @@ the class name in the =config-class= element (this requires at least schema 0.5)
</action>
...
</workflow-app>
-</verbatim>
+```
Another example of this can be found in the "map-reduce" example that comes with Oozie.
-A useful tip: The initial =JobConf= passed to the =configure= method includes all of the properties listed in the =configuration=
+A useful tip: The initial `JobConf` passed to the `configure` method includes all of the properties listed in the `configuration`
section of the MR action in a workflow. If you need to pass any information to your OozieActionConfigurator, you can simply put
them here.
-#StreamingMapReduceAction
----+++++ 3.2.2.3 Streaming
+<a name="StreamingMapReduceAction"></a>
+##### 3.2.2.3 Streaming
-Streaming information can be specified in the =streaming= element.
+Streaming information can be specified in the `streaming` element.
-The =mapper= and =reducer= elements are used to specify the executable/script to be used as mapper and reducer.
+The `mapper` and `reducer` elements are used to specify the executable/script to be used as mapper and reducer.
-User defined scripts must be bundled with the workflow application and they must be declared in the =files= element of
-the streaming configuration. If the are not declared in the =files= element of the configuration it is assumed they
+User defined scripts must be bundled with the workflow application and they must be declared in the `files` element of
+the streaming configuration. If the are not declared in the `files` element of the configuration it is assumed they
will be available (and in the command PATH) of the Hadoop slave machines.
Some streaming jobs require Files found on HDFS to be available to the mapper/reducer scripts. This is done using
-the =file= and =archive= elements described in the previous section.
+the `file` and `archive` elements described in the previous section.
-The Mapper/Reducer can be overridden by a =mapred.mapper.class= or =mapred.reducer.class= properties in the =job-xml=
-file or =configuration= elements.
+The Mapper/Reducer can be overridden by a `mapred.mapper.class` or `mapred.reducer.class` properties in the `job-xml`
+file or `configuration` elements.
-#PipesMapReduceAction
----+++++ 3.2.2.4 Pipes
+<a name="PipesMapReduceAction"></a>
+##### 3.2.2.4 Pipes
-Pipes information can be specified in the =pipes= element.
+Pipes information can be specified in the `pipes` element.
A subset of the command line options which can be used while using the Hadoop Pipes Submitter can be specified
-via elements - =map=, =reduce=, =inputformat=, =partitioner=, =writer=, =program=.
+via elements - `map`, `reduce`, `inputformat`, `partitioner`, `writer`, `program`.
-The =program= element is used to specify the executable/script to be used.
+The `program` element is used to specify the executable/script to be used.
User defined program must be bundled with the workflow application.
Some pipe jobs require Files found on HDFS to be available to the mapper/reducer scripts. This is done using
-the =file= and =archive= elements described in the previous section.
+the `file` and `archive` elements described in the previous section.
+
+Pipe properties can be overridden by specifying them in the `job-xml` file or `configuration` element.
-Pipe properties can be overridden by specifying them in the =job-xml= file or =configuration= element.
+##### 3.2.2.5 Syntax
----+++++ 3.2.2.5 Syntax
-<verbatim>
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<action name="[NODE-NAME]">
@@ -774,40 +814,41 @@ Pipe properties can be overridden by specifying them in the =job-xml= file or =c
</action>
...
</workflow-app>
-</verbatim>
+```
-The =prepare= element, if present, indicates a list of paths to delete before starting the job. This should be used
+The `prepare` element, if present, indicates a list of paths to delete before starting the job. This should be used
exclusively for directory cleanup or dropping of hcatalog table or table partitions for the job to be executed. The delete operation
-will be performed in the =fs.default.name= filesystem for hdfs URIs. The format for specifying hcatalog table URI is
+will be performed in the `fs.default.name` filesystem for hdfs URIs. The format for specifying hcatalog table URI is
hcat://[metastore server]:[port]/[database name]/[table name] and format to specify a hcatalog table partition URI is
-hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value].
-In case of a hcatalog URI, the hive-site.xml needs to be shipped using =file= tag and the hcatalog and hive jars
-need to be placed in workflow lib directory or specified using =archive= tag.
+`hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value]`.
+In case of a hcatalog URI, the hive-site.xml needs to be shipped using `file` tag and the hcatalog and hive jars
+need to be placed in workflow lib directory or specified using `archive` tag.
-The =job-xml= element, if present, must refer to a Hadoop JobConf =job.xml= file bundled in the workflow application.
-By default the =job.xml= file is taken from the workflow application namenode, regardless the namenode specified for the action.
-To specify a =job.xml= on another namenode use a fully qualified file path.
-The =job-xml= element is optional and as of schema 0.4, multiple =job-xml= elements are allowed in order to specify multiple Hadoop JobConf =job.xml= files.
+The `job-xml` element, if present, must refer to a Hadoop JobConf `job.xml` file bundled in the workflow application.
+By default the `job.xml` file is taken from the workflow application namenode, regardless the namenode specified for the action.
+To specify a `job.xml` on another namenode use a fully qualified file path.
+The `job-xml` element is optional and as of schema 0.4, multiple `job-xml` elements are allowed in order to specify multiple Hadoop JobConf `job.xml` files.
-The =configuration= element, if present, contains JobConf properties for the Hadoop job.
+The `configuration` element, if present, contains JobConf properties for the Hadoop job.
-Properties specified in the =configuration= element override properties specified in the file specified in the
- =job-xml= element.
+Properties specified in the `configuration` element override properties specified in the file specified in the
+ `job-xml` element.
-As of schema 0.5, the =config-class= element, if present, contains a class that implements OozieActionConfigurator that can be used
+As of schema 0.5, the `config-class` element, if present, contains a class that implements OozieActionConfigurator that can be used
to further configure the MapReduce job.
-Properties specified in the =config-class= class override properties specified in =configuration= element.
+Properties specified in the `config-class` class override properties specified in `configuration` element.
External Stats can be turned on/off by specifying the property _oozie.action.external.stats.write_ as _true_ or _false_ in the configuration element of workflow.xml. The default value for this property is _false_.
-The =file= element, if present, must specify the target symbolic link for binaries by separating the original file and target with a # (file#target-sym-link). This is not required for libraries.
+The `file` element, if present, must specify the target symbolic link for binaries by separating the original file and target with a # (file#target-sym-link). This is not required for libraries.
-The =mapper= and =reducer= process for streaming jobs, should specify the executable command with URL encoding. e.g. '%' should be replaced by '%25'.
+The `mapper` and `reducer` process for streaming jobs, should specify the executable command with URL encoding. e.g. '%' should be replaced by '%25'.
-*Example:*
+**Example:**
-<verbatim>
+
+```
<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:1.0">
...
<action name="myfirstHadoopJob">
@@ -842,14 +883,15 @@ The =mapper= and =reducer= process for streaming jobs, should specify the execut
</action>
...
</workflow-app>
-</verbatim>
+```
In the above example, the number of Reducers to be used by the Map/Reduce job has to be specified as a parameter of
the workflow job configuration when creating the workflow job.
-*Streaming Example:*
+**Streaming Example:**
+
-<verbatim>
+```
<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
...
<action name="firstjob">
@@ -885,12 +927,13 @@ the workflow job configuration when creating the workflow job.
</action>
...
</workflow-app>
-</verbatim>
+```
-*Pipes Example:*
+**Pipes Example:**
-<verbatim>
+
+```
<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
...
<action name="firstjob">
@@ -920,43 +963,45 @@ the workflow job configuration when creating the workflow job.
</action>
...
</workflow-app>
-</verbatim>
+```
-#PigAction
----++++ 3.2.3 Pig Action
+<a name="PigAction"></a>
+#### 3.2.3 Pig Action
-The =pig= action starts a Pig job.
+The `pig` action starts a Pig job.
The workflow job will wait until the pig job completes before continuing to the next action.
-The =pig= action has to be configured with the resource-manager, name-node, pig script and the necessary parameters and
+The `pig` action has to be configured with the resource-manager, name-node, pig script and the necessary parameters and
configuration to run the Pig job.
-A =pig= action can be configured to perform HDFS files/directories cleanup or HCatalog partitions cleanup before
+A `pig` action can be configured to perform HDFS files/directories cleanup or HCatalog partitions cleanup before
starting the Pig job. This capability enables Oozie to retry a Pig job in the situation of a transient failure (Pig
creates temporary directories for intermediate data, thus a retry without cleanup would fail).
-Hadoop JobConf properties can be specified as part of
- * the =config-default.xml= or
+Hadoop JobConf properties can be specified as part of
+
+ * the `config-default.xml` or
* JobConf XML file bundled with the workflow application or
- * <global> tag in workflow definition or
- * Inline =pig= action configuration.
+ * \<global\> tag in workflow definition or
+ * Inline `pig` action configuration.
-The configuration properties are loaded in the following above order i.e. =job-xml= and =configuration=, and
+The configuration properties are loaded in the following above order i.e. `job-xml` and `configuration`, and
the precedence order is later values override earlier values.
Inline property values can be parameterized (templatized) using EL expressions.
-The YARN =yarn.resourcemanager.address= and HDFS =fs.default.name= properties must not be present in the job-xml and inline
+The YARN `yarn.resourcemanager.address` and HDFS `fs.default.name` properties must not be present in the job-xml and inline
configuration.
As with Hadoop map-reduce jobs, it is possible to add files and archives to be available to the Pig job, refer to
section [#FilesArchives][Adding Files and Archives for the Job].
-*Syntax for Pig actions in Oozie schema 1.0:*
-<verbatim>
+**Syntax for Pig actions in Oozie schema 1.0:**
+
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<action name="[NODE-NAME]">
@@ -994,10 +1039,11 @@ section [#FilesArchives][Adding Files and Archives for the Job].
</action>
...
</workflow-app>
-</verbatim>
+```
-*Syntax for Pig actions in Oozie schema 0.2:*
-<verbatim>
+**Syntax for Pig actions in Oozie schema 0.2:**
+
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.2">
...
<action name="[NODE-NAME]">
@@ -1035,11 +1081,12 @@ section [#FilesArchives][Adding Files and Archives for the Job].
</action>
...
</workflow-app>
-</verbatim>
+```
+
+**Syntax for Pig actions in Oozie schema 0.1:**
-*Syntax for Pig actions in Oozie schema 0.1:*
-<verbatim>
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:0.1">
...
<action name="[NODE-NAME]">
@@ -1074,46 +1121,47 @@ section [#FilesArchives][Adding Files and Archives for the Job].
</action>
...
</workflow-app>
-</verbatim>
+```
-The =prepare= element, if present, indicates a list of paths to delete before starting the job. This should be used
+The `prepare` element, if present, indicates a list of paths to delete before starting the job. This should be used
exclusively for directory cleanup or dropping of hcatalog table or table partitions for the job to be executed. The delete operation
-will be performed in the =fs.default.name= filesystem for hdfs URIs. The format for specifying hcatalog table URI is
+will be performed in the `fs.default.name` filesystem for hdfs URIs. The format for specifying hcatalog table URI is
hcat://[metastore server]:[port]/[database name]/[table name] and format to specify a hcatalog table partition URI is
-hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value].
-In case of a hcatalog URI, the hive-site.xml needs to be shipped using =file= tag and the hcatalog and hive jars
-need to be placed in workflow lib directory or specified using =archive= tag.
+`hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value]`.
+In case of a hcatalog URI, the hive-site.xml needs to be shipped using `file` tag and the hcatalog and hive jars
+need to be placed in workflow lib directory or specified using `archive` tag.
-The =job-xml= element, if present, must refer to a Hadoop JobConf =job.xml= file bundled in the workflow application.
-The =job-xml= element is optional and as of schema 0.4, multiple =job-xml= elements are allowed in order to specify multiple Hadoop JobConf =job.xml= files.
+The `job-xml` element, if present, must refer to a Hadoop JobConf `job.xml` file bundled in the workflow application.
+The `job-xml` element is optional and as of schema 0.4, multiple `job-xml` elements are allowed in order to specify multiple Hadoop JobConf `job.xml` files.
-The =configuration= element, if present, contains JobConf properties for the underlying Hadoop jobs.
+The `configuration` element, if present, contains JobConf properties for the underlying Hadoop jobs.
-Properties specified in the =configuration= element override properties specified in the file specified in the
- =job-xml= element.
+Properties specified in the `configuration` element override properties specified in the file specified in the
+ `job-xml` element.
External Stats can be turned on/off by specifying the property _oozie.action.external.stats.write_ as _true_ or _false_ in the configuration element of workflow.xml. The default value for this property is _false_.
The inline and job-xml configuration properties are passed to the Hadoop jobs submitted by Pig runtime.
-The =script= element contains the pig script to execute. The pig script can be templatized with variables of the
-form =${VARIABLE}=. The values of these variables can then be specified using the =params= element.
+The `script` element contains the pig script to execute. The pig script can be templatized with variables of the
+form `${VARIABLE}`. The values of these variables can then be specified using the `params` element.
NOTE: Oozie will perform the parameter substitution before firing the pig job. This is different from the
-[[http://wiki.apache.org/pig/ParameterSubstitution][parameter substitution mechanism provided by Pig]], which has a
+[parameter substitution mechanism provided by Pig](http://wiki.apache.org/pig/ParameterSubstitution), which has a
few limitations.
-The =params= element, if present, contains parameters to be passed to the pig script.
+The `params` element, if present, contains parameters to be passed to the pig script.
-*In Oozie schema 0.2:*
-The =arguments= element, if present, contains arguments to be passed to the pig script.
+**In Oozie schema 0.2:**
+The `arguments` element, if present, contains arguments to be passed to the pig script.
All the above elements can be parameterized (templatized) using EL expressions.
-*Example for Oozie schema 0.2:*
+**Example for Oozie schema 0.2:**
+
-<verbatim>
+```
<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.2">
...
<action name="myfirstpigjob">
@@ -1144,12 +1192,13 @@ All the above elements can be parameterized (templatized) using EL expressions.
</action>
...
</workflow-app>
-</verbatim>
+```
-*Example for Oozie schema 0.1:*
+**Example for Oozie schema 0.1:**
-<verbatim>
+
+```
<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1">
...
<action name="myfirstpigjob">
@@ -1174,35 +1223,36 @@ All the above elements can be parameterized (templatized) using EL expressions.
</action>
...
</workflow-app>
-</verbatim>
+```
-#FsAction
----++++ 3.2.4 Fs (HDFS) action
+<a name="FsAction"></a>
+#### 3.2.4 Fs (HDFS) action
-The =fs= action allows to manipulate files and directories in HDFS from a workflow application. The supported commands
-are =move=, =delete=, =mkdir=, =chmod=, =touchz=, =setrep= and =chgrp=.
+The `fs` action allows to manipulate files and directories in HDFS from a workflow application. The supported commands
+are `move`, `delete`, `mkdir`, `chmod`, `touchz`, `setrep` and `chgrp`.
The FS commands are executed synchronously from within the FS action, the workflow job will wait until the specified
file commands are completed before continuing to the next action.
-Path names specified in the =fs= action can be parameterized (templatized) using EL expressions.
-Path name should be specified as a absolute path. In case of =move=, =delete=, =chmod= and =chgrp= commands, a glob pattern can also be specified instead of an absolute path.
-For =move=, glob pattern can only be specified for source path and not the target.
+Path names specified in the `fs` action can be parameterized (templatized) using EL expressions.
+Path name should be specified as a absolute path. In case of `move`, `delete`, `chmod` and `chgrp` commands, a glob pattern can also be specified instead of an absolute path.
+For `move`, glob pattern can only be specified for source path and not the target.
Each file path must specify the file system URI, for move operations, the target must not specify the system URI.
-*IMPORTANT:* For the purposes of copying files within a cluster it is recommended to refer to the =distcp= action
-instead. Refer to [[DG_DistCpActionExtension][=distcp=]] action to copy files within a cluster.
+**IMPORTANT:** For the purposes of copying files within a cluster it is recommended to refer to the `distcp` action
+instead. Refer to [`distcp`](DG_DistCpActionExtension.html) action to copy files within a cluster.
+
+**IMPORTANT:** All the commands within `fs` action do not happen atomically, if a `fs` action fails half way in the
+commands being executed, successfully executed commands are not rolled back. The `fs` action, before executing any
+command must check that source paths exist and target paths don't exist (constraint regarding target relaxed for the `move` action. See below for details), thus failing before executing any command.
+Therefore the validity of all paths specified in one `fs` action are evaluated before any of the file operation are
+executed. Thus there is less chance of an error occurring while the `fs` action executes.
-*IMPORTANT:* All the commands within =fs= action do not happen atomically, if a =fs= action fails half way in the
-commands being executed, successfully executed commands are not rolled back. The =fs= action, before executing any
-command must check that source paths exist and target paths don't exist (constraint regarding target relaxed for the =move= action. See below for details), thus failing before executing any command.
-Therefore the validity of all paths specified in one =fs= action are evaluated before any of the file operation are
-executed. Thus there is less chance of an error occurring while the =fs= action executes.
+**Syntax:**
-*Syntax:*
-<verbatim>
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<action name="[NODE-NAME]">
@@ -1226,50 +1276,52 @@ executed. Thus there is less chance of an error occurring while the =fs= action
</action>
...
</workflow-app>
-</verbatim>
+```
-The =delete= command deletes the specified path, if it is a directory it deletes recursively all its content and then
+The `delete` command deletes the specified path, if it is a directory it deletes recursively all its content and then
deletes the directory. By default it does skip trash. It can be moved to trash by setting the value of skip-trash to
'false'. It can also be used to drop hcat tables/partitions. This is the only FS command which supports HCatalog URIs as well.
For eg:
-<verbatim>
+
+```
<delete path='hcat://[metastore server]:[port]/[database name]/[table name]'/>
OR
<delete path='hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value];...'/>
-</verbatim>
+```
-The =mkdir= command creates the specified directory, it creates all missing directories in the path. If the directory
+The `mkdir` command creates the specified directory, it creates all missing directories in the path. If the directory
already exist it does a no-op.
-In the =move= command the =source= path must exist. The following scenarios are addressed for a =move=:
+In the `move` command the `source` path must exist. The following scenarios are addressed for a `move`:
- * The file system URI(e.g. hdfs://{nameNode}) can be skipped in the =target= path. It is understood to be the same as that of the source. But if the target path does contain the system URI, it cannot be different than that of the source.
- * The parent directory of the =target= path must exist
- * For the =target= path, if it is a file, then it must not already exist.
- * However, if the =target= path is an already existing directory, the =move= action will place your =source= as a child of the =target= directory.
+ * The file system URI(e.g. `hdfs://{nameNode}`) can be skipped in the `target` path. It is understood to be the same as that of the source. But if the target path does contain the system URI, it cannot be different than that of the source.
+ * The parent directory of the `target` path must exist
+ * For the `target` path, if it is a file, then it must not already exist.
+ * However, if the `target` path is an already existing directory, the `move` action will place your `source` as a child of the `target` directory.
-The =chmod= command changes the permissions for the specified path. Permissions can be specified using the Unix Symbolic
+The `chmod` command changes the permissions for the specified path. Permissions can be specified using the Unix Symbolic
representation (e.g. -rwxrw-rw-) or an octal representation (755).
-When doing a =chmod= command on a directory, by default the command is applied to the directory and the files one level
-within the directory. To apply the =chmod= command to the directory, without affecting the files within it,
-the =dir-files= attribute must be set to =false=. To apply the =chmod= command
-recursively to all levels within a directory, put a =recursive= element inside the <chmod> element.
+When doing a `chmod` command on a directory, by default the command is applied to the directory and the files one level
+within the directory. To apply the `chmod` command to the directory, without affecting the files within it,
+the `dir-files` attribute must be set to `false`. To apply the `chmod` command
+recursively to all levels within a directory, put a `recursive` element inside the \<chmod\> element.
-The =touchz= command creates a zero length file in the specified path if none exists. If one already exists, then touchz will perform a touch operation.
+The `touchz` command creates a zero length file in the specified path if none exists. If one already exists, then touchz will perform a touch operation.
Touchz works only for absolute paths.
-The =chgrp= command changes the group for the specified path.
-When doing a =chgrp= command on a directory, by default the command is applied to the directory and the files one level
-within the directory. To apply the =chgrp= command to the directory, without affecting the files within it,
-the =dir-files= attribute must be set to =false=.
-To apply the =chgrp= command recursively to all levels within a directory, put a =recursive= element inside the <chgrp> element.
+The `chgrp` command changes the group for the specified path.
+When doing a `chgrp` command on a directory, by default the command is applied to the directory and the files one level
+within the directory. To apply the `chgrp` command to the directory, without affecting the files within it,
+the `dir-files` attribute must be set to `false`.
+To apply the `chgrp` command recursively to all levels within a directory, put a `recursive` element inside the \<chgrp\> element.
-The =setrep= command changes replication factor of an hdfs file(s). Changing RF of directories or symlinks is not
+The `setrep` command changes replication factor of an hdfs file(s). Changing RF of directories or symlinks is not
supported; this action requires an argument for RF.
-*Example:*
+**Example:**
-<verbatim>
+
+```
<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
...
<action name="hdfscommands">
@@ -1286,25 +1338,26 @@ supported; this action requires an argument for RF.
</action>
...
</workflow-app>
-</verbatim>
+```
In the above example, a directory named after the workflow job ID is created and the input of the job, passed as
workflow configuration parameter, is archived under the previously created directory.
-As of schema 0.4, if a =name-node= element is specified, then it is not necessary for any of the paths to start with the file system
-URI as it is taken from the =name-node= element. This is also true if the name-node is specified in the global section
-(see [[WorkflowFunctionalSpec#GlobalConfigurations][Global Configurations]])
+As of schema 0.4, if a `name-node` element is specified, then it is not necessary for any of the paths to start with the file system
+URI as it is taken from the `name-node` element. This is also true if the name-node is specified in the global section
+(see [Global Configurations](WorkflowFunctionalSpec.html#GlobalConfigurations))
-As of schema 0.4, zero or more =job-xml= elements can be specified; these must refer to Hadoop JobConf =job.xml= formatted files
+As of schema 0.4, zero or more `job-xml` elements can be specified; these must refer to Hadoop JobConf `job.xml` formatted files
bundled in the workflow application. They can be used to set additional properties for the FileSystem instance.
-As of schema 0.4, if a =configuration= element is specified, then it will also be used to set additional JobConf properties for the
-FileSystem instance. Properties specified in the =configuration= element override properties specified in the files specified
-by any =job-xml= elements.
+As of schema 0.4, if a `configuration` element is specified, then it will also be used to set additional JobConf properties for the
+FileSystem instance. Properties specified in the `configuration` element override properties specified in the files specified
+by any `job-xml` elements.
+
+**Example:**
-*Example:*
-<verbatim>
+```
<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.4">
...
<action name="hdfscommands">
@@ -1324,21 +1377,22 @@ by any =job-xml= elements.
</action>
...
</workflow-app>
-</verbatim>
+```
-#SubWorkflowAction
----++++ 3.2.5 Sub-workflow Action
+<a name="SubWorkflowAction"></a>
+#### 3.2.5 Sub-workflow Action
-The =sub-workflow= action runs a child workflow job, the child workflow job can be in the same Oozie system or in
+The `sub-workflow` action runs a child workflow job, the child workflow job can be in the same Oozie system or in
another Oozie system.
The parent workflow job will wait until the child workflow job has completed.
There can be several sub-workflows defined within a single workflow, each under its own action element.
-*Syntax:*
+**Syntax:**
-<verbatim>
+
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<action name="[NODE-NAME]">
@@ -1358,22 +1412,23 @@ There can be several sub-workflows defined within a single workflow, each under
</action>
...
</workflow-app>
-</verbatim>
+```
The child workflow job runs in the same Oozie system instance where the parent workflow job is running.
-The =app-path= element specifies the path to the workflow application of the child workflow job.
+The `app-path` element specifies the path to the workflow application of the child workflow job.
-The =propagate-configuration= flag, if present, indicates that the workflow job configuration should be propagated to
+The `propagate-configuration` flag, if present, indicates that the workflow job configuration should be propagated to
the child workflow.
-The =configuration= section can be used to specify the job properties that are required to run the child workflow job.
+The `configuration` section can be used to specify the job properties that are required to run the child workflow job.
+
+The configuration of the `sub-workflow` action can be parameterized (templatized) using EL expressions.
-The configuration of the =sub-workflow= action can be parameterized (templatized) using EL expressions.
+**Example:**
-*Example:*
-<verbatim>
+```
<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
...
<action name="a">
@@ -1391,94 +1446,97 @@ The configuration of the =sub-workflow= action can be parameterized (templatized
</action>
...
</workflow-app>
-</verbatim>
+```
-In the above example, the workflow definition with the name =child-wf= will be run on the Oozie instance at
- =.http://myhost:11000/oozie=. The specified workflow application must be already deployed on the target Oozie instance.
+In the above example, the workflow definition with the name `child-wf` will be run on the Oozie instance at
+ `.http://myhost:11000/oozie`. The specified workflow application must be already deployed on the target Oozie instance.
-A configuration parameter =input.dir= is being passed as job property to the child workflow job.
+A configuration parameter `input.dir` is being passed as job property to the child workflow job.
-The subworkflow can inherit the lib jars from the parent workflow by setting =oozie.subworkflow.classpath.inheritance= to true
-in oozie-site.xml or on a per-job basis by setting =oozie.wf.subworkflow.classpath.inheritance= to true in a job.properties file.
-If both are specified, =oozie.wf.subworkflow.classpath.inheritance= has priority. If the subworkflow and the parent have
-conflicting jars, the subworkflow's jar has priority. By default, =oozie.wf.subworkflow.classpath.inheritance= is set to false.
+The subworkflow can inherit the lib jars from the parent workflow by setting `oozie.subworkflow.classpath.inheritance` to true
+in oozie-site.xml or on a per-job basis by setting `oozie.wf.subworkflow.classpath.inheritance` to true in a job.properties file.
+If both are specified, `oozie.wf.subworkflow.classpath.inheritance` has priority. If the subworkflow and the parent have
+conflicting jars, the subworkflow's jar has priority. By default, `oozie.wf.subworkflow.classpath.inheritance` is set to false.
-To prevent errant workflows from starting infinitely recursive subworkflows, =oozie.action.subworkflow.max.depth= can be specified
+To prevent errant workflows from starting infinitely recursive subworkflows, `oozie.action.subworkflow.max.depth` can be specified
in oozie-site.xml to set the maximum depth of subworkflow calls. For example, if set to 3, then a workflow can start subwf1, which
can start subwf2, which can start subwf3; but if subwf3 tries to start subwf4, then the action will fail. The default is 50.
-#JavaAction
----++++ 3.2.6 Java Action
+<a name="JavaAction"></a>
+#### 3.2.6 Java Action
-The =java= action will execute the =public static void main(String[] args)= method of the specified main Java class.
+The `java` action will execute the `public static void main(String[] args)` method of the specified main Java class.
Java applications are executed in the Hadoop cluster as map-reduce job with a single Mapper task.
The workflow job will wait until the java application completes its execution before continuing to the next action.
-The =java= action has to be configured with the resource-manager, name-node, main Java class, JVM options and arguments.
+The `java` action has to be configured with the resource-manager, name-node, main Java class, JVM options and arguments.
-To indicate an =ok= action transition, the main Java class must complete gracefully the =main= method invocation.
+To indicate an `ok` action transition, the main Java class must complete gracefully the `main` method invocation.
-To indicate an =error= action transition, the main Java class must throw an exception.
+To indicate an `error` action transition, the main Java class must throw an exception.
-The main Java class can call =System.exit(int n)=. Exit code zero is regarded as OK, while non-zero exit codes will
-cause the =java= action to do an =error= transition and exit.
+The main Java class can call `System.exit(int n)`. Exit code zero is regarded as OK, while non-zero exit codes will
+cause the `java` action to do an `error` transition and exit.
-A =java= action can be configured to perform HDFS files/directories cleanup or HCatalog partitions cleanup before
+A `java` action can be configured to perform HDFS files/directories cleanup or HCatalog partitions cleanup before
starting the Java application. This capability enables Oozie to retry a Java application in the situation of a transient
or non-transient failure (This can be used to cleanup any temporary data which may have been created by the Java
application in case of failure).
-A =java= action can create a Hadoop configuration for interacting with a cluster (e.g. launching a map-reduce job).
-Oozie prepares a Hadoop configuration file which includes the environments site configuration files (e.g. hdfs-site.xml,
-mapred-site.xml, etc) plus the properties added to the =<configuration>= section of the =java= action. The Hadoop configuration
-file is made available as a local file to the Java application in its running directory. It can be added to the =java= actions
-Hadoop configuration by referencing the system property: =oozie.action.conf.xml=. For example:
+A `java` action can create a Hadoop configuration for interacting with a cluster (e.g. launching a map-reduce job).
+Oozie prepares a Hadoop configuration file which includes the environments site configuration files (e.g. hdfs-site.xml,
+mapred-site.xml, etc) plus the properties added to the `<configuration>` section of the `java` action. The Hadoop configuration
+file is made available as a local file to the Java application in its running directory. It can be added to the `java` actions
+Hadoop configuration by referencing the system property: `oozie.action.conf.xml`. For example:
-<verbatim>
+
+```
// loading action conf prepared by Oozie
Configuration actionConf = new Configuration(false);
actionConf.addResource(new Path("file:///", System.getProperty("oozie.action.conf.xml")));
-</verbatim>
+```
-If =oozie.action.conf.xml= is not added then the job will pick up the mapred-default properties and this may result
+If `oozie.action.conf.xml` is not added then the job will pick up the mapred-default properties and this may result
in unexpected behaviour. For repeated configuration properties later values override earlier ones.
Inline property values can be parameterized (templatized) using EL expressions.
-The YARN =yarn.resourcemanager.address= (=resource-manager=) and HDFS =fs.default.name= (=name-node=) properties must not be present
-in the =job-xml= and in the inline configuration.
+The YARN `yarn.resourcemanager.address` (`resource-manager`) and HDFS `fs.default.name` (`name-node`) properties must not be present
+in the `job-xml` and in the inline configuration.
-As with =map-reduce= and =pig= actions, it is possible to add files and archives to be available to the Java
+As with `map-reduce` and `pig` actions, it is possible to add files and archives to be available to the Java
application. Refer to section [#FilesArchives][Adding Files and Archives for the Job].
-The =capture-output= element can be used to propagate values back into Oozie context, which can then be accessed via
-EL-functions. This needs to be written out as a java properties format file. The filename is obtained via a System
-property specified by the constant =oozie.action.output.properties=
+The `capture-output` element can be used to propagate values back into Oozie context, which can then be accessed via
+EL-functions. This needs to be written out as a java properties format file. The filename is obtained via a System
+property specified by the constant `oozie.action.output.properties`
-*IMPORTANT:* In order for a Java action to succeed on a secure cluster, it must propagate the Hadoop delegation token like in the
+**IMPORTANT:** In order for a Java action to succeed on a secure cluster, it must propagate the Hadoop delegation token like in the
following code snippet (this is benign on non-secure clusters):
-<verbatim>
+
+```
// propagate delegation related props from launcher job to MR job
if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
jobConf.set("mapreduce.job.credentials.binary", System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
}
-</verbatim>
+```
-*IMPORTANT:* Because the Java application is run from within a Map-Reduce job, from Hadoop 0.20. onwards a queue must
+**IMPORTANT:** Because the Java application is run from within a Map-Reduce job, from Hadoop 0.20. onwards a queue must
be assigned to it. The queue name must be specified as a configuration property.
-*IMPORTANT:* The Java application from a Java action is executed in a single map task. If the task is abnormally terminated,
+**IMPORTANT:** The Java application from a Java action is executed in a single map task. If the task is abnormally terminated,
such as due to a TaskTracker restart (e.g. during cluster maintenance), the task will be retried via the normal Hadoop task
retry mechanism. To avoid workflow failure, the application should be written in a fashion that is resilient to such retries,
for example by detecting and deleting incomplete outputs or picking back up from complete outputs. Furthermore, if a Java action
spawns asynchronous activity outside the JVM of the action itself (such as by launching additional MapReduce jobs), the
application must consider the possibility of collisions with activity spawned by the new instance.
-*Syntax:*
+**Syntax:**
+
-<verbatim>
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<action name="[NODE-NAME]">
@@ -1514,41 +1572,43 @@ application must consider the possibility of collisions with activity spawned by
</action>
...
</workflow-app>
-</verbatim>
+```
-The =prepare= element, if present, indicates a list of paths to delete before starting the Java application. This should
+The `prepare` element, if present, indicates a list of paths to delete before starting the Java application. This should
be used exclusively for directory cleanup or dropping of hcatalog table or table partitions for the Java application to be executed.
-In case of =delete=, a glob pattern can be used to specify path.
+In case of `delete`, a glob pattern can be used to specify path.
The format for specifying hcatalog table URI is
hcat://[metastore server]:[port]/[database name]/[table name] and format to specify a hcatalog table partition URI is
-hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value].
-In case of a hcatalog URI, the hive-site.xml needs to be shipped using =file= tag and the hcatalog and hive jars
-need to be placed in workflow lib directory or specified using =archive= tag.
+`hcat://[metastore server]:[port]/[database name]/[table name]/[partkey1]=[value];[partkey2]=[value]`.
+In case of a hcatalog URI, the hive-site.xml needs to be shipped using `file` tag and the hcatalog and hive jars
+need to be placed in workflow lib directory or specified using `archive` tag.
-The =java-opts= and =java-opt= elements, if present, contains the command line parameters which are to be used to start the JVM that
-will execute the Java application. Using this element is equivalent to using the =mapred.child.java.opts=
-or =mapreduce.map.java.opts= configuration properties, with the advantage that Oozie will append to these properties instead of
+The `java-opts` and `java-opt` elements, if present, contains the command line parameters which are to be used to start the JVM that
+will execute the Java application. Using this element is equivalent to using the `mapred.child.java.opts`
+or `mapreduce.map.java.opts` configuration properties, with the advantage that Oozie will append to these properties instead of
simply setting them (e.g. if you have one of these properties specified in mapred-site.xml, setting it again in
-the =configuration= element will override it, but using =java-opts= or =java-opt= will instead append to it, preserving the original
-value). You can have either one =java-opts=, multiple =java-opt=, or neither; you cannot use both at the same time. In any case,
-Oozie will set both =mapred.child.java.opts= and =mapreduce.map.java.opts= to the same value based on combining them. In other
+the `configuration` element will override it, but using `java-opts` or `java-opt` will instead append to it, preserving the original
+value). You can have either one `java-opts`, multiple `java-opt`, or neither; you cannot use both at the same time. In any case,
+Oozie will set both `mapred.child.java.opts` and `mapreduce.map.java.opts` to the same value based on combining them. In other
words, after Oozie is finished:
-<verbatim>
+
+```
mapred.child.java.opts <-- "<mapred.child.java.opts> <mapreduce.map.java.opts> <java-opt...|java-opts>"
mapreduce.map.java.opts <-- "<mapred.child.java.opts> <mapreduce.map.java.opts> <java-opt...|java-opts>"
-</verbatim>
-In the case that parameters are repeated, the latest instance of the parameter is used by Java. This means that =java-opt=
-(or =java-opts=) has the highest priority, followed by =mapreduce.map.java.opts=, and finally =mapred.child.java.opts=. When
-multiple =java-opt= are specified, they are included from top to bottom ordering, where the bottom has highest priority.
+```
+In the case that parameters are repeated, the latest instance of the parameter is used by Java. This means that `java-opt`
+(or `java-opts`) has the highest priority, followed by `mapreduce.map.java.opts`, and finally `mapred.child.java.opts`. When
+multiple `java-opt` are specified, they are included from top to bottom ordering, where the bottom has highest priority.
-The =arg= elements, if present, contains arguments for the main function. The value of each =arg= element is considered
-a single argument and they are passed to the =main= method in the same order.
+The `arg` elements, if present, contains arguments for the main function. The value of each `arg` element is considered
+a single argument and they are passed to the `main` method in the same order.
All the above elements can be parameterized (templatized) using EL expressions.
-*Example:*
+**Example:**
+
-<verbatim>
+```
<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:1.0">
...
<action name="myfirstjavajob">
@@ -1574,43 +1634,45 @@ All the above elements can be parameterized (templatized) using EL expressions.
</action>
...
</workflow-app>
-</verbatim>
+```
----+++++ 3.2.6.1 Overriding an action's Main class
+##### 3.2.6.1 Overriding an action's Main class
This feature is useful for developers to change the Main classes without having to recompile or redeploy Oozie.
-For most actions (not just the Java action), you can override the Main class it uses by specifying the following =configuration=
+For most actions (not just the Java action), you can override the Main class it uses by specifying the following `configuration`
property and making sure that your class is included in the workflow's classpath. If you specify this in the Java action,
the main-class element has priority.
-<verbatim>
+
+```
<property>
<name>oozie.launcher.action.main.class</name>
<value>org.my.CustomMain</value>
</property>
-</verbatim>
+```
-*Note:* Most actions typically pass information to their corresponding Main in specific ways; you should look at the action's
+**Note:** Most actions typically pass information to their corresponding Main in specific ways; you should look at the action's
existing Main to see how it works before creating your own. In fact, its probably simplest to just subclass the existing Main and
add/modify/overwrite any behavior you want to change.
-#GitAction
----++++ 3.2.7 Git action
+<a name="GitAction"></a>
+#### 3.2.7 Git action
-The =git= action allows one to clone a Git repository into HDFS. The supported options are =git-uri=, =branch=, =key-path=
-and =destination-uri=.
+The `git` action allows one to clone a Git repository into HDFS. The supported options are `git-uri`, `branch`, `key-path`
+and `destination-uri`.
-The =git clone= action is executed asynchronously by one of the YARN containers assigned to run on the cluster. If an SSH key is
+The `git clone` action is executed asynchronously by one of the YARN containers assigned to run on the cluster. If an SSH key is
specified it will be created on the file system in a YARN container's local directory, relying on YARN NodeManager to remove the
file after the action has run.
-Path names specified in the =git= action should be able to be parameterized (templatized) using EL expressions,
-e.g. =${wf:user()}= . Path name should be specified as an absolute path. Each file path must specify the file system URI.
+Path names specified in the `git` action should be able to be parameterized (templatized) using EL expressions,
+e.g. `${wf:user()}` . Path name should be specified as an absolute path. Each file path must specify the file system URI.
+
+**Syntax:**
-*Syntax:*
-<verbatim>
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<action name="[NODE-NAME]">
@@ -1628,11 +1690,12 @@ e.g. =${wf:user()}= . Path name should be specified as an absolute path. Each fi
</action>
...
</workflow-app>
-</verbatim>
+```
-*Example:*
+**Example:**
-<verbatim>
+
+```
<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1">
...
<action name="clone_oozie">
@@ -1645,20 +1708,21 @@ e.g. =${wf:user()}= . Path name should be specified as an absolute path. Each fi
</action>
...
</workflow-app>
-</verbatim>
+```
-In the above example, a Git repository on e.g. GitHub.com is cloned to the HDFS directory =my_git_repo_directory= which should not
+In the above example, a Git repository on e.g. GitHub.com is cloned to the HDFS directory `my_git_repo_directory` which should not
exist previously on the filesystem. Note that repository addresses outside of GitHub.com but accessible to the YARN container
running the Git action may also be used.
-If a =name-node= element is specified, then it is not necessary for any of the paths to start with the file system URI as it is
-taken from the =name-node= element.
+If a `name-node` element is specified, then it is not necessary for any of the paths to start with the file system URI as it is
+taken from the `name-node` element.
-The =resource-manager= (Oozie 5.x) element has to be specified to name the YARN ResourceManager address.
+The `resource-manager` (Oozie 5.x) element has to be specified to name the YARN ResourceManager address.
If any of the paths need to be served from another HDFS namenode, its address has to be part of
that filesystem URI prefix:
-<verbatim>
+
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<action name="[NODE-NAME]">
@@ -1673,23 +1737,24 @@ that filesystem URI prefix:
</action>
...
</workflow-app>
-</verbatim>
+```
This is also true if the name-node is specified in the global section (see
-[[WorkflowFunctionalSpec#GlobalConfigurations][Global Configurations]]).
+[Global Configurations](WorkflowFunctionalSpec.html#GlobalConfigurations)).
-Be aware that =key-path= might point to a secure object store location other than the current =fs.defaultFS=. In that case,
+Be aware that `key-path` might point to a secure object store location other than the current `fs.defaultFS`. In that case,
appropriate file permissions are still necessary (readable by submitting user), credentials provided, etc.
-As of workflow schema 1.0, zero or more =job-xml= elements can be specified; these must refer to Hadoop JobConf =job.xml= formatted
-files bundled in the workflow application. They can be used to set additional properties for the =FileSystem= instance.
+As of workflow schema 1.0, zero or more `job-xml` elements can be specified; these must refer to Hadoop JobConf `job.xml` formatted
+files bundled in the workflow application. They can be used to set additional properties for the `FileSystem` instance.
-As of schema workflow schema 1.0, if a =configuration= element is specified, then it will also be used to set additional =JobConf=
-properties for the =FileSystem= instance. Properties specified in the =configuration= element are overridden by properties
-specified in the files specified by any =job-xml= elements.
+As of schema workflow schema 1.0, if a `configuration` element is specified, then it will also be used to set additional `JobConf`
+properties for the `FileSystem` instance. Properties specified in the `configuration` element are overridden by properties
+specified in the files specified by any `job-xml` elements.
-*Example:*
+**Example:**
+```
<workflow-app name="[WF-DEF-NAME]" xmlns="uri:oozie:workflow:1.0">
...
<action name="[NODE-NAME]">
@@ -1708,49 +1773,50 @@ specified in the files specified by any =job-xml= elements.
</action>
...
</workflow>
-</verbatim>
+```
-#WorkflowParameterization
----++ 4 Parameterization of Workflows
+<a name="WorkflowParameterization"></a>
+## 4 Parameterization of Workflows
Workflow definitions can be parameterized.
When workflow node is executed by Oozie all the ELs are resolved into concrete values.
The parameterization of workflow definitions it done using JSP Expression Language syntax from the
-[[http://jcp.org/aboutJava/communityprocess/final/jsr152/][JSP 2.0 Specification (JSP.2.3)]], allowing not only to
+[JSP 2.0 Specification (JSP.2.3)](http://jcp.org/aboutJava/communityprocess/final/jsr152/index.html), allowing not only to
support variables as parameters but also functions and complex expressions.
EL expressions can be used in the configuration values of action and decision nodes. They can be used in XML attribute
values and in XML element and attribute values.
They cannot be used in XML element and attribute names. They cannot be used in the name of a node and they cannot be
-used within the =transition= elements of a node.
+used within the `transition` elements of a node.
-#WorkflowProperties
----+++ 4.1 Workflow Job Properties (or Parameters)
+<a name="WorkflowProperties"></a>
+### 4.1 Workflow Job Properties (or Parameters)
When a workflow job is submitted to Oozie, the submitter may specify as many workflow job properties as required
(similar to Hadoop JobConf properties).
Workflow applications may define default values for the workflow job or action parameters. They must be defined in a
- =config-default.xml= file bundled with the workflow application archive (refer to section '7 Workflow
+ `config-default.xml` file bundled with the workflow application archive (refer to section '7 Workflow
Applications Packaging'). Job
<TRUNCATED>
[03/11] oozie git commit: OOZIE-2734 [docs] Switch from TWiki to
Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/ENG_Custom_Authentication.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/ENG_Custom_Authentication.twiki b/docs/src/site/twiki/ENG_Custom_Authentication.twiki
deleted file mode 100644
index 3b8202d..0000000
--- a/docs/src/site/twiki/ENG_Custom_Authentication.twiki
+++ /dev/null
@@ -1,79 +0,0 @@
-<noautolink>
-
-[[index][::Go back to Oozie Documentation Index::]]
-
----+!! Creating Custom Authentication
-
-%TOC%
-
----++ Hadoop-Auth Authentication Interfaces and classes
-
-1. =org.apache.hadoop.security.authentication.client.Authenticator:= Interface for client authentication mechanisms.
-
-The following authenticators are provided in hadoop-auth:
-
- * KerberosAuthenticator : the authenticator implements the Kerberos SPNEGO authentication sequence.
- * PseudoAuthenticator : the authenticator implementation provides an authentication equivalent to Hadoop's Simple
- authentication, it trusts the value of the 'user.name' Java System property.
-
-2. =org.apache.hadoop.security.authentication.server.AuthenticationHandler:= Interface for server authentication mechanisms.
-
- * KerberosAuthenticationHandler : the authenticator handler implements the Kerberos SPNEGO authentication mechanism for HTTP.
- * PseudoAuthenticationHandler : the authenticator handler provides a pseudo authentication mechanism that accepts the user
- name specified as a query string parameter.
-
-3. =org.apache.hadoop.security.authentication.server.AuthenticationFilter:= A servlet filter enables protecting web application
-resources with different authentication mechanisms provided by AuthenticationHandler. To enable the filter, web application
-resources file (ex. web.xml) needs to include a filter class derived from =AuthenticationFilter=.
-
-For more information have a look at the appropriate
-[[https://hadoop.apache.org/docs/r2.7.2/hadoop-auth/index.html][Hadoop documentation]].
-
----++ Provide Custom Authentication to Oozie Client
-
-Apache Oozie contains a default class =org.apache.oozie.client.AuthOozieClient= to support Kerberos HTTP SPNEGO authentication,
-pseudo/simple authentication and anonymous access for client connections.
-
-To provide other authentication mechanisms, an Oozie client should extend from =AuthOozieClient= and provide the following
-methods should be overridden by derived classes to provide custom authentication:
-
- * getAuthenticator() : return corresponding Authenticator based on value specified by user at =auth= command option.
- * createConnection() : create a singleton class at Authenticator to allow client set and get key-value configuration for
- authentication.
-
----++ Provide Custom Authentication to Oozie Server
-
-To accept custom authentication in Oozie server, a filter extends from AuthenticationFilter must be provided. This filter
-delegates to the configured authentication handler for authentication and once it obtains an =AuthenticationToken= from it, sets
-a signed HTTP cookie with the token. If HTTP cookie is provided with different key name, its cookie value can be retrieved by
-overriding =getToken()= method. Please note, only when =getToken()= return NULL, a custom authentication can be invoked and
-processed in =AuthenticationFilter.doFilter()=.
-
-The following method explains how to read it and return NULL token.
-<verbatim>
-protected AuthenticationToken getToken(HttpServletRequest request) throws IOException, AuthenticationException {
- String tokenStr = null;
- Cookie[] cookies = request.getCookies();
-
- if (cookies != null) {
- for (Cookie cookie : cookies) {
- if (cookie.getName().equals(AuthenticatedURL.AUTH_COOKIE)) {
- tokenStr = cookie.getValue();
- LOG.info("Got 'hadoop.auth' cookie from request = " + tokenStr);
- if (tokenStr != null && !tokenStr.trim().isEmpty()) {
- AuthenticationToken retToken = super.getToken(request);
- return retToken;
- }
- } else if (cookie.getName().equals("NEWAUTH")) {
- tokenStr = cookie.getValue();
- // DO NOT return the token string so request can authenticated.
- }
- }
- }
- return null;
- }
-</verbatim>
-
-[[index][::Go back to Oozie Documentation Index::]]
-
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/ENG_MiniOozie.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/ENG_MiniOozie.twiki b/docs/src/site/twiki/ENG_MiniOozie.twiki
index 0b16289..e793676 100644
--- a/docs/src/site/twiki/ENG_MiniOozie.twiki
+++ b/docs/src/site/twiki/ENG_MiniOozie.twiki
@@ -1,43 +1,46 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
----+!! Running MiniOozie Tests
+[::Go back to Oozie Documentation Index::](index.html)
-%TOC%
+# Running MiniOozie Tests
----++ System Requirements
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## System Requirements
* Unix box (tested on Mac OS X and Linux)
* Java JDK 1.8+
* Eclipse (tested on 3.5 and 3.6)
- * [[http://maven.apache.org/][Maven 3.0.1+]]
+ * [Maven 3.0.1+](http://maven.apache.org/)
The Maven command (mvn) must be in the command path.
----++ Installing Oozie Jars To Maven Cache
+## Installing Oozie Jars To Maven Cache
Oozie source tree is at Apache SVN or Apache GIT. MiniOozie sample project is under Oozie source tree.
The following command downloads Oozie trunk to local:
-<verbatim>
+
+```
$ svn co https://svn.apache.org/repos/asf/incubator/oozie/trunk
-</verbatim>
+```
OR
-<verbatim>
+
+```
$ git clone git://github.com/apache/oozie.git
-</verbatim>
+```
To run MiniOozie tests, the required jars like oozie-core, oozie-client, oozie-core-tests need to be
available in remote maven repositories or local maven repository. The local maven cache for the above
jars can be created and installed using the command:
-<verbatim>
+
+```
$ mvn clean install -DskipTests -DtestJarSimple
-</verbatim>
+```
The following properties should be specified to install correct jars for MiniOozie:
@@ -47,33 +50,34 @@ The following properties should be specified to install correct jars for MiniOoz
MiniOozie is a folder named 'minitest' under Oozie source tree. Two sample tests are included in the project.
The following command to execute tests under MiniOozie:
-<verbatim>
+
+```
$ cd minitest
$ mvn clean test
-</verbatim>
+```
----++ Create Tests Using MiniOozie
+## Create Tests Using MiniOozie
MiniOozie is a JUnit test class to test Oozie applications such as workflow and coordinator. The test case
needs to extend from MiniOozieTestCase and does the same as the example class 'WorkflowTest.java' to create Oozie
workflow application properties and workflow XML. The example file is under Oozie source tree:
- * =minitest/src/test/java/org/apache/oozie/test/WorkflowTest.java=
+ * `minitest/src/test/java/org/apache/oozie/test/WorkflowTest.java`
----++ IDE Setup
+## IDE Setup
Eclipse and IntelliJ can use directly MiniOozie Maven project files. MiniOozie project can be imported to
Eclipse and IntelliJ as independent project.
The test directories under MiniOozie are:
- * =minitest/src/test/java= : as test-source directory
- * =minitest/src/test/resources= : as test-resource directory
+ * `minitest/src/test/java` : as test-source directory
+ * `minitest/src/test/resources` : as test-resource directory
+
+Also asynchronous actions like FS action can be used / tested using `LocalOozie` / `OozieClient` API.
+Please see `fs-decision.xml` workflow example.
-Also asynchronous actions like FS action can be used / tested using =LocalOozie= / =OozieClient= API.
-Please see =fs-decision.xml= workflow example.
+[::Go back to Oozie Documentation Index::](index.html)
-[[index][::Go back to Oozie Documentation Index::]]
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/WebServicesAPI.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/WebServicesAPI.twiki b/docs/src/site/twiki/WebServicesAPI.twiki
index f9008a6..a303802 100644
--- a/docs/src/site/twiki/WebServicesAPI.twiki
+++ b/docs/src/site/twiki/WebServicesAPI.twiki
@@ -1,34 +1,34 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
+
+[::Go back to Oozie Documentation Index::](index.html)
-----
-%TOC%
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
----++ Oozie Web Services API, V1 (Workflow, Coordinator, And Bundle)
+## Oozie Web Services API, V1 (Workflow, Coordinator, And Bundle)
The Oozie Web Services API is a HTTP REST JSON API.
-All responses are in =UTF-8=.
+All responses are in `UTF-8`.
-Assuming Oozie is running at =OOZIE_URL=, the following web services end points are supported:
+Assuming Oozie is running at `OOZIE_URL`, the following web services end points are supported:
- * <OOZIE_URL>/versions
- * <OOZIE_URL>/v1/admin
- * <OOZIE_URL>/v1/job
- * <OOZIE_URL>/v1/jobs
- * <OOZIE_URL>/v2/job
- * <OOZIE_URL>/v2/jobs
- * <OOZIE_URL>/v2/admin
- * <OOZIE_URL>/v2/sla
+ * \<OOZIE_URL\>/versions
+ * \<OOZIE_URL\>/v1/admin
+ * \<OOZIE_URL\>/v1/job
+ * \<OOZIE_URL\>/v1/jobs
+ * \<OOZIE_URL\>/v2/job
+ * \<OOZIE_URL\>/v2/jobs
+ * \<OOZIE_URL\>/v2/admin
+ * \<OOZIE_URL\>/v2/sla
Documentation on the API is below; in some cases, looking at the corresponding command in the
-[[DG_CommandLineTool][Command Line Documentation]] page will provide additional details and examples. Most of the functionality
-offered by the Oozie CLI is using the WS API. If you export <code>OOZIE_DEBUG</code> then the Oozie CLI will output the WS API
+[Command Line Documentation](DG_CommandLineTool.html) page will provide additional details and examples. Most of the functionality
+offered by the Oozie CLI is using the WS API. If you export `OOZIE_DEBUG` then the Oozie CLI will output the WS API
details used by any commands you execute. This is useful for debugging purposes to or see how the Oozie CLI works with the WS API.
----+++ Versions End-Point
+### Versions End-Point
_Identical to the corresponding Oozie v0 WS API_
@@ -38,79 +38,87 @@ It support only HTTP GET request and not sub-resources.
It returns the supported Oozie protocol versions by the server.
-Current returned values are =0, 1, 2=.
+Current returned values are `0, 1, 2`.
+
+**Request:**
-*Request:*
-<verbatim>
+```
GET /oozie/versions
-</verbatim>
+```
+
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
[0,1]
-</verbatim>
+```
----+++ Admin End-Point
+### Admin End-Point
This endpoint is for obtaining Oozie system status and configuration information.
-It supports the following sub-resources: =status, os-env, sys-props, configuration, instrumentation, systems, available-timezones=.
+It supports the following sub-resources: `status, os-env, sys-props, configuration, instrumentation, systems, available-timezones`.
----++++ System Status
+#### System Status
_Identical to the corresponding Oozie v0 WS API_
A HTTP GET request returns the system status.
-*Request:*
+**Request:**
-<verbatim>
+
+```
GET /oozie/v1/admin/status
-</verbatim>
+```
+
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
{"systemMode":NORMAL}
-</verbatim>
+```
-With a HTTP PUT request it is possible to change the system status between =NORMAL=, =NOWEBSERVICE=, and =SAFEMODE=.
+With a HTTP PUT request it is possible to change the system status between `NORMAL`, `NOWEBSERVICE`, and `SAFEMODE`.
-*Request:*
+**Request:**
-<verbatim>
+
+```
PUT /oozie/v1/admin/status?systemmode=SAFEMODE
-</verbatim>
+```
+
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 200 OK
-</verbatim>
+```
----++++ OS Environment
+#### OS Environment
_Identical to the corresponding Oozie v0 WS API_
A HTTP GET request returns the Oozie system OS environment.
-*Request:*
+**Request:**
-<verbatim>
+
+```
GET /oozie/v1/admin/os-env
-</verbatim>
+```
+
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
@@ -127,23 +135,25 @@ Content-Type: application/json;charset=UTF-8
LANG: "en_US.UTF-8",
...
}
-</verbatim>
+```
----++++ Java System Properties
+#### Java System Properties
_Identical to the corresponding Oozie v0 WS API_
A HTTP GET request returns the Oozie Java system properties.
-*Request:*
+**Request:**
-<verbatim>
+
+```
GET /oozie/v1/admin/java-sys-properties
-</verbatim>
+```
+
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
@@ -154,23 +164,25 @@ Content-Type: application/json;charset=UTF-8
java.vm.info: "mixed mode",
...
}
-</verbatim>
+```
----++++ Oozie Configuration
+#### Oozie Configuration
_Identical to the corresponding Oozie v0 WS API_
A HTTP GET request returns the Oozie system configuration.
-*Request:*
+**Request:**
-<verbatim>
+
+```
GET /oozie/v1/admin/configuration
-</verbatim>
+```
+
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
@@ -185,9 +197,9 @@ Content-Type: application/json;charset=UTF-8
oozie.service.DBLiteWorkflowStoreService.oozie.autoinstall: "true",
...
}
-</verbatim>
+```
----++++ Oozie Instrumentation
+#### Oozie Instrumentation
_Identical to the corresponding Oozie v0 WS API_
@@ -196,17 +208,19 @@ Deprecated and by default disabled since 5.0.0.
A HTTP GET request returns the Oozie instrumentation information. Keep in mind that timers and counters that the Oozie server
hasn't incremented yet will not show up.
-*Note:* If Instrumentation is enabled, then Metrics is unavailable.
+**Note:** If Instrumentation is enabled, then Metrics is unavailable.
-*Request:*
+**Request:**
-<verbatim>
+
+```
GET /oozie/v1/admin/instrumentation
-</verbatim>
+```
+
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
@@ -273,9 +287,9 @@ Content-Type: application/json;charset=UTF-8
...
]
}
-</verbatim>
+```
----++++ Oozie Metrics
+#### Oozie Metrics
_Available in the Oozie v2 WS API and later_
@@ -283,19 +297,21 @@ A HTTP GET request returns the Oozie metrics information. Keep in mind that tim
hasn't incremented yet will not show up.
-*Note:* If Metrics is enabled, then Instrumentation is unavailable.
+**Note:** If Metrics is enabled, then Instrumentation is unavailable.
-*Note:* by default enabled since 5.0.0.
+**Note:** by default enabled since 5.0.0.
-*Request:*
+**Request:**
-<verbatim>
+
+```
GET /oozie/v2/admin/metrics
-</verbatim>
+```
+
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
@@ -360,42 +376,46 @@ Content-Type: application/json;charset=UTF-8
...
}
}
-</verbatim>
+```
----++++ Version
+#### Version
_Identical to the corresponding Oozie v0 WS API_
A HTTP GET request returns the Oozie build version.
-*Request:*
+**Request:**
-<verbatim>
+
+```
GET /oozie/v1/admin/build-version
-</verbatim>
+```
+
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
{buildVersion: "3.0.0-SNAPSHOT" }
-</verbatim>
+```
----++++ Available Time Zones
+#### Available Time Zones
A HTTP GET request returns the available time zones.
-*Request:*
+**Request:**
+
-<verbatim>
+```
GET /oozie/v1/admin/available-timezones
-</verbatim>
+```
-*Response:*
+**Response:**
-<verbatim>
+
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
@@ -436,33 +456,36 @@ Content-Type: application/json;charset=UTF-8
...
]
}
-</verbatim>
+```
----++++ Queue Dump
+#### Queue Dump
A HTTP GET request returns the queue dump of the Oozie system. This is an administrator debugging feature.
-*Request:*
+**Request:**
+
-<verbatim>
+```
GET /oozie/v1/admin/queue-dump
-</verbatim>
+```
----++++ Available Oozie Servers
+#### Available Oozie Servers
A HTTP GET request returns the list of available Oozie Servers. This is useful when Oozie is configured
-for [[AG_Install#HA][High Availability]]; if not, it will simply return the one Oozie Server.
+for [High Availability](AG_Install.html#HA); if not, it will simply return the one Oozie Server.
-*Request:*
+**Request:**
-<verbatim>
+
+```
GET /oozie/v2/admin/available-oozie-servers
-</verbatim>
+```
+
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
@@ -471,21 +494,23 @@ Content-Type: application/json;charset=UTF-8
"hostB": "http://hostB:11000/oozie",
"hostC": "http://hostC:11000/oozie",
}
-</verbatim>
+```
----++++ List available sharelib
+#### List available sharelib
A HTTP GET request to get list of available sharelib.
If the name of the sharelib is passed as an argument (regex supported) then all corresponding files are also listed.
-*Request:*
+**Request:**
-<verbatim>
+
+```
GET /oozie/v2/admin/list_sharelib
-</verbatim>
+```
+
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
{
@@ -500,17 +525,19 @@ Content-Type: application/json;charset=UTF-8
"pig"
]
}
-</verbatim>
+```
-*Request:*
+**Request:**
-<verbatim>
+
+```
GET /oozie/v2/admin/list_sharelib?lib=pig*
-</verbatim>
+```
+
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
@@ -529,24 +556,26 @@ Content-Type: application/json;charset=UTF-8
}
]
}
-</verbatim>
+```
----++++ Update system sharelib
+#### Update system sharelib
This webservice call makes the oozie server(s) to pick up the latest version of sharelib present
under oozie.service.WorkflowAppService.system.libpath directory based on the sharelib directory timestamp or reloads
the sharelib metafile if one is configured. The main purpose is to update the sharelib on the oozie server without restarting.
-*Request:*
+**Request:**
-<verbatim>
+
+```
GET /oozie/v2/admin/update_sharelib
-</verbatim>
+```
+
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
[
@@ -573,59 +602,62 @@ Content-Type: application/json;charset=UTF-8
}
}
]
-</verbatim>
+```
----++++ Purge Command
+#### Purge Command
Oozie admin purge command cleans up the Oozie Workflow/Coordinator/Bundle records based on the parameters.
The unit for parameters is day.
Purge command will delete the workflow records (wf=30) older than 30 days, coordinator records (coord=7) older than 7 days and
bundle records (bundle=7) older than 7 days. The limit (limit=10) defines, number of records to be fetch at a time. Turn
-(oldCoordAction=true/false) =on/off= coordinator action record purging for long running coordinators. If any of the parameter is
-not provided, then it will be taken from the =oozie-default/oozie-site= configuration.
+(oldCoordAction`true/false) `on/off= coordinator action record purging for long running coordinators. If any of the parameter is
+not provided, then it will be taken from the `oozie-default/oozie-site` configuration.
-*Request:*
+**Request:**
-<verbatim>
+
+```
GET /oozie/v2/admin/purge?wf=30&coord=7&bundle=7&limit=10&oldCoordAction=true
-</verbatim>
+```
+
+**Response:**
-*Response:*
-<verbatim>
+```
{
"purge": "Purge command executed successfully"
}
-</verbatim>
+```
----+++ Job and Jobs End-Points
+### Job and Jobs End-Points
_Modified in Oozie v1 WS API_
These endpoints are for submitting, managing and retrieving information of workflow, coordinator, and bundle jobs.
----++++ Job Submission
+#### Job Submission
----++++ Standard Job Submission
+#### Standard Job Submission
An HTTP POST request with an XML configuration as payload creates a job.
The type of job is determined by the presence of one of the following 3 properties:
- * =oozie.wf.application.path= : path to a workflow application directory, creates a workflow job
- * =oozie.coord.application.path= : path to a coordinator application file, creates a coordinator job
- * =oozie.bundle.application.path= : path to a bundle application file, creates a bundle job
-
+ * `oozie.wf.application.path` : path to a workflow application directory, creates a workflow job
+ * `oozie.coord.application.path` : path to a coordinator application file, creates a coordinator job
+ * `oozie.bundle.application.path` : path to a bundle application file, creates a bundle job
+
Or, if none of those are present, the jobtype parameter determines the type of job to run. It can either be mapreduce or pig.
-*Request:*
+**Request:**
+
-<verbatim>
+```
POST /oozie/v1/jobs
Content-Type: application/xml;charset=UTF-8
.
@@ -641,47 +673,50 @@ Content-Type: application/xml;charset=UTF-8
</property>
...
</configuration>
-</verbatim>
+```
-*Response:*
+**Response:**
-<verbatim>
+
+```
HTTP/1.1 201 CREATED
Content-Type: application/json;charset=UTF-8
.
{
id: "job-3"
}
-</verbatim>
+```
-A created job will be in =PREP= status. If the query string parameter 'action=start' is provided in
-the POST URL, the job will be started immediately and its status will be =RUNNING=.
+A created job will be in `PREP` status. If the query string parameter 'action=start' is provided in
+the POST URL, the job will be started immediately and its status will be `RUNNING`.
-Coordinator jobs with start time in the future they will not create any action until the start time
+Coordinator jobs with start time in the future they will not create any action until the start time
happens.
-A coordinator job will remain in =PREP= status until it's triggered, in which case it will change to =RUNNING= status.
+A coordinator job will remain in `PREP` status until it's triggered, in which case it will change to `RUNNING` status.
The 'action=start' parameter is not valid for coordinator jobs.
----++++ Proxy MapReduce Job Submission
+#### Proxy MapReduce Job Submission
You can submit a Workflow that contains a single MapReduce action without writing a workflow.xml. Any required Jars or other files
must already exist in HDFS.
The following properties are required; any additional parameters needed by the MapReduce job can also be specified here:
- * =fs.default.name=: The NameNode
- * =mapred.job.tracker=: The JobTracker
- * =mapred.mapper.class=: The map-task classname
- * =mapred.reducer.class=: The reducer-task classname
- * =mapred.input.dir=: The map-task input directory
- * =mapred.output.dir=: The reduce-task output directory
- * =user.name=: The username of the user submitting the job
- * =oozie.libpath=: A directory in HDFS that contains necessary Jars for your job
- * =oozie.proxysubmission=: Must be set to =true=
-
-*Request:*
-
-<verbatim>
+
+ * `fs.default.name`: The NameNode
+ * `mapred.job.tracker`: The JobTracker
+ * `mapred.mapper.class`: The map-task classname
+ * `mapred.reducer.class`: The reducer-task classname
+ * `mapred.input.dir`: The map-task input directory
+ * `mapred.output.dir`: The reduce-task output directory
+ * `user.name`: The username of the user submitting the job
+ * `oozie.libpath`: A directory in HDFS that contains necessary Jars for your job
+ * `oozie.proxysubmission`: Must be set to `true`
+
+**Request:**
+
+
+```
POST /oozie/v1/jobs?jobtype=mapreduce
Content-Type: application/xml;charset=UTF-8
.
@@ -724,52 +759,58 @@ Content-Type: application/xml;charset=UTF-8
<value>true</value>
</property>
</configuration>
-</verbatim>
+```
-*Response:*
+**Response:**
-<verbatim>
+
+```
HTTP/1.1 201 CREATED
Content-Type: application/json;charset=UTF-8
.
{
id: "job-3"
}
-</verbatim>
+```
----++++ Proxy Pig Job Submission
+#### Proxy Pig Job Submission
You can submit a Workflow that contains a single Pig action without writing a workflow.xml. Any required Jars or other files must
already exist in HDFS.
The following properties are required:
- * =fs.default.name=: The NameNode
- * =mapred.job.tracker=: The JobTracker
- * =user.name=: The username of the user submitting the job
- * =oozie.pig.script=: Contains the pig script you want to run (the actual script, not a file path)
- * =oozie.libpath=: A directory in HDFS that contains necessary Jars for your job
- * =oozie.proxysubmission=: Must be set to =true=
+
+ * `fs.default.name`: The NameNode
+ * `mapred.job.tracker`: The JobTracker
+ * `user.name`: The username of the user submitting the job
+ * `oozie.pig.script`: Contains the pig script you want to run (the actual script, not a file path)
+ * `oozie.libpath`: A directory in HDFS that contains necessary Jars for your job
+ * `oozie.proxysubmission`: Must be set to `true`
The following properties are optional:
- * =oozie.pig.script.params.size=: The number of parameters you'll be passing to Pig
- required =oozie.pig.script.params.n=: A parameter (variable definition for the script) in 'key=value' format, the 'n' should be an integer starting with 0 to indicate the parameter number
- * =oozie.pig.options.size=: The number of options you'll be passing to Pig
- * =oozie.pig.options.n=: An argument to pass to Pig, the 'n' should be an integer starting with 0 to indicate the option number
-The =oozie.pig.options.n= parameters are sent directly to Pig without any modification unless they start with =-D=, in which case
-they are put into the <code><configuration></code> element of the action.
+ * `oozie.pig.script.params.size`: The number of parameters you'll be passing to Pig
+ required
+ * `oozie.pig.script.params.n`: A parameter (variable definition for the script) in 'key=value' format, the 'n' should be an integer starting with 0 to indicate the parameter number
+ * `oozie.pig.options.size`: The number of options you'll be passing to Pig
+ * `oozie.pig.options.n`: An argument to pass to Pig, the 'n' should be an integer starting with 0 to indicate the option number
+
+The `oozie.pig.options.n` parameters are sent directly to Pig without any modification unless they start with `-D`, in which case
+they are put into the `<configuration>` element of the action.
+
+In addition to passing parameters to Pig with `oozie.pig.script.params.n`, you can also create a properties file on HDFS and
+reference it with the `-param_file` option in `oozie.pig.script.options.n`; both are shown in the following example.
-In addition to passing parameters to Pig with =oozie.pig.script.params.n=, you can also create a properties file on HDFS and
-reference it with the =-param_file= option in =oozie.pig.script.options.n=; both are shown in the following example.
-<verbatim>
+```
$ hadoop fs -cat /user/rkanter/pig_params.properties
INPUT=/user/rkanter/examples/input-data/text
-</verbatim>
+```
-*Request:*
+**Request:**
-<verbatim>
+
+```
POST /oozie/v1/jobs?jobtype=pig
Content-Type: application/xml;charset=UTF-8
.
@@ -824,44 +865,48 @@ Content-Type: application/xml;charset=UTF-8
<value>true</value>
</property>
</configuration>
-</verbatim>
+```
+
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 201 CREATED
Content-Type: application/json;charset=UTF-8
.
{
id: "job-3"
}
-</verbatim>
+```
----++++ Proxy Hive Job Submission
+#### Proxy Hive Job Submission
You can submit a Workflow that contains a single Hive action without writing a workflow.xml. Any required Jars or other files must
already exist in HDFS.
The following properties are required:
- * =fs.default.name=: The NameNode
- * =mapred.job.tracker=: The JobTracker
- * =user.name=: The username of the user submitting the job
- * =oozie.hive.script=: Contains the hive script you want to run (the actual script, not a file path)
- * =oozie.libpath=: A directory in HDFS that contains necessary Jars for your job
- * =oozie.proxysubmission=: Must be set to =true=
+
+ * `fs.default.name`: The NameNode
+ * `mapred.job.tracker`: The JobTracker
+ * `user.name`: The username of the user submitting the job
+ * `oozie.hive.script`: Contains the hive script you want to run (the actual script, not a file path)
+ * `oozie.libpath`: A directory in HDFS that contains necessary Jars for your job
+ * `oozie.proxysubmission`: Must be set to `true`
The following properties are optional:
- * =oozie.hive.script.params.size=: The number of parameters you'll be passing to Hive
- * =oozie.hive.script.params.n=: A parameter (variable definition for the script) in 'key=value' format, the 'n' should be an integer starting with 0 to indicate the parameter number
- * =oozie.hive.options.size=: The number of options you'll be passing to Hive
- * =oozie.hive.options.n=: An argument to pass to Hive, the 'n' should be an integer starting with 0 to indicate the option number
-The =oozie.hive.options.n= parameters are sent directly to Hive without any modification unless they start with =-D=, in which case
-they are put into the <code><configuration></code> element of the action.
+ * `oozie.hive.script.params.size`: The number of parameters you'll be passing to Hive
+ * `oozie.hive.script.params.n`: A parameter (variable definition for the script) in 'key=value' format, the 'n' should be an integer starting with 0 to indicate the parameter number
+ * `oozie.hive.options.size`: The number of options you'll be passing to Hive
+ * `oozie.hive.options.n`: An argument to pass to Hive, the 'n' should be an integer starting with 0 to indicate the option number
+
+The `oozie.hive.options.n` parameters are sent directly to Hive without any modification unless they start with `-D`, in which case
+they are put into the `<configuration>` element of the action.
-*Request:*
+**Request:**
-<verbatim>
+
+```
POST /oozie/v1/jobs?jobtype=hive
Content-Type: application/xml;charset=UTF-8
.
@@ -907,39 +952,43 @@ Content-Type: application/xml;charset=UTF-8
<value>true</value>
</property>
</configuration>
-</verbatim>
+```
+
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 201 CREATED
Content-Type: application/json;charset=UTF-8
.
{
id: "job-3"
}
-</verbatim>
+```
----++++ Proxy Sqoop Job Submission
+#### Proxy Sqoop Job Submission
You can submit a Workflow that contains a single Sqoop command without writing a workflow.xml. Any required Jars or other
files must already exist in HDFS.
The following properties are required:
- * =fs.default.name=: The NameNode
- * =mapred.job.tracker=: The JobTracker
- * =user.name=: The username of the user submitting the job
- * =oozie.sqoop.command=: The sqoop command you want to run where each argument occupies one line or separated by "\n"
- * =oozie.libpath=: A directory in HDFS that contains necessary Jars for your job
- * =oozie.proxysubmission=: Must be set to =true=
+
+ * `fs.default.name`: The NameNode
+ * `mapred.job.tracker`: The JobTracker
+ * `user.name`: The username of the user submitting the job
+ * `oozie.sqoop.command`: The sqoop command you want to run where each argument occupies one line or separated by "\n"
+ * `oozie.libpath`: A directory in HDFS that contains necessary Jars for your job
+ * `oozie.proxysubmission`: Must be set to `true`
The following properties are optional:
- * =oozie.sqoop.options.size=: The number of options you'll be passing to Sqoop Hadoop job
- * =oozie.sqoop.options.n=: An argument to pass to Sqoop hadoop job conf, the 'n' should be an integer starting with 0 to indicate the option number
-*Request:*
+ * `oozie.sqoop.options.size`: The number of options you'll be passing to Sqoop Hadoop job
+ * `oozie.sqoop.options.n`: An argument to pass to Sqoop hadoop job conf, the 'n' should be an integer starting with 0 to indicate the option number
+
+**Request:**
-<verbatim>
+
+```
POST /oozie/v1/jobs?jobtype=sqoop
Content-Type: application/xml;charset=UTF-8
.
@@ -981,52 +1030,56 @@ Content-Type: application/xml;charset=UTF-8
<value>true</value>
</property>
</configuration>
-</verbatim>
+```
+
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 201 CREATED
Content-Type: application/json;charset=UTF-8
.
{
id: "job-3"
}
-</verbatim>
+```
----++++ Managing a Job
+#### Managing a Job
A HTTP PUT request starts, suspends, resumes, kills, update or dryruns a job.
-*Request:*
+**Request:**
+
-<verbatim>
+```
PUT /oozie/v1/job/job-3?action=start
-</verbatim>
+```
-*Response:*
+**Response:**
-<verbatim>
+
+```
HTTP/1.1 200 OK
-</verbatim>
+```
Valid values for the 'action' parameter are 'start', 'suspend', 'resume', 'kill', 'dryrun', 'rerun', and 'change'.
Rerunning and changing a job require additional parameters, and are described below:
----+++++ Re-Running a Workflow Job
+##### Re-Running a Workflow Job
-A workflow job in =SUCCEEDED=, =KILLED= or =FAILED= status can be partially rerun specifying a list
+A workflow job in `SUCCEEDED`, `KILLED` or `FAILED` status can be partially rerun specifying a list
of workflow nodes to skip during the rerun. All the nodes in the skip list must have complete its
execution.
The rerun job will have the same job ID.
-A rerun request is done with a HTTP PUT request with a =rerun= action.
+A rerun request is done with a HTTP PUT request with a `rerun` action.
+
+**Request:**
-*Request:*
-<verbatim>
+```
PUT /oozie/v1/job/job-3?action=rerun
Content-Type: application/xml;charset=UTF-8
.
@@ -1046,140 +1099,153 @@ Content-Type: application/xml;charset=UTF-8
</property>
...
</configuration>
-</verbatim>
+```
-*Response:*
+**Response:**
-<verbatim>
+
+```
HTTP/1.1 200 OK
-</verbatim>
+```
----+++++ Re-Running a coordinator job
+##### Re-Running a coordinator job
-A coordinator job in =RUNNING= =SUCCEEDED=, =KILLED= or =FAILED= status can be partially rerun by specifying the coordinator actions
+A coordinator job in `RUNNING` `SUCCEEDED`, `KILLED` or `FAILED` status can be partially rerun by specifying the coordinator actions
to re-execute.
-A rerun request is done with an HTTP PUT request with a =coord-rerun= =action=.
+A rerun request is done with an HTTP PUT request with a `coord-rerun` `action`.
+
+The `type` of the rerun can be `date` or `action`.
-The =type= of the rerun can be =date= or =action=.
+The `scope` of the rerun depends on the type:
+* `date`: a comma-separated list of date ranges. Each date range element is specified with dates separated by `::`
+* `action`: a comma-separated list of action ranges. Each action range is specified with two action numbers separated by `-`
-The =scope= of the rerun depends on the type:
-* =date=: a comma-separated list of date ranges. Each date range element is specified with dates separated by =::=
-* =action=: a comma-separated list of action ranges. Each action range is specified with two action numbers separated by =-=
+The `refresh` parameter can be `true` or `false` to specify if the user wants to refresh an action's input and output events.
-The =refresh= parameter can be =true= or =false= to specify if the user wants to refresh an action's input and output events.
+The `nocleanup` parameter can be `true` or `false` to specify is the user wants to cleanup output events for the rerun actions.
-The =nocleanup= parameter can be =true= or =false= to specify is the user wants to cleanup output events for the rerun actions.
+**Request:**
-*Request:*
-<verbatim>
+```
PUT /oozie/v1/job/job-3?action=coord-rerun&type=action&scope=1-2&refresh=false&nocleanup=false
.
-</verbatim>
+```
+
+or
-or
-<verbatim>
+```
PUT /oozie/v1/job/job-3?action=coord-rerun&type=date2009-02-01T00:10Z::2009-03-01T00:10Z&scope=&refresh=false&nocleanup=false
.
-</verbatim>
+```
-*Response:*
+**Response:**
-<verbatim>
+
+```
HTTP/1.1 200 OK
-</verbatim>
+```
----+++++ Re-Running a bundle job
+##### Re-Running a bundle job
-A coordinator job in =RUNNING= =SUCCEEDED=, =KILLED= or =FAILED= status can be partially rerun by specifying the coordinators to
+A coordinator job in `RUNNING` `SUCCEEDED`, `KILLED` or `FAILED` status can be partially rerun by specifying the coordinators to
re-execute.
-A rerun request is done with an HTTP PUT request with a =bundle-rerun= =action=.
+A rerun request is done with an HTTP PUT request with a `bundle-rerun` `action`.
+
+A comma separated list of coordinator job names (not IDs) can be specified in the `coord-scope` parameter.
-A comma separated list of coordinator job names (not IDs) can be specified in the =coord-scope= parameter.
+The `date-scope` parameter is a comma-separated list of date ranges. Each date range element is specified with dates separated
+by `::`. If empty or not included, Oozie will figure this out for you
-The =date-scope= parameter is a comma-separated list of date ranges. Each date range element is specified with dates separated
-by =::=. If empty or not included, Oozie will figure this out for you
+The `refresh` parameter can be `true` or `false` to specify if the user wants to refresh the coordinator's input and output events.
-The =refresh= parameter can be =true= or =false= to specify if the user wants to refresh the coordinator's input and output events.
+The `nocleanup` parameter can be `true` or `false` to specify is the user wants to cleanup output events for the rerun coordinators.
-The =nocleanup= parameter can be =true= or =false= to specify is the user wants to cleanup output events for the rerun coordinators.
+**Request:**
-*Request:*
-<verbatim>
+```
PUT /oozie/v1/job/job-3?action=bundle-rerun&coord-scope=coord-1&refresh=false&nocleanup=false
.
-</verbatim>
+```
-*Response:*
+**Response:**
-<verbatim>
+
+```
HTTP/1.1 200 OK
-</verbatim>
+```
----+++++ Changing endtime/concurrency/pausetime of a Coordinator Job
+##### Changing endtime/concurrency/pausetime of a Coordinator Job
-A coordinator job not in =KILLED= status can have it's endtime, concurrency, or pausetime changed.
+A coordinator job not in `KILLED` status can have it's endtime, concurrency, or pausetime changed.
-A change request is done with an HTTP PUT request with a =change= =action=.
+A change request is done with an HTTP PUT request with a `change` `action`.
-The =value= parameter can contain any of the following:
+The `value` parameter can contain any of the following:
* endtime: the end time of the coordinator job.
* concurrency: the concurrency of the coordinator job.
* pausetime: the pause time of the coordinator job.
-Multiple arguments can be passed to the =value= parameter by separating them with a ';' character.
+Multiple arguments can be passed to the `value` parameter by separating them with a ';' character.
If an already-succeeded job changes its end time, its status will become running.
-*Request:*
+**Request:**
+
-<verbatim>
+```
PUT /oozie/v1/job/job-3?action=change&value=endtime=2011-12-01T05:00Z
.
-</verbatim>
+```
or
-<verbatim>
+
+```
PUT /oozie/v1/job/job-3?action=change&value=concurrency=100
.
-</verbatim>
+```
or
-<verbatim>
+
+```
PUT /oozie/v1/job/job-3?action=change&value=pausetime=2011-12-01T05:00Z
.
-</verbatim>
+```
or
-<verbatim>
+
+```
PUT /oozie/v1/job/job-3?action=change&value=endtime=2011-12-01T05:00Z;concurrency=100;pausetime=2011-12-01T05:00Z
.
-</verbatim>
+```
+
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 200 OK
-</verbatim>
+```
----+++++ Updating coordinator definition and properties
-Existing coordinator definition and properties will be replaced by new definition and properties. Refer [[DG_CommandLineTool#Updating_coordinator_definition_and_properties][Updating coordinator definition and properties]]
+##### Updating coordinator definition and properties
+Existing coordinator definition and properties will be replaced by new definition and properties. Refer [Updating coordinator definition and properties](DG_CommandLineTool.html#Updating_coordinator_definition_and_properties)
-<verbatim>
+
+```
PUT oozie/v2/job/0000000-140414102048137-oozie-puru-C?action=update
-</verbatim>
+```
+
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
{"update":
@@ -1197,26 +1263,28 @@ Content-Type: application/json;charset=UTF-8
<name>queueName<\/name>\r\n******************************************\n"
}
}
-</verbatim>
+```
----++++ Job Information
+#### Job Information
A HTTP GET request retrieves the job information.
-*Request:*
+**Request:**
+
-<verbatim>
+```
GET /oozie/v1/job/job-3?show=info&timezone=GMT
-</verbatim>
+```
-*Response for a workflow job:*
+**Response for a workflow job:**
-<verbatim>
+
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
-{
+{
id: "0-200905191240-oozie-W",
appName: "indexer-workflow",
appPath: "hdfs://user/bansalm/indexer.wf",
@@ -1250,11 +1318,12 @@ Content-Type: application/json;charset=UTF-8
...
]
}
-</verbatim>
+```
-*Response for a coordinator job:*
+**Response for a coordinator job:**
-<verbatim>
+
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
@@ -1282,11 +1351,12 @@ Content-Type: application/json;charset=UTF-8
nominalTime: "Fri, 01 Jan 2010 01:00:00 GMT",
...
}
-</verbatim>
+```
+
+**Response for a bundle job:**
-*Response for a bundle job:*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
@@ -1315,9 +1385,9 @@ Content-Type: application/json;charset=UTF-8
}
...
}
-</verbatim>
+```
-*Getting all the Workflows corresponding to a Coordinator Action:*
+**Getting all the Workflows corresponding to a Coordinator Action:**
A coordinator action kicks off different workflows for its original run and all subsequent reruns.
Getting a list of those workflow ids is a useful tool to keep track of your actions' runs and
@@ -1326,13 +1396,15 @@ and start and end times for quick reference.
Both v1 and v2 API are supported. v0 is not supported.
-<verbatim>
+
+```
GET /oozie/v2/job/0000001-111219170928042-oozie-joe-C@1?show=allruns
-</verbatim>
+```
+
+**Response**
-*Response*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
@@ -1356,45 +1428,51 @@ Content-Type: application/json;charset=UTF-8
"endTime":"Mon, 24 Mar 2014 23:44:24 GMT"
}
]}
-</verbatim>
+```
-An alternate API is also available for the same output. With this API, one can pass the coordinator *JOB* Id
-followed by query params - type=action and scope=<action-number>. One single action number can be passed at a time.
+An alternate API is also available for the same output. With this API, one can pass the coordinator **JOB** Id
+followed by query params - `type=action` and `scope=<action-number>`. One single action number can be passed at a time.
-<verbatim>
+
+```
GET /oozie/v2/job/0000001-111219170928042-oozie-joe-C?show=allruns&type=action&scope=1
-</verbatim>
+```
+
+**Retrieve a subset of actions**
-*Retrieve a subset of actions*
+Query parameters, `offset` and `length` can be specified with a workflow job to retrieve specific actions. Default is offset=0, len=1000
-Query parameters, =offset= and =length= can be specified with a workflow job to retrieve specific actions. Default is offset=0, len=1000
-<verbatim>
+```
GET /oozie/v1/job/0000002-130507145349661-oozie-joe-W?show=info&offset=5&len=10
-</verbatim>
-Query parameters, =offset=, =length=, =filter= can be specified with a coordinator job to retrieve specific actions.
-Query parameter, =order= with value "desc" can be used to retrieve the latest coordinator actions materialized instead of actions from @1.
-Query parameters =filter= can be used to retrieve coordinator actions matching specific status.
+```
+Query parameters, `offset`, `length`, `filter` can be specified with a coordinator job to retrieve specific actions.
+Query parameter, `order` with value "desc" can be used to retrieve the latest coordinator actions materialized instead of actions from @1.
+Query parameters `filter` can be used to retrieve coordinator actions matching specific status.
Default is offset=0, len=0 for v2/job (i.e., does not return any coordinator actions) and offset=0, len=1000 with v1/job and v0/job.
-So if you need actions to be returned with v2 API, specifying =len= parameter is necessary.
-Default =order= is "asc".
-<verbatim>
+So if you need actions to be returned with v2 API, specifying `len` parameter is necessary.
+Default `order` is "asc".
+
+```
GET /oozie/v1/job/0000001-111219170928042-oozie-joe-C?show=info&offset=5&len=10&filter=status%3DKILLED&order=desc
-</verbatim>
-Note that the filter is URL encoded, its decoded value is <code>status=KILLED</code>.
-<verbatim>
+```
+Note that the filter is URL encoded, its decoded value is `status=KILLED`.
+
+```
GET /oozie/v1/job/0000001-111219170928042-oozie-joe-C?show=info&filter=status%21%3DSUCCEEDED&order=desc
-</verbatim>
+```
This retrieves coordinator actions except for SUCCEEDED status, which is useful for debugging.
-*Retrieve information of the retry attempts of the workflow action:*
+**Retrieve information of the retry attempts of the workflow action:**
-<verbatim>
+
+```
GET oozie/v2/job/0000000-161212175234862-oozie-puru-W@pig-node?show=retries
-</verbatim>
+```
+
+**Response**
-*Response*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
@@ -1415,21 +1493,23 @@ Content-Type: application/json;charset=UTF-8
}
]
}
-</verbatim>
+```
----++++ Job Application Definition
+#### Job Application Definition
A HTTP GET request retrieves the workflow or a coordinator job definition file.
-*Request:*
+**Request:**
-<verbatim>
+
+```
GET /oozie/v1/job/job-3?show=definition
-</verbatim>
+```
+
+**Response for a workflow job:**
-*Response for a workflow job:*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/xml;charset=UTF-8
.
@@ -1439,53 +1519,57 @@ Content-Type: application/xml;charset=UTF-8
...
<end name='end' />
</workflow-app>
-</verbatim>
+```
-*Response for a coordinator job:*
+**Response for a coordinator job:**
-<verbatim>
+
+```
HTTP/1.1 200 OK
Content-Type: application/xml;charset=UTF-8
.
<?xml version="1.0" encoding="UTF-8"?>
-<coordinator-app name='abc-app' xmlns="uri:oozie:coordinator:0.1" frequency="${days(1)}
+<coordinator-app name='abc-app' xmlns="uri:oozie:coordinator:0.1" frequency="${days(1)}
start="2009-01-01T00:00Z" end="2009-12-31T00:00Z" timezone="America/Los_Angeles">
<datasets>
...
</datasets>
...
</coordinator-app>
-</verbatim>
+```
+
+**Response for a bundle job:**
-*Response for a bundle job:*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/xml;charset=UTF-8
.
<?xml version="1.0" encoding="UTF-8"?>
-<bundle-app name='abc-app' xmlns="uri:oozie:coordinator:0.1"
+<bundle-app name='abc-app' xmlns="uri:oozie:coordinator:0.1"
start="2009-01-01T00:00Z" end="2009-12-31T00:00Z"">
<datasets>
...
</datasets>
...
</bundle-app>
-</verbatim>
+```
----++++ Job Log
+#### Job Log
An HTTP GET request retrieves the job log.
-*Request:*
+**Request:**
+
-<verbatim>
+```
GET /oozie/v1/job/job-3?show=log
-</verbatim>
+```
-*Response:*
+**Response:**
-<verbatim>
+
+```
HTTP/1.1 200 OK
Content-Type: text/plain;charset=UTF-8
.
@@ -1493,21 +1577,23 @@ Content-Type: text/plain;charset=UTF-8
23:21:31,272 TRACE oozieapp:526 - USER[bansalm] GROUP[other] TOKEN[-] APP[test-wf] JOB[0-20090518232130-oozie-tucu] ACTION[mr-1] Start
23:21:31,305 TRACE oozieapp:526 - USER[bansalm] GROUP[other] TOKEN[-] APP[test-wf] JOB[0-20090518232130-oozie-tucu] ACTION[mr-1] End
...
-</verbatim>
+```
----++++ Job Error Log
+#### Job Error Log
An HTTP GET request retrieves the job error log.
-*Request:*
+**Request:**
+
-<verbatim>
+```
GET /oozie/v2/job/0000000-150121110331712-oozie-puru-B?show=errorlog
-</verbatim>
+```
-*Response:*
+**Response:**
-<verbatim>
+
+```
HTTP/1.1 200 OK
Content-Type: text/plain;charset=UTF-8
2015-01-21 11:33:29,090 WARN CoordSubmitXCommand:523 - SERVER[-] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-150121110331712-oozie-puru-B] ACTION[] SAXException :
@@ -1515,46 +1601,50 @@ org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 22; cvc-complex-typ
at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
at org.apache.xerces.util.ErrorHandlerWrapper.error(Unknown Source)
...
-</verbatim>
+```
----++++ Job Audit Log
+#### Job Audit Log
An HTTP GET request retrieves the job audit log.
-*Request:*
+**Request:**
+
-<verbatim>
+```
GET /oozie/v2/job/0000000-150322000230582-oozie-puru-C?show=auditlog
-</verbatim>
+```
-*Response:*
+**Response:**
-<verbatim>
+
+```
HTTP/1.1 200 OK
Content-Type: text/plain;charset=UTF-8
2015-03-22 00:04:35,494 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [start], PARAMETER [null], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
2015-03-22 00:05:13,823 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
2015-03-22 00:06:59,561 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
2015-03-22 23:22:20,012 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [suspend], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
-2015-03-22 23:28:48,218 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [resume], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]</verbatim>
-</verbatim>
+2015-03-22 23:28:48,218 INFO oozieaudit:520 - IP [-], USER [purushah], GROUP [null], APP [-], JOBID [0000000-150322000230582-oozie-puru-C], OPERATION [resume], PARAMETER [0000000-150322000230582-oozie-puru-C], STATUS [SUCCESS], HTTPCODE [200], ERRORCODE [null], ERRORMESSAGE [null]
+```
----++++ Filtering the server logs with logfilter options
+#### Filtering the server logs with logfilter options
User can provide multiple option to filter logs using -logfilter opt1=val1;opt2=val1;opt3=val1. This can be used to fetch only just logs of interest faster as fetching Oozie server logs is slow due to the overhead of pattern matching.
-<verbatim>
+
+```
GET /oozie/v1/job/0000003-140319184715726-oozie-puru-C?show=log&logfilter=limit=3;loglevel=WARN
-</verbatim>
+```
-Refer to the [[DG_CommandLineTool#Filtering_the_server_logs_with_logfilter_options][Filtering the server logs with logfilter options]] for more details.
+Refer to the [Filtering the server logs with logfilter options](DG_CommandLineTool.html#Filtering_the_server_logs_with_logfilter_options) for more details.
----++++ Job graph
+#### Job graph
+
+An `HTTP GET` request returns the image of the workflow DAG (rendered as a PNG or SVG image, or as a DOT string).
-An =HTTP GET= request returns the image of the workflow DAG (rendered as a PNG or SVG image, or as a DOT string).
* The nodes that are being executed are painted yellow
* The nodes that have successfully executed are painted green
* The nodes that have failed execution are painted red
@@ -1563,132 +1653,148 @@ An =HTTP GET= request returns the image of the workflow DAG (rendered as a PNG o
* An arc painted red marks the failure of the node and highlights the _error_ action
* An arc painted gray marks a path not taken yet
-*PNG request:*
-<verbatim>
+**PNG request:**
+
+```
GET /oozie/v1/job/job-3?show=graph[&show-kill=true][&format=png]
-</verbatim>
+```
-*PNG response:*
-<verbatim>
+**PNG response:**
+
+```
HTTP/1.1 200 OK
Content-Type: image/png
Content-Length: {image_size_in_bytes}
{image_bits}
+```
+
+**SVG request:**
-*SVG request:*
-<verbatim>
+```
GET /oozie/v1/job/job-3?show=graph[&show-kill=true]&format=svg
-</verbatim>
+```
-*SVG response:*
-<verbatim>
+**SVG response:**
+
+```
HTTP/1.1 200 OK
Content-Type: image/svg+xml
Content-Length: {image_size_in_bytes}
{image_bits}
+```
+
+**DOT request:**
-*DOT request:*
-<verbatim>
+```
GET /oozie/v1/job/job-3?show=graph[&show-kill=true]&format=dot
-</verbatim>
+```
-*DOT response:*
-<verbatim>
+**DOT response:**
+
+```
HTTP/1.1 200 OK
Content-Type: text/plain
Content-Length: {dot_size_in_bytes}
{dot_bytes}
-</verbatim>
+```
-The optional =show-kill= parameter shows =kill= node in the graph. Valid values for this parameter are =1=, =yes=, and =true=.
-This parameter has no effect when workflow fails and the failure node leads to the =kill= node; in that case =kill= node is shown
+The optional `show-kill` parameter shows `kill` node in the graph. Valid values for this parameter are `1`, `yes`, and `true`.
+This parameter has no effect when workflow fails and the failure node leads to the `kill` node; in that case `kill` node is shown
always.
-The optional =format= parameter describes whether the response has to be rendered as a PNG image, or an SVG image, or a DOT string.
-When omitted, =format= is considered as =png= for backwards compatibility. Oozie Web UI uses the =svg= =format=.
+The optional `format` parameter describes whether the response has to be rendered as a PNG image, or an SVG image, or a DOT string.
+When omitted, `format` is considered as `png` for backwards compatibility. Oozie Web UI uses the `svg` `format`.
The node labels are the node names provided in the workflow XML.
-This API returns =HTTP 400= when run on a resource other than a workflow, viz. bundle and coordinator.
+This API returns `HTTP 400` when run on a resource other than a workflow, viz. bundle and coordinator.
----++++ Job Status
+#### Job Status
-An =HTTP GET= request that returns the current status (e.g. =SUCCEEDED=, =KILLED=, etc) of a given job. If you are only interested
-in the status, and don't want the rest of the information that the =info= query provides, it is recommended to use this call
+An `HTTP GET` request that returns the current status (e.g. `SUCCEEDED`, `KILLED`, etc) of a given job. If you are only interested
+in the status, and don't want the rest of the information that the `info` query provides, it is recommended to use this call
as it is more efficient.
-*Request*
-<verbatim>
+**Request**
+
+```
GET /oozie/v2/job/0000000-140908152307821-oozie-rkan-C?show=status
-</verbatim>
+```
-*Response*
+**Response**
-<verbatim>
+
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
{
"status" : "SUCCEEDED"
}
-</verbatim>
+```
It accepts any valid Workflow Job ID, Coordinator Job ID, Coordinator Action ID, or Bundle Job ID.
----++++ Changing job SLA definition and alerting
-An =HTTP PUT= request to change job SLA alert status/SLA definition.
+#### Changing job SLA definition and alerting
+An `HTTP PUT` request to change job SLA alert status/SLA definition.
* All sla commands takes actions-list or date parameter.
- * =date=: a comma-separated list of date ranges. Each date range element is specified with dates separated by =::=
- * =action-list=: a comma-separated list of action ranges. Each action range is specified with two action numbers separated by =-=
- * For bundle jobs additional =coordinators= (coord_name/id) parameter can be passed.
- * Sla change command need extra parameter =value= to specify new sla definition.
-
-
+ * `date`: a comma-separated list of date ranges. Each date range element is specified with dates separated by `::`
+ * `action-list`: a comma-separated list of action ranges. Each action range is specified with two action numbers separated by `-`
+ * For bundle jobs additional `coordinators` (coord_name/id) parameter can be passed.
+ * Sla change command need extra parameter `value` to specify new sla definition.
* Changing SLA definition
+
SLA definition of should-start, should-end, nominal-time and max-duration can be changed.
-<verbatim>
+
+```
PUT /oozie/v2/job/0000003-140319184715726-oozie-puru-C?action=sla-change&value=<key>=<value>;...;<key>=<value>
-</verbatim>
+```
* Disabling SLA alert
-<verbatim>
+
+```
PUT /oozie/v2/job/0000003-140319184715726-oozie-puru-C?action=sla-disable&action-list=3-4
-</verbatim>
+```
Will disable SLA alert for actions 3 and 4.
-<verbatim>
+
+```
PUT /oozie/v1/job/0000003-140319184715726-oozie-puru-C?action=sla-disable&date=2009-02-01T00:10Z::2009-03-01T00:10Z
-</verbatim>
+```
Will disable SLA alert for actions whose nominal time is in-between 2009-02-01T00:10Z 2009-03-01T00:10Z (inclusive).
-<verbatim>
+
+```
PUT /oozie/v1/job/0000004-140319184715726-oozie-puru-B?action=sla-disable&date=2009-02-01T00:10Z::2009-03-01T00:10Z&coordinators=abc
-</verbatim>
+```
For bundle jobs additional coordinators (list of comma separated coord_name/id) parameter can be passed.
* Enabling SLA alert
-<verbatim>
+
+```
PUT /oozie/v2/job/0000003-140319184715726-oozie-puru-C?action=sla-enable&action-list=1,14,17-20
-</verbatim>
+```
Will enable SLA alert for actions 1,14,17,18,19,20.
----+++ Getting missing dependencies of coordinator action(s)
+### Getting missing dependencies of coordinator action(s)
-<verbatim>
+
+```
GET oozie/v2/job/0000000-170104115137443-oozie-puru-C?show=missing-dependencies&action-list=1,20
-</verbatim>
+```
+
+**Response**
-*Response*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
@@ -1728,22 +1834,24 @@ Content-Type: application/json;charset=UTF-8
"id": 20
}]
}
-</verbatim>
----++++ Jobs Information
+```
+#### Jobs Information
A HTTP GET request retrieves workflow and coordinator jobs information.
-*Request:*
+**Request:**
-<verbatim>
+
+```
GET /oozie/v1/jobs?filter=user%3Dbansalm&offset=1&len=50&timezone=GMT
-</verbatim>
+```
+
+Note that the filter is URL encoded, its decoded value is `user=bansalm`.
-Note that the filter is URL encoded, its decoded value is <code>user=bansalm</code>.
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
@@ -1793,11 +1901,12 @@ Content-Type: application/json;charset=UTF-8
...
]
}
-</verbatim>
+```
No action information is returned when querying for multiple jobs.
-The syntax for the filter is <verbatim>[NAME=VALUE][;NAME=VALUE]*</verbatim>
+
+The syntax for the filter is `[NAME=VALUE][;NAME=VALUE]*`
Valid filter names are:
@@ -1809,36 +1918,38 @@ Valid filter names are:
* status: the status of the job
* startCreatedTime : the start of the window about workflow job's created time
* endCreatedTime : the end of above window
- * sortby: order the results. Supported values for =sortby= are: =createdTime= and =lastModifiedTime=
+ * sortby: order the results. Supported values for `sortby` are: `createdTime` and `lastModifiedTime`
The query will do an AND among all the filter names.
The query will do an OR among all the filter values for the same name. Multiple values must be specified as different
name value pairs.
-Additionally the =offset= and =len= parameters can be used for pagination. The start parameter is base 1.
+Additionally the `offset` and `len` parameters can be used for pagination. The start parameter is base 1.
-Moreover, the =jobtype= parameter could be used to determine what type of job is looking for.
-The valid values of job type are: =wf=, =coordinator= or =bundle=.
+Moreover, the `jobtype` parameter could be used to determine what type of job is looking for.
+The valid values of job type are: `wf`, `coordinator` or `bundle`.
-startCreatedTime and endCreatedTime should be specified either in *ISO8601 (UTC)* format *(yyyy-MM-dd'T'HH:mm'Z')* or
+startCreatedTime and endCreatedTime should be specified either in **ISO8601 (UTC)** format **(yyyy-MM-dd'T'HH:mm'Z')** or
a offset value in days or hours or minutes from the current time. For example, -2d means the (current time - 2 days),
-3h means the (current time - 3 hours), -5m means the (current time - 5 minutes).
----++++ Bulk modify jobs
+#### Bulk modify jobs
A HTTP PUT request can kill, suspend, or resume all jobs that satisfy the url encoded parameters.
-*Request:*
+**Request:**
+
-<verbatim>
+```
PUT /oozie/v1/jobs?action=kill&filter=name%3Dcron-coord&offset=1&len=50&jobtype=coordinator
-</verbatim>
+```
This request will kill all the coordinators with name=cron-coord up to 50 of them.
-Note that the filter is URL encoded, its decoded value is <code>name=cron-coord</code>.
-The syntax for the filter is <verbatim>[NAME=VALUE][;NAME=VALUE]*</verbatim>
+Note that the filter is URL encoded, its decoded value is `name=cron-coord`.
+
+The syntax for the filter is `[NAME=VALUE][;NAME=VALUE]*`
Valid filter names are:
@@ -1852,13 +1963,14 @@ The query will do an AND among all the filter names.
The query will do an OR among all the filter values for the same name. Multiple values must be specified as different
name value pairs.
-Additionally the =offset= and =len= parameters can be used for pagination. The start parameter is base 1.
+Additionally the `offset` and `len` parameters can be used for pagination. The start parameter is base 1.
+
+Moreover, the `jobtype` parameter could be used to determine what type of job is looking for.
+The valid values of job type are: `wf`, `coordinator` or `bundle`
-Moreover, the =jobtype= parameter could be used to determine what type of job is looking for.
-The valid values of job type are: =wf=, =coordinator= or =bundle=
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
@@ -1895,17 +2007,19 @@ Content-Type: application/json;charset=UTF-8
...
]
}
-</verbatim>
+```
-<verbatim>
+
+```
PUT /oozie/v1/jobs?action=suspend&filter=status%3Drunning&offset=1&len=50&jobtype=wf
-</verbatim>
+```
This request will suspend all the workflows with status=running up to 50 of them.
-Note that the filter is URL encoded, its decoded value is <code>status=running</code>.
+Note that the filter is URL encoded, its decoded value is `status=running`.
+
+**Response:**
-*Response:*
-<verbatim>
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
@@ -1943,21 +2057,21 @@ Content-Type: application/json;charset=UTF-8
...
]
}
-</verbatim>
+```
----++++ Jobs information using Bulk API
+#### Jobs information using Bulk API
A HTTP GET request retrieves a bulk response for all actions, corresponding to a particular bundle, that satisfy user specified criteria.
This is useful for monitoring purposes, where user can find out about the status of downstream jobs with a single bulk request.
The criteria are used for filtering the actions returned. Valid options (_case insensitive_) for these request criteria are:
- * *bundle*: the application name from the bundle definition
- * *coordinators*: the application name(s) from the coordinator definition.
- * *actionStatus*: the status of coordinator action (Valid values are WAITING, READY, SUBMITTED, RUNNING, SUSPENDED, TIMEDOUT, SUCCEEDED, KILLED, FAILED)
- * *startCreatedTime*: the start of the window you want to look at, of the actions' created time
- * *endCreatedTime*: the end of above window
- * *startScheduledTime*: the start of the window you want to look at, of the actions' scheduled i.e. nominal time.
- * *endScheduledTime*: the end of above window
+ * **bundle**: the application name from the bundle definition
+ * **coordinators**: the application name(s) from the coordinator definition.
+ * **actionStatus**: the status of coordinator action (Valid values are WAITING, READY, SUBMITTED, RUNNING, SUSPENDED, TIMEDOUT, SUCCEEDED, KILLED, FAILED)
+ * **startCreatedTime**: the start of the window you want to look at, of the actions' created time
+ * **endCreatedTime**: the end of above window
+ * **startScheduledTime**: the start of the window you want to look at, of the actions' scheduled i.e. nominal time.
+ * **endScheduledTime**: the end of above window
Specifying 'bundle' is REQUIRED. All the rest are OPTIONAL but that might result in thousands of results depending on the size of your job. (pagination comes into play then)
@@ -1966,29 +2080,32 @@ For e.g if the query string is only "bundle=MyBundle", the response will have al
The query will do an AND among all the filter names, and OR among each filter name's values.
-The syntax for the request criteria is <verbatim>[NAME=VALUE][;NAME=VALUE]*</verbatim>
+
+The syntax for the request criteria is `[NAME=VALUE][;NAME=VALUE]*`
For 'coordinators' and 'actionStatus', if user wants to check for multiple values, they can be passed in a comma-separated manner.
-*Note*: The query will do an OR among them. Hence no need to repeat the criteria name
+**Note**: The query will do an OR among them. Hence no need to repeat the criteria name
-All the time values should be specified in *ISO8601 (UTC)* format i.e. *yyyy-MM-dd'T'HH:mm'Z'*
+All the time values should be specified in **ISO8601 (UTC)** format i.e. **yyyy-MM-dd'T'HH:mm'Z'**
-Additionally the =offset= and =len= parameters can be used as usual for pagination. The start parameter is base 1.
+Additionally the `offset` and `len` parameters can be used as usual for pagination. The start parameter is base 1.
If you specify a coordinator in the list, that does not exist, no error is thrown; simply the response will be empty or pertaining to the other valid coordinators.
However, if bundle name provided does not exist, an error is thrown.
-*Request:*
+**Request:**
+
-<verbatim>
+```
GET /oozie/v1/jobs?bulk=bundle%3Dmy-bundle-app;coordinators%3Dmy-coord-1,my-coord-5;actionStatus%3DKILLED&offset=1&len=50
-</verbatim>
+```
-Note that the filter is URL encoded, its decoded value is <code>user=chitnis</code>. If typing in browser URL, one can type decoded value itself i.e. using '='
+Note that the filter is URL encoded, its decoded value is `user=chitnis`. If typing in browser URL, one can type decoded value itself i.e. using '='
-*Response:*
+**Response:**
-<verbatim>
+
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
@@ -2054,22 +2171,22 @@ Content-Type: application/json;charset=UTF-8
...
]
}
-</verbatim>
+```
----++ Oozie Web Services API, V2 (Workflow , Coordinator And Bundle)
+## Oozie Web Services API, V2 (Workflow , Coordinator And Bundle)
The Oozie Web Services API is a HTTP REST JSON API.
-All responses are in =UTF-8=.
+All responses are in `UTF-8`.
-Assuming Oozie is running at =OOZIE_URL=, the following web services end points are supported:
+Assuming Oozie is running at `OOZIE_URL`, the following web services end points are supported:
- * <OOZIE_URL>/versions
- * <OOZIE_URL>/v2/admin
- * <OOZIE_URL>/v2/job
- * <OOZIE_URL>/v2/jobs
+ * \<OOZIE_URL\>/versions
+ * \<OOZIE_URL\>/v2/admin
+ * \<OOZIE_URL\>/v2/job
+ * \<OOZIE_URL\>/v2/jobs
-*Changes in v2 job API:*
+**Changes in v2 job API:**
There is a difference in the JSON format of Job Information API (*/job) particularly for map-reduce action.
No change for other actions.
@@ -2078,39 +2195,43 @@ In v2, externalId and consoleUrl point to launcher job ID, and externalChildIDs
v2 supports retrieving of JMS topic on which job notifications are sent
-*REST API URL:*
+**REST API URL:**
+
-<verbatim>
+```
GET http://localhost:11000/oozie/v2/job/0000002-130507145349661-oozie-vira-W?show=jmstopic
-</verbatim>
+```
-*Changes in v2 admin API:*
+**Changes in v2 admin API:**
v2 adds support for retrieving JMS connection information related to JMS notifications.
-*REST API URL:*
+**REST API URL:**
-<verbatim>
+
+```
GET http://localhost:11000/oozie/v2/admin/jmsinfo
-</verbatim>
+```
v2/jobs remain the same as v1/jobs
----+++ Job and Jobs End-Points
+### Job and Jobs End-Points
----++++ Job Information
+#### Job Information
A HTTP GET request retrieves the job information.
-*Request:*
+**Request:**
+
-<verbatim>
+```
GET /oozie/v2/job/job-3?show=info&timezone=GMT
-</verbatim>
+```
-*Response for a workflow job:*
+**Response for a workflow job:**
-<verbatim>
+
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
@@ -2152,38 +2273,41 @@ Content-Type: application/json;charset=UTF-8
...
]
}
-</verbatim>
+```
----++++ Managing a Job
----+++++ Ignore a Coordinator Job or Action
+#### Managing a Job
+##### Ignore a Coordinator Job or Action
-A ignore request is done with an HTTP PUT request with a =ignore=
+A ignore request is done with an HTTP PUT request with a `ignore`
-The =type= parameter supports =action= only.
-The =scope= parameter can contain coordinator action id(s) to be ignored.
-Multiple action ids can be passed to the =scope= parameter
+The `type` parameter supports `action` only.
+The `scope` parameter can contain coordinator action id(s) to be ignored.
+Multiple action ids can be passed to the `scope` parameter
-*Request:*
+**Request:**
Ignore a coordinator job
-<verbatim>
+
+```
PUT /oozie/v2/job/job-3?action=ignore
-</verbatim>
+```
Ignore coordinator actions
-<verbatim>
+
+```
PUT /oozie/v2/job/job-3?action=ignore&type=action&scope=3-4
-</verbatim>
+```
----+++ Validate End-Point
+### Validate End-Point
This endpoint is to validate a workflow, coordinator, bundle XML file.
----++++ Validate a local file
+#### Validate a local file
+
+**Request:**
-*Request:*
-<verbatim>
+```
POST /oozie/v2/validate?file=/home/test/myApp/workflow.xml
Content-Type: application/xml;charset=UTF-8
.
@@ -2207,43 +2331,46 @@ Content-Type: application/xml;charset=UTF-8
</kill>
<end name="end"/>
</workflow-app>
-</verbatim>
+```
-*Response:*
+**Response:**
-<verbatim>
+
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
{
validate: "Valid workflow-app"
}
-</verbatim>
+```
----++++ Validate a file in HDFS
+#### Validate a file in HDFS
You can validate a workflow, coordinator, bundle XML file in HDFS. The XML file must already exist in HDFS.
-*Request:*
+**Request:**
+
-<verbatim>
+```
POST /oozie/v2/validate?file=hdfs://localhost:8020/user/test/myApp/workflow.xml
Content-Type: application/xml;charset=UTF-8
.
-</verbatim>
+```
-*Response:*
+**Response:**
-<verbatim>
+
+```
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
.
{
validate: "Valid workflow-app"
}
-</verbatim>
+```
+
-</noautolink>
[11/11] oozie git commit: OOZIE-2734 [docs] Switch from TWiki to
Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
OOZIE-2734 [docs] Switch from TWiki to Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Project: http://git-wip-us.apache.org/repos/asf/oozie/repo
Commit: http://git-wip-us.apache.org/repos/asf/oozie/commit/4e5b3cb5
Tree: http://git-wip-us.apache.org/repos/asf/oozie/tree/4e5b3cb5
Diff: http://git-wip-us.apache.org/repos/asf/oozie/diff/4e5b3cb5
Branch: refs/heads/master
Commit: 4e5b3cb596ef5ffb68f401b050b30484ba21b268
Parents: d5c4f3b
Author: Andras Piros <an...@cloudera.com>
Authored: Fri Sep 14 16:05:17 2018 +0200
Committer: Andras Piros <an...@cloudera.com>
Committed: Fri Sep 14 16:05:17 2018 +0200
----------------------------------------------------------------------
README.txt | 4 +-
docs/pom.xml | 22 +-
docs/src/site/markdown/AG_OozieLogging.md | 89 +
docs/src/site/markdown/DG_QuickStart.md | 240 ++
docs/src/site/markdown/ENG_Building.md | 281 ++
.../site/markdown/ENG_Custom_Authentication.md | 80 +
docs/src/site/markdown/index.md | 109 +
.../src/site/twiki/AG_ActionConfiguration.twiki | 38 +-
.../src/site/twiki/AG_HadoopConfiguration.twiki | 47 +-
docs/src/site/twiki/AG_Install.twiki | 1012 +++---
docs/src/site/twiki/AG_Monitoring.twiki | 61 +-
docs/src/site/twiki/AG_OozieLogging.twiki | 87 -
docs/src/site/twiki/AG_OozieUpgrade.twiki | 39 +-
docs/src/site/twiki/BundleFunctionalSpec.twiki | 203 +-
.../site/twiki/CoordinatorFunctionalSpec.twiki | 2938 +++++++++---------
.../site/twiki/DG_ActionAuthentication.twiki | 76 +-
docs/src/site/twiki/DG_CommandLineTool.twiki | 1027 +++---
docs/src/site/twiki/DG_CoordinatorRerun.twiki | 23 +-
.../site/twiki/DG_CustomActionExecutor.twiki | 59 +-
.../site/twiki/DG_DistCpActionExtension.twiki | 76 +-
.../site/twiki/DG_EmailActionExtension.twiki | 81 +-
docs/src/site/twiki/DG_Examples.twiki | 102 +-
docs/src/site/twiki/DG_FluentJobAPI.twiki | 164 +-
.../src/site/twiki/DG_HCatalogIntegration.twiki | 77 +-
.../site/twiki/DG_Hive2ActionExtension.twiki | 109 +-
.../src/site/twiki/DG_HiveActionExtension.twiki | 134 +-
docs/src/site/twiki/DG_JMSNotifications.twiki | 117 +-
docs/src/site/twiki/DG_Overview.twiki | 31 +-
docs/src/site/twiki/DG_QuickStart.twiki | 230 --
docs/src/site/twiki/DG_SLAMonitoring.twiki | 230 +-
.../site/twiki/DG_ShellActionExtension.twiki | 151 +-
.../site/twiki/DG_SparkActionExtension.twiki | 173 +-
.../site/twiki/DG_SqoopActionExtension.twiki | 118 +-
docs/src/site/twiki/DG_SshActionExtension.twiki | 78 +-
docs/src/site/twiki/DG_WorkflowReRun.twiki | 27 +-
docs/src/site/twiki/ENG_Building.twiki | 270 --
.../site/twiki/ENG_Custom_Authentication.twiki | 79 -
docs/src/site/twiki/ENG_MiniOozie.twiki | 52 +-
docs/src/site/twiki/WebServicesAPI.twiki | 1205 +++----
.../src/site/twiki/WorkflowFunctionalSpec.twiki | 1694 +++++-----
docs/src/site/twiki/index.twiki | 111 -
pom.xml | 33 +-
release-log.txt | 1 +
43 files changed, 6252 insertions(+), 5526 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/README.txt
----------------------------------------------------------------------
diff --git a/README.txt b/README.txt
index 8cc0f59..f9c6bdf 100644
--- a/README.txt
+++ b/README.txt
@@ -42,7 +42,7 @@ More inforamtion could be found at:
http://oozie.apache.org/
Oozie Quick Start:
-http://oozie.apache.org/docs/4.3.0/DG_QuickStart.html
+http://oozie.apache.org/docs/5.0.0/DG_QuickStart.html
Supported Hadoop Versions:
@@ -50,7 +50,7 @@ Supported Hadoop Versions:
This version of Oozie was primarily tested against Hadoop 2.4.x and 2.6.x.
---------------------------------------
+
If you have any questions/issues, please send an email to:
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/pom.xml
----------------------------------------------------------------------
diff --git a/docs/pom.xml b/docs/pom.xml
index ae2e94c..35b868f 100644
--- a/docs/pom.xml
+++ b/docs/pom.xml
@@ -78,7 +78,7 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-project-info-reports-plugin</artifactId>
<!-- Using version because plugin management does not work for reporting plugins -->
- <version>2.2</version>
+ <version>2.9</version>
<reportSets>
<reportSet>
<reports>
@@ -89,7 +89,12 @@
</plugins>
</reporting>
<build>
- <plugins>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-project-info-reports-plugin</artifactId>
+ <version>2.9</version>
+ </plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-site-plugin</artifactId>
@@ -99,16 +104,19 @@
</configuration>
<dependencies>
<dependency>
- <!-- Customized Doxia Maven Plugin for twiki documentation -->
+ <groupId>org.apache.velocity</groupId>
+ <artifactId>velocity</artifactId>
+ <version>1.5</version>
+ </dependency>
+ <dependency>
<groupId>org.apache.maven.doxia</groupId>
- <artifactId>doxia-module-twiki</artifactId>
- <version>1.0-alpha-9.2y</version>
+ <artifactId>doxia-module-markdown</artifactId>
+ <version>1.8</version>
</dependency>
<dependency>
- <!-- Customized Doxia Maven Plugin for twiki documentation -->
<groupId>org.apache.maven.doxia</groupId>
<artifactId>doxia-core</artifactId>
- <version>1.0-alpha-9.2y</version>
+ <version>1.7</version>
</dependency>
</dependencies>
<executions>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/markdown/AG_OozieLogging.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/AG_OozieLogging.md b/docs/src/site/markdown/AG_OozieLogging.md
new file mode 100644
index 0000000..873e9e3
--- /dev/null
+++ b/docs/src/site/markdown/AG_OozieLogging.md
@@ -0,0 +1,89 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Oozie Logging
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Default Oozie Logging
+
+Oozie's logging properties can be configured in its log4j properties file (default is `oozie-log4j.properties`). Most log messages
+are configured by default to be written to the `oozie` appender.
+
+The default configuration for the `oozie` appender is shown below.
+
+### Default Configuration
+
+
+```
+log4j.appender.oozie=org.apache.log4j.rolling.RollingFileAppender
+log4j.appender.oozie.RollingPolicy=org.apache.oozie.util.OozieRollingPolicy
+log4j.appender.oozie.File=${oozie.log.dir}/oozie.log
+log4j.appender.oozie.Append=true
+log4j.appender.oozie.layout=org.apache.log4j.PatternLayout
+log4j.appender.oozie.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n
+log4j.appender.oozie.RollingPolicy.FileNamePattern=${log4j.appender.oozie.File}-%d{yyyy-MM-dd-HH}
+log4j.appender.oozie.RollingPolicy.MaxHistory=720
+```
+
+In this configuration, the active log file will be named `oozie.log` and all old log files will be named `oozie.log-yyyy-MM-dd-HH`
+(where `yyyy-MM-dd-HH` is the time that that log file was created; e.g. 2012-07-21-05). All log files are in the same directory
+(whatever `oozie.log.dir` is assigned to). A maximum of 720 older log files will be retained. The active log file is rolled every
+hour, so 720 old logs means that they are kept for 30 days before being deleted.
+
+To keep all old logs instead of deleting them, `log4j.appender.oozie.RollingPolicy.MaxHistory` can be set to `-1`.
+Additionally, `log4j.appender.oozie.RollingPolicy` can be set to `org.apache.log4j.rolling.TimeBasedRollingPolicy`, which has the
+same exact behavior as `org.apache.oozie.util.OozieRollingPolicy` except that it does not delete old logs.
+
+### Restrictions
+
+In order for Oozie logging to work 100% correctly, the following restrictions must be observed (described below and in
+the `oozie-log4j.properties` file):
+
+* The appender that Oozie uses must be named "oozie" (i.e. `log4j.appender.oozie`)
+
+* `log4j.appender.oozie.RollingPolicy.FileNamePattern` must end with "-%d{yyyy-MM-dd-HH}.gz" or "-%d{yyyy-MM-dd-HH}".
+If it ends with ".gz" the old logs will be compressed when rolled
+
+* `log4j.appender.oozie.RollingPolicy.FileNamePattern` must start with the value of `log4j.appender.oozie.File`
+
+## Previous Default Oozie Logging
+
+Oozie previously used the logging configuration shown below as the default for the `oozie` appender. The other appender that Oozie
+writes to still use a configuration similar to this.
+
+### Previous Default Configuration
+
+
+```
+log4j.appender.oozie=org.apache.log4j.DailyRollingFileAppender
+log4j.appender.oozie.File=${oozie.log.dir}/oozie.log
+log4j.appender.oozie.Append=true
+log4j.appender.oozie.layout=org.apache.log4j.PatternLayout
+log4j.appender.oozie.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n
+log4j.appender.oozie.DatePattern='.'yyyy-MM-dd-HH
+```
+
+In this configuration, the active log file will be named `oozie.log` and all old log files will be named `oozie.log.yyyy-MM-dd-HH`
+(where `yyyy-MM-dd-HH` is the time that the log file was created; e.g. 2012-07-21-05). All log files are in the same directory
+(whatever `oozie.log.dir` is assigned to). All older log files are retained. The active log file is rolled every hour.
+
+### Restrictions
+
+In order for Oozie logging to work 100% correctly, the following restrictions must be observed (described below and in the
+`oozie-log4j.properties` file):
+
+* The appender that Oozie uses must be named "oozie" (i.e. `log4j.appender.oozie`)
+
+* `log4j.appender.oozie.DatePattern` must end with either "dd" or "HH". If it ends with "HH", the log will be rolled every hour;
+if it ends with "dd", the log will be rolled every day.
+
+## Other Oozie Logging
+
+While Oozie can technically use any valid log4j Appender or configurations that violate the above restrictions, certain
+features related to logs may be disabled and/or not work correctly, and is thus not advised.
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/markdown/DG_QuickStart.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/DG_QuickStart.md b/docs/src/site/markdown/DG_QuickStart.md
new file mode 100644
index 0000000..64f67ce
--- /dev/null
+++ b/docs/src/site/markdown/DG_QuickStart.md
@@ -0,0 +1,240 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Oozie Quick Start
+
+These instructions install and run Oozie using an embedded Jetty server and an embedded Derby database.
+
+For detailed install and configuration instructions refer to [Oozie Install](AG_Install.html).
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Building Oozie
+
+### System Requirements:
+ * Unix box (tested on Mac OS X and Linux)
+ * Java JDK 1.8+
+ * Maven 3.0.1+
+ * Hadoop 2.6.0+
+ * Pig 0.10.1+
+
+JDK commands (java, javac) must be in the command path.
+
+The Maven command (mvn) must be in the command path.
+
+### Building Oozie
+
+Download a source distribution of Oozie from the "Releases" drop down menu on the [Oozie site](http://oozie.apache.org).
+
+Expand the source distribution `tar.gz` and change directories into it.
+
+The simplest way to build Oozie is to run the `mkdistro.sh` script:
+
+```
+$ bin/mkdistro.sh [-DskipTests]
+```
+
+
+Running `mkdistro.sh` will create the binary distribution of Oozie. By default, oozie war will not contain hadoop and
+hcatalog libraries, however they are required for oozie to work. There are 2 options to add these libraries:
+
+1. At install time, copy the hadoop and hcatalog libraries to libext and run oozie-setup.sh to setup Oozie. This is
+ suitable when same oozie package needs to be used in multiple set-ups with different hadoop/hcatalog versions.
+
+2. Build with -Puber which will bundle the required libraries in the oozie war. Further, the following options are
+ available to customise the versions of the dependencies:
+ ```
+ -Dhadoop.version=<version> - default 2.6.0
+ -Ptez - Bundle tez jars in hive and pig sharelibs. Useful if you want to use tez
+ +as the execution engine for those applications.
+ -Dpig.version=<version> - default 0.16.0
+ -Dpig.classifier=<classifier> - default h2
+ -Dsqoop.version=<version> - default 1.4.3
+ -Dsqoop.classifier=<classifier> - default hadoop100
+ -Djetty.version=<version> - default 9.3.20.v20170531
+ -Dopenjpa.version=<version> - default 2.2.2
+ -Dxerces.version=<version> - default 2.10.0
+ -Dcurator.version=<version> - default 2.5.0
+ -Dhive.version=<version - default 1.2.0
+ -Dhbase.version=<version> - default 1.2.3
+ -Dtez.version=<version> - default 0.8.4
+
+ *IMPORTANT:* Profile hadoop-3 must be activated if building against Hadoop 3
+ ```
+
+ More details on building Oozie can be found on the [Building Oozie](ENG_Building.html) page.
+
+## Server Installation
+
+### System Requirements
+
+ * Unix (tested in Linux and Mac OS X)
+ * Java 1.8+
+ * Hadoop
+ * [Apache Hadoop](http://hadoop.apache.org) (tested with 1.2.1 & 2.6.0+)
+ * ExtJS library (optional, to enable Oozie webconsole)
+ * [ExtJS 2.2](http://archive.cloudera.com/gplextras/misc/ext-2.2.zip)
+
+The Java 1.8+ `bin` directory should be in the command path.
+
+### Server Installation
+
+**IMPORTANT:** Oozie ignores any set value for `OOZIE_HOME`, Oozie computes its home automatically.
+
+ * Build an Oozie binary distribution
+ * Download a Hadoop binary distribution
+ * Download ExtJS library (it must be version 2.2)
+
+**NOTE:** The ExtJS library is not bundled with Oozie because it uses a different license.
+
+**NOTE:** Oozie UI browser compatibility Chrome (all), Firefox (3.5), Internet Explorer (8.0), Opera (10.5).
+
+**NOTE:** It is recommended to use a Oozie Unix user for the Oozie server.
+
+Expand the Oozie distribution `tar.gz`.
+
+Expand the Hadoop distribution `tar.gz` (as the Oozie Unix user).
+
+<a name="HadoopProxyUser"></a>
+
+**NOTE:** Configure the Hadoop cluster with proxyuser for the Oozie process.
+
+The following two properties are required in Hadoop core-site.xml:
+
+
+```
+ <!-- OOZIE -->
+ <property>
+ <name>hadoop.proxyuser.[OOZIE_SERVER_USER].hosts</name>
+ <value>[OOZIE_SERVER_HOSTNAME]</value>
+ </property>
+ <property>
+ <name>hadoop.proxyuser.[OOZIE_SERVER_USER].groups</name>
+ <value>[USER_GROUPS_THAT_ALLOW_IMPERSONATION]</value>
+ </property>
+```
+
+Replace the capital letter sections with specific values and then restart Hadoop.
+
+The ExtJS library is optional (only required for the Oozie web-console to work)
+
+**IMPORTANT:** all Oozie server scripts (`oozie-setup.sh`, `oozied.sh`, `oozie-start.sh`, `oozie-run.sh`
+and `oozie-stop.sh`) run only under the Unix user that owns the Oozie installation directory,
+if necessary use `sudo -u OOZIE_USER` when invoking the scripts.
+
+As of Oozie 3.3.2, use of `oozie-start.sh`, `oozie-run.sh`, and `oozie-stop.sh` has
+been deprecated and will print a warning. The `oozied.sh` script should be used
+instead; passing it `start`, `run`, or `stop` as an argument will perform the
+behaviors of `oozie-start.sh`, `oozie-run.sh`, and `oozie-stop.sh` respectively.
+
+Create a **libext/** directory in the directory where Oozie was expanded.
+
+If using the ExtJS library copy the ZIP file to the **libext/** directory. If hadoop and hcatalog libraries are not
+already included in the war, add the corresponding libraries to **libext/** directory.
+
+A "sharelib create -fs fs_default_name [-locallib sharelib]" command is available when running oozie-setup.sh
+for uploading new sharelib into hdfs where the first argument is the default fs name
+and the second argument is the Oozie sharelib to install, it can be a tarball or the expanded version of it.
+If the second argument is omitted, the Oozie sharelib tarball from the Oozie installation directory will be used.
+Upgrade command is deprecated, one should use create command to create new version of sharelib.
+Sharelib files are copied to new `lib_<timestamped>` directory. At start, server picks the sharelib from latest time-stamp directory.
+While starting server also purge sharelib directory which is older than sharelib retention days
+(defined as oozie.service.ShareLibService.temp.sharelib.retention.days and 7 days is default).
+
+db create|upgrade|postupgrade -run [-sqlfile \<FILE\>] command is for create, upgrade or postupgrade oozie db with an
+optional sql file
+
+Run the `oozie-setup.sh` script to configure Oozie with all the components added to the **libext/** directory.
+
+
+```
+$ bin/oozie-setup.sh sharelib create -fs <FS_URI> [-locallib <PATH>]
+ sharelib upgrade -fs <FS_URI> [-locallib <PATH>]
+ db create|upgrade|postupgrade -run [-sqlfile <FILE>]
+```
+
+**IMPORTANT**: If the Oozie server needs to establish secure connection with an external server with a self-signed certificate,
+make sure you specify the location of a truststore that contains required certificates. It can be done by configuring
+`oozie.https.truststore.file` in `oozie-site.xml`, or by setting the `javax.net.ssl.trustStore` system property.
+If it is set in both places, the value passed as system property will be used.
+
+Create the Oozie DB using the 'ooziedb.sh' command line tool:
+
+
+```
+$ bin/ooziedb.sh create -sqlfile oozie.sql -run
+
+Validate DB Connection.
+DONE
+Check DB schema does not exist
+DONE
+Check OOZIE_SYS table does not exist
+DONE
+Create SQL schema
+DONE
+DONE
+Create OOZIE_SYS table
+DONE
+
+Oozie DB has been created for Oozie version '3.2.0'
+
+$
+```
+
+Start Oozie as a daemon process run:
+
+
+```
+$ bin/oozied.sh start
+```
+
+To start Oozie as a foreground process run:
+
+
+```
+$ bin/oozied.sh run
+```
+
+Check the Oozie log file `logs/oozie.log` to ensure Oozie started properly.
+
+Using the Oozie command line tool check the status of Oozie:
+
+
+```
+$ bin/oozie admin -oozie http://localhost:11000/oozie -status
+```
+
+Using a browser go to the [Oozie web console](http://localhost:11000/oozie.html), Oozie status should be **NORMAL**.
+
+Refer to the [Running the Examples](DG_Examples.html) document for details on running the examples.
+
+## Client Installation
+
+### System Requirements
+
+ * Unix (tested in Linux and Mac OS X)
+ * Java 1.8+
+
+The Java 1.8+ `bin` directory should be in the command path.
+
+### Client Installation
+
+Copy and expand the `oozie-client` TAR.GZ file bundled with the distribution. Add the `bin/` directory to the `PATH`.
+
+Refer to the [Command Line Interface Utilities](DG_CommandLineTool.html) document for a full reference of the `oozie`
+command line tool.
+
+NOTE: The Oozie server installation includes the Oozie client. The Oozie client should be installed in remote machines
+only.
+
+<a name="OozieShareLib"></a>
+## Oozie Share Lib Installation
+
+Oozie share lib has been installed by oozie-setup.sh create command explained in the earlier section.
+
+See the [Workflow Functional Specification](WorkflowFunctionalSpec.html#ShareLib) and [Installation](AG_Install.html#Oozie_Share_Lib) for more information about the Oozie ShareLib.
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/markdown/ENG_Building.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/ENG_Building.md b/docs/src/site/markdown/ENG_Building.md
new file mode 100644
index 0000000..92f988a
--- /dev/null
+++ b/docs/src/site/markdown/ENG_Building.md
@@ -0,0 +1,281 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Building Oozie
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## System Requirements
+
+ * Unix box (tested on Mac OS X and Linux)
+ * Java JDK 1.8+
+ * [Maven 3.0.1+](http://maven.apache.org/)
+ * [Hadoop 2.6.0+](http://hadoop.apache.org/core/releases.html)
+ * [Pig 0.10.1+](http://hadoop.apache.org/pig/releases.html)
+
+JDK commands (java, javac) must be in the command path.
+
+The Maven command (mvn) must be in the command path.
+
+## Oozie Documentation Generation
+
+To generate the documentation, Oozie uses a Doxia plugin for Maven with markdown support.
+
+
+
+
+
+
+```
+$ mvn install
+```
+
+<a name="SshSetup"></a>
+## Passphrase-less SSH Setup
+
+**NOTE: SSH actions are deprecated in Oozie 2.**
+
+To run SSH Testcases and for easier Hadoop start/stop configure SSH to localhost to be passphrase-less.
+
+Create your SSH keys without a passphrase and add the public key to the authorized file:
+
+
+```
+$ ssh-keygen -t dsa
+$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys2
+```
+
+Test that you can ssh without password:
+
+
+```
+$ ssh localhost
+```
+
+## Building with different Java Versions
+
+Oozie requires a minimum Java version of 1.8. Any newer version can be used but by default bytecode will be generated
+which is compatible with 1.8. This can be changed by specifying the build property **targetJavaVersion**.
+
+## Building and Testing Oozie
+
+The JARs for the specified Hadoop and Pig versions must be available in one of the Maven repositories defined in Oozie
+main 'pom.xml' file. Or they must be installed in the local Maven cache.
+
+### Examples Running Oozie Testcases with Different Configurations
+
+**Using embedded Hadoop minicluster with 'simple' authentication:**
+
+
+```
+$ mvn clean test
+```
+
+**Using a Hadoop cluster with 'simple' authentication:**
+
+
+```
+$ mvn clean test -Doozie.test.hadoop.minicluster=false
+```
+
+**Using embedded Hadoop minicluster with 'simple' authentication and Derby database:**
+
+
+```
+$ mvn clean test -Doozie.test.hadoop.minicluster=false -Doozie.test.db=derby
+```
+
+**Using a Hadoop cluster with 'kerberos' authentication:**
+
+
+```
+$ mvn clean test -Doozie.test.hadoop.minicluster=false -Doozie.test.hadoop.security=kerberos
+```
+
+NOTE: The embedded minicluster cannot be used when testing with 'kerberos' authentication.
+
+**Using a custom Oozie configuration for testcases:**
+
+
+```
+$ mvn clean test -Doozie.test.config.file=/home/tucu/custom-oozie-sitel.xml
+```
+
+**Running the testcases with different databases:**
+
+
+```
+$ mvn clean test -Doozie.test.db=[hsqldb*|derby|mysql|postgres|oracle]
+```
+
+Using `mysql` and `oracle` enables profiles that will include their JARs files in the build. If using
+ `oracle`, the Oracle JDBC JAR file must be manually installed in the local Maven cache (the JAR is
+not available in public Maven repos).
+
+### Build Options Reference
+
+All these options can be set using **-D**.
+
+Except for the options marked with `(*)`, the options can be specified in the `test.properties` in the root
+of the Oozie project. The options marked with `(*)` are used in Maven POMs, thus they don't take effect if
+specified in the `test.properties` file (which is loaded by the `XTestCase` class at class initialization time).
+
+**hadoop.version** `(*)`: indicates the Hadoop version you wish to build Oozie against specifically. It will
+substitute this value in the Oozie POM properties and pull the corresponding Hadoop artifacts from Maven.
+The default version is 2.6.0 and that is the minimum supported Hadoop version.
+
+**generateSite** (*): generates Oozie documentation, default is undefined (no documentation is generated)
+
+**skipTests** (*): skips the execution of all testcases, no value required, default is undefined
+
+**test**= (*): runs a single test case, to run a test give the test class name without package and extension, no default
+
+**oozie.test.db**= (*): indicates the database to use for running the testcases, supported values are 'hsqldb', 'derby',
+ 'mysql', 'postgres' and 'oracle'; default value is 'hsqldb'. For each database there is
+ `core/src/test/resources/DATABASE-oozie-site.xml` file preconfigured.
+
+**oozie.test.properties** (*): indicates the file to load the test properties from, by default is `test.properties`.
+Having this option allows having different test properties sets, for example: minicluster, simple & kerberos.
+
+**oozie.test.waitfor.ratio**= : multiplication factor for testcases using waitfor, the ratio is used to adjust the
+effective time out. For slow machines the ratio should be increased. The default value is `1`.
+
+**oozie.test.config.file**= : indicates a custom Oozie configuration file for running the testcases. The specified file
+must be an absolute path. For example, it can be useful to specify different database than HSQL for running the
+testcases.
+
+**oozie.test.hadoop.minicluster**= : indicates if Hadoop minicluster should be started for testcases, default value 'true'
+
+**oozie.test.job.tracker**= : indicates the URI of the JobTracker when using a Hadoop cluster for testing, default value
+'localhost:8021'
+
+**oozie.test.name.node**= : indicates the URI of the NameNode when using a Hadoop cluster for testing, default value
+'`hdfs://localhost:8020`'
+
+**oozie.test.hadoop.security**= : indicates the type of Hadoop authentication for testing, valid values are 'simple' or
+'kerberos, default value 'simple'
+
+**oozie.test.kerberos.keytab.file**= : indicates the location of the keytab file, default value
+'${user.home}/oozie.keytab'
+
+**oozie.test.kerberos.realm**= : indicates the Kerberos real, default value 'LOCALHOST'
+
+**oozie.test.kerberos.oozie.principal**= : indicates the Kerberos principal for oozie, default value
+'${user.name}/localhost'
+
+**oozie.test.kerberos.jobtracker.principal**= : indicates the Kerberos principal for the JobTracker, default value
+'mapred/localhost'
+
+**oozie.test.kerberos.namenode.principal**= : indicates the Kerberos principal for the NameNode, default value
+'hdfs/localhost'
+
+**oozie.test.user.oozie**= : specifies the user ID used to start Oozie server in testcases, default value
+is `${user.name}`.
+
+**oozie.test.user.test**= : specifies primary user ID used as the user submitting jobs to Oozie Server in testcases,
+default value is `test`.
+
+**oozie.test.user.test2**= : specifies secondary user ID used as the user submitting jobs to Oozie Server in testcases,
+default value is `test2`.
+
+**oozie.test.user.test3**= : specifies secondary user ID used as the user submitting jobs to Oozie Server in testcases,
+default value is `test3`.
+
+**oozie.test.group**= : specifies group ID used as group when submitting jobs to Oozie Server in testcases,
+default value is `testg`.
+
+NOTE: The users/group specified in **oozie.test.user.test2**, **oozie.test.user.test3** and **oozie.test.user.group**
+are used for the authorization testcases only.
+
+**oozie.test.dir**` : specifies the directory where the `oozietests` directory will be created, default value is `/tmp=.
+The `oozietests` directory is used by testcases when they need a local filesystem directory.
+
+**hadoop.log.dir**= : specifies the directory where Hadoop minicluster will write its logs during testcases, default
+value is `/tmp`.
+
+**test.exclude**` : specifies a testcase class (just the class name) to exclude for the tests run, for example =TestSubmitCommand`.
+
+**test.exclude.pattern**` : specifies one or more patterns for testcases to exclude, for example =**/Test*Command.java`.
+
+### Testing Map Reduce Pipes Action
+
+Pipes testcases require Hadoop's **wordcount-simple** pipes binary example to run. The **wordcount-simple** pipes binary
+should be compiled for the build platform and copied into Oozie's **core/src/test/resources/** directory. The binary file
+must be named **wordcount-simple**.
+
+If the **wordcount-simple** pipes binary file is not available the testcase will do a NOP and it will print to its output
+file the following message 'SKIPPING TEST: TestPipesMain, binary 'wordcount-simple' not available in the classpath'.
+
+There are 2 testcases that use the **wordcount-simple** pipes binary, **TestPipesMain** and **TestMapReduceActionExecutor**,
+the 'SKIPPING TEST..." message would appear in the testcase log file of both testcases.
+
+### Testing using dist_test and grind
+
+Testing using [dist_test](https://github.com/cloudera/dist_test) framework with
+[grind](https://github.com/cloudera/dist_test/blob/master/docs/grind.md) front end might not work using the default 3.0.2
+version of the maven dependency plugin. It is necessary to downgrade to version 2.10 using
+`-Dmaven-dependency-plugin.version=2.10` .
+
+Maven flags for grind can be specified using `GRIND_MAVEN_FLAGS` environment variable:
+
+
+```
+export GRIND_MAVEN_FLAGS=-Dmaven.dependency.plugin.version=2.10
+grind test --java-version 8
+```
+
+## Building an Oozie Distribution
+
+An Oozie distribution bundles an embedded Jetty server.
+
+The simplest way to build Oozie is to run the `mkdistro.sh` script:
+
+```
+$ bin/mkdistro.sh [-DskipTests]
+Running =mkdistro.sh= will create the binary distribution of Oozie. The following options are available to customise
+the versions of the dependencies:
+-Puber - Bundle required hadoop and hcatalog libraries in oozie war
+-Dhadoop.version=<version> - default 2.6.0
+-Ptez - Bundle tez jars in hive and pig sharelibs. Useful if you want to use tez
+as the execution engine for those applications.
+-Dpig.version=<version> - default 0.16.0
+-Dpig.classifier=<classifier> - default h2
+-Dsqoop.version=<version> - default 1.4.3
+-Dsqoop.classifier=<classifier> - default hadoop100
+-jetty.version=<version> - default 9.3.20.v20170531
+-Dopenjpa.version=<version> - default 2.2.2
+-Dxerces.version=<version> - default 2.10.0
+-Dcurator.version=<version> - default 2.5.0
+-Dhive.version=<version> - default 1.2.0
+-Dhbase.version=<version> - default 1.2.3
+-Dtez.version=<version> - default 0.8.4
+```
+
+**IMPORTANT:** Profile hadoop-3 must be activated if building against Hadoop 3
+
+The following properties should be specified when building a release:
+
+ * -DgenerateDocs : forces the generation of Oozie documentation
+ * -Dbuild.time= : timestamps the distribution
+ * -Dvc.revision= : specifies the source control revision number of the distribution
+ * -Dvc.url= : specifies the source control URL of the distribution
+
+The provided `bin/mkdistro.sh` script runs the above Maven invocation setting all these properties to the
+right values (the 'vc.*' properties are obtained from the local git repository).
+
+## IDE Setup
+
+Eclipse and IntelliJ can use directly Oozie Maven project files.
+
+The only special consideration is that the following source directories from the `client` module must be added to
+the `core` module source path:
+
+ * `client/src/main/java` : as source directory
+ * `client/src/main/resources` : as source directory
+ * `client/src/test/java` : as test-source directory
+ * `client/src/test/resources` : as test-source directory
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/markdown/ENG_Custom_Authentication.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/ENG_Custom_Authentication.md b/docs/src/site/markdown/ENG_Custom_Authentication.md
new file mode 100644
index 0000000..c70355f
--- /dev/null
+++ b/docs/src/site/markdown/ENG_Custom_Authentication.md
@@ -0,0 +1,80 @@
+
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Creating Custom Authentication
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Hadoop-Auth Authentication Interfaces and classes
+
+1. `org.apache.hadoop.security.authentication.client.Authenticator:` Interface for client authentication mechanisms.
+
+ The following authenticators are provided in hadoop-auth:
+
+ * KerberosAuthenticator : the authenticator implements the Kerberos SPNEGO authentication sequence.
+ * PseudoAuthenticator : the authenticator implementation provides an authentication equivalent to Hadoop's Simple
+ authentication, it trusts the value of the 'user.name' Java System property.
+
+2. `org.apache.hadoop.security.authentication.server.AuthenticationHandler:` Interface for server authentication mechanisms.
+
+ * KerberosAuthenticationHandler : the authenticator handler implements the Kerberos SPNEGO authentication mechanism for HTTP.
+ * PseudoAuthenticationHandler : the authenticator handler provides a pseudo authentication mechanism that accepts the user
+ name specified as a query string parameter.
+
+3. `org.apache.hadoop.security.authentication.server.AuthenticationFilter:` A servlet filter enables protecting web application
+ resources with different authentication mechanisms provided by AuthenticationHandler. To enable the filter, web application
+ resources file (ex. web.xml) needs to include a filter class derived from `AuthenticationFilter`.
+
+ For more information have a look at the appropriate
+ [Hadoop documentation](https://hadoop.apache.org/docs/r2.7.2/hadoop-auth/index.html).
+
+## Provide Custom Authentication to Oozie Client
+
+Apache Oozie contains a default class `org.apache.oozie.client.AuthOozieClient` to support Kerberos HTTP SPNEGO authentication,
+pseudo/simple authentication and anonymous access for client connections.
+
+To provide other authentication mechanisms, an Oozie client should extend from `AuthOozieClient` and provide the following
+methods should be overridden by derived classes to provide custom authentication:
+
+ * getAuthenticator() : return corresponding Authenticator based on value specified by user at `auth` command option.
+ * createConnection() : create a singleton class at Authenticator to allow client set and get key-value configuration for
+ authentication.
+
+## Provide Custom Authentication to Oozie Server
+
+To accept custom authentication in Oozie server, a filter extends from AuthenticationFilter must be provided. This filter
+delegates to the configured authentication handler for authentication and once it obtains an `AuthenticationToken` from it, sets
+a signed HTTP cookie with the token. If HTTP cookie is provided with different key name, its cookie value can be retrieved by
+overriding `getToken()` method. Please note, only when `getToken()` return NULL, a custom authentication can be invoked and
+processed in `AuthenticationFilter.doFilter()`.
+
+The following method explains how to read it and return NULL token.
+
+```
+protected AuthenticationToken getToken(HttpServletRequest request) throws IOException, AuthenticationException {
+ String tokenStr = null;
+ Cookie[] cookies = request.getCookies();
+
+ if (cookies != null) {
+ for (Cookie cookie : cookies) {
+ if (cookie.getName().equals(AuthenticatedURL.AUTH_COOKIE)) {
+ tokenStr = cookie.getValue();
+ LOG.info("Got 'hadoop.auth' cookie from request = " + tokenStr);
+ if (tokenStr != null && !tokenStr.trim().isEmpty()) {
+ AuthenticationToken retToken = super.getToken(request);
+ return retToken;
+ }
+ } else if (cookie.getName().equals("NEWAUTH")) {
+ tokenStr = cookie.getValue();
+ // DO NOT return the token string so request can authenticated.
+ }
+ }
+ }
+ return null;
+ }
+```
+
+[::Go back to Oozie Documentation Index::](index.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/markdown/index.md
----------------------------------------------------------------------
diff --git a/docs/src/site/markdown/index.md b/docs/src/site/markdown/index.md
new file mode 100644
index 0000000..0216222
--- /dev/null
+++ b/docs/src/site/markdown/index.md
@@ -0,0 +1,109 @@
+
+
+# Oozie, Workflow Engine for Apache Hadoop
+
+Oozie v3 is a server based _Bundle Engine_ that provides a higher-level oozie abstraction that will batch a set of coordinator applications. The user will be able to start/stop/suspend/resume/rerun a set coordinator jobs in the bundle level resulting a better and easy operational control.
+
+Oozie v2 is a server based _Coordinator Engine_ specialized in running workflows based on time and data triggers.
+(e.g. wait for my input data to exist before running my workflow).
+
+Oozie v1 is a server based _Workflow Engine_ specialized in running workflow jobs with actions that
+execute Hadoop Map/Reduce and Pig jobs.
+
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Distribution Contents
+
+Oozie distribution consists of a single 'tar.gz' file containing:
+
+ * Readme, license, notice & [Release log](./release-log.txt) files.
+ * Oozie server: `oozie-server` directory.
+ * Scripts: `bin/` directory, client and server scripts.
+ * Binaries: `lib/` directory, client JAR files.
+ * Configuration: `conf/` server configuration directory.
+ * Archives:
+ * `oozie-client-*.tar.gz`: Client tools.
+ * `oozie.war`: Oozie WAR file.
+ * `docs.zip`: Documentation.
+ * `oozie-examples-*.tar.gz`: Examples.
+ * `oozie-sharelib-*.tar.gz`: Share libraries (with Streaming, Pig JARs).
+
+## Quick Start
+
+Enough reading already? Follow the steps in [Oozie Quick Start](DG_QuickStart.html) to get Oozie up and running.
+
+## Developer Documentation
+
+ * [Overview](DG_Overview.html)
+ * [Oozie Quick Start](DG_QuickStart.html)
+ * [Running the Examples](DG_Examples.html)
+ * [Workflow Functional Specification](WorkflowFunctionalSpec.html)
+ * [Coordinator Functional Specification](CoordinatorFunctionalSpec.html)
+ * [Bundle Functional Specification](BundleFunctionalSpec.html)
+ * [EL Expression Language Quick Reference](https://docs.oracle.com/javaee/7/tutorial/jsf-el.htm)
+ * [Command Line Tool](DG_CommandLineTool.html)
+ * [Workflow Re-runs Explained](DG_WorkflowReRun.html)
+ * [HCatalog Integration Explained](DG_HCatalogIntegration.html)
+ * [Oozie Client Javadocs](./client/apidocs/index.html)
+ * [Oozie Core Javadocs](./core/apidocs/index.html)
+ * [Oozie Web Services API](WebServicesAPI.html)
+ * [Action Authentication](DG_ActionAuthentication.html)
+
+### Action Extensions
+
+ * [Email Action](DG_EmailActionExtension.html)
+ * [Shell Action](DG_ShellActionExtension.html)
+ * [Hive Action](DG_HiveActionExtension.html)
+ * [Hive 2 Action](DG_Hive2ActionExtension.html)
+ * [Sqoop Action](DG_SqoopActionExtension.html)
+ * [Ssh Action](DG_SshActionExtension.html)
+ * [DistCp Action](DG_DistCpActionExtension.html)
+ * [Spark Action](DG_SparkActionExtension.html)
+ * [Writing a Custom Action Executor](DG_CustomActionExecutor.html)
+
+### Job Status and SLA Monitoring
+
+ * [JMS Notifications for Job and SLA](DG_JMSNotifications.html)
+ * [Configuring and Monitoring SLA](DG_SLAMonitoring.html)
+
+## Administrator Documentation
+
+ * [Oozie Install](AG_Install.html)
+ * [Oozie Logging](AG_OozieLogging.html)
+ * [Hadoop Configuration](AG_HadoopConfiguration.html)
+ * [Action Configuration](AG_ActionConfiguration.html)
+ * [Oozie Monitoring](AG_Monitoring.html)
+ * [Command Line Tool](DG_CommandLineTool.html)
+ * [Oozie Upgrade](AG_OozieUpgrade.html)
+
+<a name="LicenseInfo"></a>
+## Licensing Information
+
+Oozie is distributed under [Apache License 2.0](http://www.apache.org/licenses/LICENSE-2.0).
+
+For details on the license of the dependent components, refer to the
+[Dependencies Report, Licenses section](./dependencies.html#Licenses).
+
+Oozie bundles an embedded Jetty 9.x.
+
+Some of the components in the dependencies report don't mention their license in the published POM. They are:
+
+ * JDOM: [JDOM License](http://www.jdom.org/docs/faq.html#a0030) (Apache style).
+ * Oro: [Apache License 2.0](http://www.apache.org/licenses/LICENSE-2.0).
+
+
+
+## Engineering Documentation
+
+ * [Building Oozie](ENG_Building.html)
+ * [Dependencies Report](./dependencies.html)
+
+## MiniOozie Documentation
+
+ * [Testing User Oozie Applications Using MiniOozie](ENG_MiniOozie.html)
+
+## Oozie User Authentication Documentation
+
+ * [Create Custom Oozie Authentication](ENG_Custom_Authentication.html)
+
+
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/AG_ActionConfiguration.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/AG_ActionConfiguration.twiki b/docs/src/site/twiki/AG_ActionConfiguration.twiki
index 8c032a7..e020d1a 100644
--- a/docs/src/site/twiki/AG_ActionConfiguration.twiki
+++ b/docs/src/site/twiki/AG_ActionConfiguration.twiki
@@ -1,24 +1,24 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
----+!! Action Configuration
+[::Go back to Oozie Documentation Index::](index.html)
+
+# Action Configuration
Oozie supports providing default configuration for actions of a particular
action type and default configuration for all actions
----++ Hadoop Default Configuration Values
+## Hadoop Default Configuration Values
-Oozie supports action configuration equivalent to the component's =*-site.xml= and =*.properties= files.
+Oozie supports action configuration equivalent to the component's `*-site.xml` and `*.properties` files.
-The configuration property in the =oozie-site.xml= is =oozie.service.HadoopAccessorService.action.configurations=
-and its value must follow the pattern _[AUTHORITY=ACTION-CONF-DIR_,]*. Where _AUTHORITY_ is the _HOST:PORT_ of
+The configuration property in the `oozie-site.xml` is `oozie.service.HadoopAccessorService.action.configurations`
+and its value must follow the pattern _\[AUTHORITY=ACTION-CONF-DIR_,\]*. Where _AUTHORITY_ is the _HOST:PORT_ of
the Hadoop service (JobTracker/ResourceManager or HDFS). The _ACTION-CONF-DIR_ is the action configuration directory. If the specified directory is a relative path, it will be looked under the Oozie configuration directory. An absolute path can
also be specified. Oozie will load and process the action configuration files in the following order.
1. All files in _default_/*.xml (sorted by lexical name, files with names lexically lower have lesser precedence than the following ones), if present.
1. _default_.xml, if present.
- 1. All supported files in _actionname_/*, e.g. _actionname_/*.xml and _actionname_/*.properties (based on filename extension, sorted by lexical name, files with names lexically lower have lesser precedence than the following ones), if present.
+ 1. All supported files in _actionname_/\*, e.g. _actionname_/\*.xml and _actionname_/*.properties (based on filename extension, sorted by lexical name, files with names lexically lower have lesser precedence than the following ones), if present.
1. _actionname_.xml, if present.
@@ -26,7 +26,7 @@ For example, for _Hive_ action (which has the _actionname_ defined as _hive_ ),
1. All files in _default_/*.xml, if present
1. _default_.xml, if present.
- 1. All files in _hive_/*.xml and _hive_/*.properties, if present
+ 1. All files in _hive_/\*.xml and _hive_/\*.properties, if present
1. _hive_.xml, if present.
@@ -35,22 +35,23 @@ Files processed earlier for an action have the lowest precedence and can have th
In addition to explicit authorities, a '*' wildcard is supported. The configuration file associated with the wildcard
will be used as default if there is no action configuration for the requested Hadoop service.
-For example, the configuration in the =oozie-site.xml= would look like:
+For example, the configuration in the `oozie-site.xml` would look like:
+
-<verbatim>
+```
...
<property>
<name>oozie.service.HadoopAccessorService.action.configurations</name>
<value>*=hadoop-conf,jt-bar:8021=bar-cluster,nn-bar:8020=bar-cluster</value>
</property>
...
-</verbatim>
+```
The action configuration files use the Hadoop configuration syntax.
By default Oozie does not define any default action configurations.
----++ Dependency deduplication
+## Dependency deduplication
Using Oozie with Hadoop 3 may require to have dependency file names distinguishable,
which means having two files on sharelib and in your app's dependencies with identical names, leads to job submission failure.
@@ -62,7 +63,8 @@ where dependency with greater prio is used.
Real world example:
You have an application workflow which is uploaded to HDFS in /apps/app directory. You have your app.jar and dependency jars.
You also define a spark action in your workflow and set use system libs; the HDFS tree is similar to this:
-<verbatim>
+
+```
+ /apps/app/
- app.jar
- workflow.xml
@@ -75,11 +77,11 @@ You also define a spark action in your workflow and set use system libs; the HDF
- jackson-annotations-1.0.jar
+ oozie
- jackson-annotations-1.0.jar
-</verbatim>
+```
The deduplicator code will create the following list of files:
-=/apps/app/app.jar,/apps/app/libs/jackson-annotations-1.0.jar=
+`/apps/app/app.jar,/apps/app/libs/jackson-annotations-1.0.jar`
And no other files will be passed at job submission.
-[[index][::Go back to Oozie Documentation Index::]]
+[::Go back to Oozie Documentation Index::](index.html)
+
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/AG_HadoopConfiguration.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/AG_HadoopConfiguration.twiki b/docs/src/site/twiki/AG_HadoopConfiguration.twiki
index 528bf4a..ab71d7c 100644
--- a/docs/src/site/twiki/AG_HadoopConfiguration.twiki
+++ b/docs/src/site/twiki/AG_HadoopConfiguration.twiki
@@ -1,14 +1,15 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
----+!! Hadoop Configuration
+[::Go back to Oozie Documentation Index::](index.html)
----++ Hadoop Services Whitelisting
+# Hadoop Configuration
+
+## Hadoop Services Whitelisting
Oozie supports whitelisting Hadoop services (JobTracker, HDFS), via 2 configuration properties:
-<verbatim>
+
+```
...
<property>
<name>oozie.service.HadoopAccessorService.jobTracker.whitelist</name>
@@ -25,43 +26,45 @@ Oozie supports whitelisting Hadoop services (JobTracker, HDFS), via 2 configurat
</description>
</property>
...
-</verbatim>
+```
-The value must follow the pattern =[AUTHORITY,...]=. Where =AUTHORITY= is the =HOST:PORT= of
+The value must follow the pattern `[AUTHORITY,...]`. Where `AUTHORITY` is the `HOST:PORT` of
the Hadoop service (JobTracker, HDFS).
If the value is empty any HOST:PORT is accepted. Empty is the default value.
----++ Hadoop Default Configuration Values
+## Hadoop Default Configuration Values
-Oozie supports Hadoop configuration equivalent to the Hadoop =*-site.xml= files.
+Oozie supports Hadoop configuration equivalent to the Hadoop `*-site.xml` files.
-The configuration property in the =oozie-site.xml= is =oozie.service.HadoopAccessorService.hadoop.configurations=
-and its value must follow the pattern =[<AUTHORITY>=<HADOOP_CONF_DIR>,]*=. Where =<AUTHORITY>= is the =HOST:PORT= of
-the Hadoop service (JobTracker, HDFS). The =<HADOOP_CONF_DIR>= is a Hadoop configuration directory. If the specified
+The configuration property in the `oozie-site.xml` is `oozie.service.HadoopAccessorService.hadoop.configurations`
+and its value must follow the pattern `[<AUTHORITY>=<HADOOP_CONF_DIR>,]*`. Where `<AUTHORITY>` is the `HOST:PORT` of
+the Hadoop service (JobTracker, HDFS). The `<HADOOP_CONF_DIR>` is a Hadoop configuration directory. If the specified
directory is a relative path, it will be looked under the Oozie configuration directory. And absolute path can
- also be specified. Oozie will load the Hadoop =*-site.xml= files in the following order: core-site.xml, hdfs-site.xml,
+ also be specified. Oozie will load the Hadoop `*-site.xml` files in the following order: core-site.xml, hdfs-site.xml,
mapred-site.xml, yarn-site.xml, hadoop-site.xml, ssl-client.xml.
In addition to explicit authorities, a '*' wildcard is supported. The configuration file associated with the wildcard
will be used as default if there is no configuration for the requested Hadoop service.
-For example, the configuration in the =oozie-site.xml= would look like:
+For example, the configuration in the `oozie-site.xml` would look like:
+
-<verbatim>
+```
...
<property>
<name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
<value>*=hadoop-conf,jt-bar:8021=bar-cluster,nn-bar:8020=bar-cluster</value>
</property>
...
-</verbatim>
+```
The Hadoop configuration files use the Hadoop configuration syntax.
-By default Oozie defines =*=hadoop-conf= and the default values of the =hadoop-site.xml= file are:
+By default Oozie defines `*=hadoop-conf` and the default values of the `hadoop-site.xml` file are:
-<verbatim>
+
+```
<configuration>
<property>
<name>mapreduce.jobtracker.kerberos.principal</name>
@@ -80,12 +83,12 @@ By default Oozie defines =*=hadoop-conf= and the default values of the =hadoop-s
<value>yarn</value>
</property>
</configuration>
-</verbatim>
+```
----++ Limitations
+## Limitations
All actions in a workflow application must interact with the same Hadoop JobTracker and NameNode.
-[[index][::Go back to Oozie Documentation Index::]]
+[::Go back to Oozie Documentation Index::](index.html)
+
-</noautolink>
[09/11] oozie git commit: OOZIE-2734 [docs] Switch from TWiki to
Markdown (asalamon74 via andras.piros, pbacsko, gezapeti)
Posted by an...@apache.org.
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/AG_Monitoring.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/AG_Monitoring.twiki b/docs/src/site/twiki/AG_Monitoring.twiki
index 425f554..4877dfc 100644
--- a/docs/src/site/twiki/AG_Monitoring.twiki
+++ b/docs/src/site/twiki/AG_Monitoring.twiki
@@ -1,33 +1,34 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
----+!! Oozie Monitoring
+[::Go back to Oozie Documentation Index::](index.html)
-%TOC%
+# Oozie Monitoring
----++ Oozie Instrumentation
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Oozie Instrumentation
Oozie code is instrumented in several places to collect runtime metrics. The instrumentation data can be used to
determine the health of the system, performance of the system, and to tune the system.
This comes in two flavors:
+
* metrics (by default enabled since 5.0.0)
* instrumentation (deprecated and by default disabled since 5.0.0)
-The instrumentation is accessible via the Admin web-services API (see the [[WebServicesAPI#Oozie_Metrics][metrics]] and
-[[WebServicesAPI#Oozie_Instrumentation][instrumentation]] Web Services API documentations for more details) and is also written on
+The instrumentation is accessible via the Admin web-services API (see the [metrics](WebServicesAPI.html#Oozie_Metrics) and
+[instrumentation](WebServicesAPI.html#Oozie_Instrumentation) Web Services API documentations for more details) and is also written on
regular intervals to an instrumentation log.
Instrumentation data includes variables, samplers, timers and counters.
----+++ Variables
+### Variables
* oozie
* version: Oozie build version.
* configuration
- * config.dir: directory from where the configuration files are loaded. If null, all configuration files are loaded from the classpath. [[AG_Install#Oozie_Configuration][Configuration files are described here]].
+ * config.dir: directory from where the configuration files are loaded. If null, all configuration files are loaded from the classpath. [Configuration files are described here](AG_Install.html#Oozie_Configuration).
* config.file: the Oozie custom configuration for the instance.
* jvm
@@ -43,7 +44,7 @@ Instrumentation data includes variables, samplers, timers and counters.
* from.classpath: whether the config file has been read from the classpath or from the config directory.
* reload.interval: interval at which the config file will be reloaded. 0 if the config file will never be reloaded, when loaded from the classpath is never reloaded.
----+++ Samplers - Poll data at a fixed interval (default 1 sec) and report an average utilization over a longer period of time (default 60 seconds).
+### Samplers - Poll data at a fixed interval (default 1 sec) and report an average utilization over a longer period of time (default 60 seconds).
Poll for data over fixed interval and generate an average over the time interval. Unless specified, all samplers in
Oozie work on a 1 minute interval.
@@ -64,14 +65,16 @@ Oozie work on a 1 minute interval.
* requests
* version
----+++ Counters - Maintain statistics about the number of times an event has occurred, for the running Oozie instance. The values are reset if the Oozie instance is restarted.
+### Counters - Maintain statistics about the number of times an event has occurred, for the running Oozie instance. The values are reset if the Oozie instance is restarted.
* action.executors - Counters related to actions.
* [action_type]#action.[operation_performed] (start, end, check, kill)
* [action_type]#ex.[exception_type] (transient, non-transient, error, failed)
- * e.g. <verbatim>
- ssh#action.end: 306
- ssh#action.start: 316 </verbatim>
+ * e.g.
+```
+ssh#action.end: 306
+ssh#action.start: 316
+```
* callablequeue - count of events in various execution queues.
* delayed.queued: Number of commands queued with a delay.
@@ -113,7 +116,7 @@ Oozie work on a 1 minute interval.
* version
* version-GET
----+++ Timers - Maintain information about the time spent in various operations.
+### Timers - Maintain information about the time spent in various operations.
* action.executors - Counters related to actions.
* [action_type]#action.[operation_performed] (start, end, check, kill)
@@ -156,39 +159,41 @@ Oozie work on a 1 minute interval.
* version
* version-GET
----++ Oozie JVM Thread Dump
- The =admin/jvminfo.jsp= servlet can be used to get some basic jvm stats and thread dump.
-For eg: http://localhost:11000/oozie/admin/jvminfo.jsp?cpuwatch=1000&threadsort=cpu. It takes the following optional
+## Oozie JVM Thread Dump
+The `admin/jvminfo.jsp` servlet can be used to get some basic jvm stats and thread dump.
+For eg: `http://localhost:11000/oozie/admin/jvminfo.jsp?cpuwatch=1000&threadsort=cpu`. It takes the following optional
query parameters:
+
* threadsort - The order in which the threads are sorted for display. Valid values are name, cpu, state. Default is state.
* cpuwatch - Time interval in milliseconds to monitor cpu usage of threads. Default value is 0.
----++ Monitoring Database Schema Integrity
+## Monitoring Database Schema Integrity
Oozie stores all of its state in a database. Hence, ensuring that the database schema is correct is very important to ensuring that
-Oozie is healthy and behaves correctly. To help with this, Oozie includes a =SchemaCheckerService= which periodically runs and
+Oozie is healthy and behaves correctly. To help with this, Oozie includes a `SchemaCheckerService` which periodically runs and
performs a series of checks on the database schema. More specifically, it checks the following:
+
* Existence of the required tables
* Existence of the required columns in each table
* Each column has the correct type and default value
* Existence of the required primary keys and indexes
-After each run, the =SchemaCheckerService= writes the result of the checks to the Oozie log and to the "schema-checker.status"
+After each run, the `SchemaCheckerService` writes the result of the checks to the Oozie log and to the "schema-checker.status"
instrumentation variable. If there's a problem, it will be logged at the ERROR level, while correct checks are logged at the DEBUG
level.
-By default, the =SchemaCheckerService= runs every 7 days. This can be configured
-by =oozie.service.SchemaCheckerService.check.interval=
+By default, the `SchemaCheckerService` runs every 7 days. This can be configured
+by `oozie.service.SchemaCheckerService.check.interval`
-By default, the =SchemaCheckerService= will consider "extra" tables, columns, and indexes to be incorrect. Advanced users who have
+By default, the `SchemaCheckerService` will consider "extra" tables, columns, and indexes to be incorrect. Advanced users who have
added additional tables, columns, and indexes can tell Oozie to ignore these by
-setting =oozie.service.SchemaCheckerService.ignore.extras= to =false=.
+setting `oozie.service.SchemaCheckerService.ignore.extras` to `false`.
-The =SchemaCheckerService= currently only supports MySQL, PostgreSQL, and Oracle databases. SQL Server and Derby are currently not
+The `SchemaCheckerService` currently only supports MySQL, PostgreSQL, and Oracle databases. SQL Server and Derby are currently not
supported.
When Oozie HA is enabled, only one of the Oozie servers will perform the checks.
-[[index][::Go back to Oozie Documentation Index::]]
+[::Go back to Oozie Documentation Index::](index.html)
+
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/AG_OozieLogging.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/AG_OozieLogging.twiki b/docs/src/site/twiki/AG_OozieLogging.twiki
deleted file mode 100644
index ecdcfd3..0000000
--- a/docs/src/site/twiki/AG_OozieLogging.twiki
+++ /dev/null
@@ -1,87 +0,0 @@
-<noautolink>
-
-[[index][::Go back to Oozie Documentation Index::]]
-
----+!! Oozie Logging
-
-%TOC%
-
----++ Default Oozie Logging
-
-Oozie's logging properties can be configured in its log4j properties file (default is =oozie-log4j.properties=). Most log messages
-are configured by default to be written to the =oozie= appender.
-
-The default configuration for the =oozie= appender is shown below.
-
----+++ Default Configuration
-
-<verbatim>
-log4j.appender.oozie=org.apache.log4j.rolling.RollingFileAppender
-log4j.appender.oozie.RollingPolicy=org.apache.oozie.util.OozieRollingPolicy
-log4j.appender.oozie.File=${oozie.log.dir}/oozie.log
-log4j.appender.oozie.Append=true
-log4j.appender.oozie.layout=org.apache.log4j.PatternLayout
-log4j.appender.oozie.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n
-log4j.appender.oozie.RollingPolicy.FileNamePattern=${log4j.appender.oozie.File}-%d{yyyy-MM-dd-HH}
-log4j.appender.oozie.RollingPolicy.MaxHistory=720
-</verbatim>
-
-In this configuration, the active log file will be named =oozie.log= and all old log files will be named =oozie.log-yyyy-MM-dd-HH=
-(where =yyyy-MM-dd-HH= is the time that that log file was created; e.g. 2012-07-21-05). All log files are in the same directory
-(whatever =oozie.log.dir= is assigned to). A maximum of 720 older log files will be retained. The active log file is rolled every
-hour, so 720 old logs means that they are kept for 30 days before being deleted.
-
-To keep all old logs instead of deleting them, =log4j.appender.oozie.RollingPolicy.MaxHistory= can be set to =-1=.
-Additionally, =log4j.appender.oozie.RollingPolicy= can be set to =org.apache.log4j.rolling.TimeBasedRollingPolicy=, which has the
-same exact behavior as =org.apache.oozie.util.OozieRollingPolicy= except that it does not delete old logs.
-
----+++ Restrictions
-
-In order for Oozie logging to work 100% correctly, the following restrictions must be observed (described below and in
-the =oozie-log4j.properties= file):
-
-* The appender that Oozie uses must be named "oozie" (i.e. =log4j.appender.oozie=)
-
-* =log4j.appender.oozie.RollingPolicy.FileNamePattern= must end with "-%d{yyyy-MM-dd-HH}.gz" or "-%d{yyyy-MM-dd-HH}".
-If it ends with ".gz" the old logs will be compressed when rolled
-
-* =log4j.appender.oozie.RollingPolicy.FileNamePattern= must start with the value of =log4j.appender.oozie.File=
-
----++ Previous Default Oozie Logging
-
-Oozie previously used the logging configuration shown below as the default for the =oozie= appender. The other appender that Oozie
-writes to still use a configuration similar to this.
-
----+++ Previous Default Configuration
-
-<verbatim>
-log4j.appender.oozie=org.apache.log4j.DailyRollingFileAppender
-log4j.appender.oozie.File=${oozie.log.dir}/oozie.log
-log4j.appender.oozie.Append=true
-log4j.appender.oozie.layout=org.apache.log4j.PatternLayout
-log4j.appender.oozie.layout.ConversionPattern=%d{ISO8601} %5p %c{1}:%L - %m%n
-log4j.appender.oozie.DatePattern='.'yyyy-MM-dd-HH
-</verbatim>
-
-In this configuration, the active log file will be named =oozie.log= and all old log files will be named =oozie.log.yyyy-MM-dd-HH=
-(where =yyyy-MM-dd-HH= is the time that the log file was created; e.g. 2012-07-21-05). All log files are in the same directory
-(whatever =oozie.log.dir= is assigned to). All older log files are retained. The active log file is rolled every hour.
-
----+++ Restrictions
-
-In order for Oozie logging to work 100% correctly, the following restrictions must be observed (described below and in the
-=oozie-log4j.properties= file):
-
-* The appender that Oozie uses must be named "oozie" (i.e. =log4j.appender.oozie=)
-
-* =log4j.appender.oozie.DatePattern= must end with either "dd" or "HH". If it ends with "HH", the log will be rolled every hour;
-if it ends with "dd", the log will be rolled every day.
-
----++ Other Oozie Logging
-
-While Oozie can technically use any valid log4j Appender or configurations that violate the above restrictions, certain
-features related to logs may be disabled and/or not work correctly, and is thus not advised.
-
-[[index][::Go back to Oozie Documentation Index::]]
-
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/AG_OozieUpgrade.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/AG_OozieUpgrade.twiki b/docs/src/site/twiki/AG_OozieUpgrade.twiki
index 3024064..ec24011 100644
--- a/docs/src/site/twiki/AG_OozieUpgrade.twiki
+++ b/docs/src/site/twiki/AG_OozieUpgrade.twiki
@@ -1,12 +1,12 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
----+!! Oozie Upgrade
+[::Go back to Oozie Documentation Index::](index.html)
-%TOC%
+# Oozie Upgrade
----+ Preparation
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
+
+## Preparation
Make sure there are not Workflows in RUNNING or SUSPENDED status, otherwise the database upgrade will fail.
@@ -14,29 +14,30 @@ Shutdown Oozie and backup the Oozie database.
Copy the oozie-site.xml from your current setup.
----+ Oozie Server Upgrade
+## Oozie Server Upgrade
Expand the new Oozie tarball in a new location.
-Edit the new =oozie-site.xml= setting all custom properties values from the old =oozie-site.xml=
+Edit the new `oozie-site.xml` setting all custom properties values from the old `oozie-site.xml`
IMPORTANT: From Oozie 2.x to Oozie 3.x the names of the database configuration properties have
-changed. Their prefix has changed from =oozie.service.StoreService.*= to =oozie.service.JPAService.*=.
+changed. Their prefix has changed from `oozie.service.StoreService.*` to `oozie.service.JPAService.*`.
Make sure you are using the new prefix.
-After upgrading the Oozie server, the =oozie-setup.sh= MUST be rerun before starting the
+After upgrading the Oozie server, the `oozie-setup.sh` MUST be rerun before starting the
upgraded Oozie server.
Oozie database migration is required when there Oozie database schema changes, like
upgrading from Oozie 2.x to Oozie 3.x.
Configure the oozie-site.xml with the correct database configuration properties as
-explained in the 'Database Configuration' section in [[AG_Install][Oozie Install]].
+explained in the 'Database Configuration' section in [Oozie Install](AG_Install.html).
-Once =oozie-site.xml= has been configured with the database configuration execute the =ooziedb.sh=
+Once `oozie-site.xml` has been configured with the database configuration execute the `ooziedb.sh`
command line tool to upgrade the database:
-<verbatim>
+
+```
$ bin/ooziedb.sh upgrade -run
Validate DB Connection.
@@ -64,22 +65,22 @@ Oozie DB has been upgraded to Oozie version '3.2.0'
The SQL commands have been written to: /tmp/ooziedb-5737263881793872034.sql
$
-</verbatim>
+```
The new version of the Oozie server is ready to be started.
-NOTE: If using MySQL or Oracle, copy the corresponding JDBC driver JAR file to the =libext/= directory before running
-the =ooziedb.sh= command line tool.
+NOTE: If using MySQL or Oracle, copy the corresponding JDBC driver JAR file to the `libext/` directory before running
+the `ooziedb.sh` command line tool.
-NOTE: If instead using the '-run' option, the '-sqlfile <FILE>' option is used, then all the
+NOTE: If instead using the '-run' option, the `-sqlfile <FILE>` option is used, then all the
database changes will be written to the specified file and the database won't be modified.
----+ Oozie Client Upgrade
+## Oozie Client Upgrade
While older Oozie clients work with newer Oozie server, to have access to all the
functionality of the Oozie server the same version of Oozie client should be installed
and used by users.
-[[index][::Go back to Oozie Documentation Index::]]
+[::Go back to Oozie Documentation Index::](index.html)
+
-</noautolink>
http://git-wip-us.apache.org/repos/asf/oozie/blob/4e5b3cb5/docs/src/site/twiki/BundleFunctionalSpec.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/BundleFunctionalSpec.twiki b/docs/src/site/twiki/BundleFunctionalSpec.twiki
index 9749df5..5301429 100644
--- a/docs/src/site/twiki/BundleFunctionalSpec.twiki
+++ b/docs/src/site/twiki/BundleFunctionalSpec.twiki
@@ -1,110 +1,111 @@
-<noautolink>
-[[index][::Go back to Oozie Documentation Index::]]
+
+[::Go back to Oozie Documentation Index::](index.html)
-----
----+!! Oozie Bundle Specification
+# Oozie Bundle Specification
The goal of this document is to define a new oozie abstraction called bundle system specialized in submitting and maintaining a set of coordinator applications.
-%TOC%
+<!-- MACRO{toc|fromDepth=1|toDepth=4} -->
----++ Changelog
+## Changelog
----++ 1. Bundle Overview
+## 1. Bundle Overview
Bundle is a higher-level oozie abstraction that will batch a set of coordinator applications. The user will be able to start/stop/suspend/resume/rerun in the bundle level resulting a better and easy operational control.
-More specifically, the oozie *Bundle* system allows the user to define and execute a bunch of coordinator applications often called a data pipeline. There is no explicit dependency among the coordinator applications in a bundle. However, a user could use the data dependency of coordinator applications to create an implicit data application pipeline.
+More specifically, the oozie **Bundle** system allows the user to define and execute a bunch of coordinator applications often called a data pipeline. There is no explicit dependency among the coordinator applications in a bundle. However, a user could use the data dependency of coordinator applications to create an implicit data application pipeline.
----++ 2. Definitions
+## 2. Definitions
-*Kick-off-time:* The time when a bundle should start and submit coordinator applications.
+**Kick-off-time:** The time when a bundle should start and submit coordinator applications.
-*Bundle Application:* A bundle application defines a set of coordinator applications and when to start those. Normally, bundle applications are parameterized. A bundle application is written in XML.
+**Bundle Application:** A bundle application defines a set of coordinator applications and when to start those. Normally, bundle applications are parameterized. A bundle application is written in XML.
-*Bundle Job:* A bundle job is an executable instance of a bundle application. A job submission is done by submitting a job configuration that resolves all parameters in the application definition.
+**Bundle Job:** A bundle job is an executable instance of a bundle application. A job submission is done by submitting a job configuration that resolves all parameters in the application definition.
-*Bundle Definition Language:* The language used to describe bundle applications.
+**Bundle Definition Language:** The language used to describe bundle applications.
----++ 3. Expression Language for Parameterization
+## 3. Expression Language for Parameterization
Bundle application definitions can be parameterized with variables.
At job submission time all the parameters are resolved into concrete values.
-The parameterization of bundle definitions is done using JSP Expression Language syntax from the [[http://jcp.org/aboutJava/communityprocess/final/jsr152/][JSP 2.0 Specification (JSP.2.3)]], allowing not only to support variables as parameters but also complex expressions.
+The parameterization of bundle definitions is done using JSP Expression Language syntax from the [JSP 2.0 Specification (JSP.2.3)](http://jcp.org/aboutJava/communityprocess/final/jsr152/index.html), allowing not only to support variables as parameters but also complex expressions.
EL expressions can be used in XML attribute values and XML text element values. They cannot be used in XML element and XML attribute names.
----++ 4. Bundle Job
+## 4. Bundle Job
----+++ 4.1. Bundle Job Status
+### 4.1. Bundle Job Status
-At any time, a bundle job is in one of the following status: *PREP, RUNNING, RUNNINGWITHERROR, SUSPENDED, PREPSUSPENDED, SUSPENDEDWITHERROR, PAUSED, PAUSEDWITHERROR, PREPPAUSED, SUCCEEDED, DONEWITHERROR, KILLED, FAILED*.
+At any time, a bundle job is in one of the following status: **PREP, RUNNING, RUNNINGWITHERROR, SUSPENDED, PREPSUSPENDED, SUSPENDEDWITHERROR, PAUSED, PAUSEDWITHERROR, PREPPAUSED, SUCCEEDED, DONEWITHERROR, KILLED, FAILED**.
----+++ 4.2. Transitions of Bundle Job Status
+### 4.2. Transitions of Bundle Job Status
Valid bundle job status transitions are:
- * *PREP --> PREPSUSPENDED | PREPPAUSED | RUNNING | KILLED*
- * *RUNNING --> RUNNINGWITHERROR | SUSPENDED | PAUSED | SUCCEEDED | KILLED*
- * *RUNNINGWITHERROR --> RUNNING | SUSPENDEDWITHERROR | PAUSEDWITHERROR | DONEWITHERROR | FAILED | KILLED*
- * *PREPSUSPENDED --> PREP | KILLED*
- * *SUSPENDED --> RUNNING | KILLED*
- * *SUSPENDEDWITHERROR --> RUNNINGWITHERROR | KILLED*
- * *PREPPAUSED --> PREP | KILLED*
- * *PAUSED --> SUSPENDED | RUNNING | KILLED*
- * *PAUSEDWITHERROR --> SUSPENDEDWITHERROR | RUNNINGWITHERROR | KILLED*
+ * **PREP --> PREPSUSPENDED | PREPPAUSED | RUNNING | KILLED**
+ * **RUNNING --> RUNNINGWITHERROR | SUSPENDED | PAUSED | SUCCEEDED | KILLED**
+ * **RUNNINGWITHERROR --> RUNNING | SUSPENDEDWITHERROR | PAUSEDWITHERROR | DONEWITHERROR | FAILED | KILLED**
+ * **PREPSUSPENDED --> PREP | KILLED**
+ * **SUSPENDED --> RUNNING | KILLED**
+ * **SUSPENDEDWITHERROR --> RUNNINGWITHERROR | KILLED**
+ * **PREPPAUSED --> PREP | KILLED**
+ * **PAUSED --> SUSPENDED | RUNNING | KILLED**
+ * **PAUSEDWITHERROR --> SUSPENDEDWITHERROR | RUNNINGWITHERROR | KILLED**
----+++ 4.3. Details of Status Transitions
-When a bundle job is submitted, oozie parses the bundle job XML. Oozie then creates a record for the bundle with status *PREP* and returns a unique ID.
+### 4.3. Details of Status Transitions
+When a bundle job is submitted, oozie parses the bundle job XML. Oozie then creates a record for the bundle with status **PREP** and returns a unique ID.
-When a user requests to suspend a bundle job that is in *PREP* state, oozie puts the job in status *PREPSUSPENDED*. Similarly, when pause time reaches for a bundle job with *PREP* status, oozie puts the job in status *PREPPAUSED*.
+When a user requests to suspend a bundle job that is in **PREP** state, oozie puts the job in status **PREPSUSPENDED**. Similarly, when pause time reaches for a bundle job with **PREP** status, oozie puts the job in status **PREPPAUSED**.
-Conversely, when a user requests to resume a *PREPSUSPENDED* bundle job, oozie puts the job in status *PREP*. And when pause time is reset for a bundle job that is in *PREPPAUSED* state, oozie puts the job in status *PREP*.
+Conversely, when a user requests to resume a **PREPSUSPENDED** bundle job, oozie puts the job in status **PREP**. And when pause time is reset for a bundle job that is in **PREPPAUSED** state, oozie puts the job in status **PREP**.
There are two ways a bundle job could be started.
- * If =kick-off-time= (defined in the bundle xml) reaches. The default value is null which means starts coordinators NOW.
+* If `kick-off-time` (defined in the bundle xml) reaches. The default value is null which means starts coordinators NOW.
+
+* If user sends a start request to START the bundle.
- * If user sends a start request to START the bundle.
+When a bundle job starts, oozie puts the job in status **RUNNING** and it submits all the coordinator jobs. If any coordinator job goes to **FAILED/KILLED/DONEWITHERROR** state, the bundle job is put in **RUNNINGWITHERROR**
-When a bundle job starts, oozie puts the job in status *RUNNING* and it submits all the coordinator jobs. If any coordinator job goes to *FAILED/KILLED/DONEWITHERROR* state, the bundle job is put in *RUNNINGWITHERROR*
+When a user requests to kill a bundle job, oozie puts the job in status **KILLED** and it sends kill to all submitted coordinator jobs.
-When a user requests to kill a bundle job, oozie puts the job in status *KILLED* and it sends kill to all submitted coordinator jobs.
+When a user requests to suspend a bundle job that is in **RUNNING** status, oozie puts the job in status **SUSPENDED** and it suspends all submitted coordinator jobs. Similarly, when a user requests to suspend a bundle job that is in **RUNNINGWITHERROR** status, oozie puts the job in status **SUSPENDEDWITHERROR** and it suspends all submitted coordinator jobs.
-When a user requests to suspend a bundle job that is in *RUNNING* status, oozie puts the job in status *SUSPENDED* and it suspends all submitted coordinator jobs. Similarly, when a user requests to suspend a bundle job that is in *RUNNINGWITHERROR* status, oozie puts the job in status *SUSPENDEDWITHERROR* and it suspends all submitted coordinator jobs.
+When pause time reaches for a bundle job that is in **RUNNING** status, oozie puts the job in status **PAUSED**. When pause time reaches for a bundle job that is in **RUNNINGWITHERROR** status, oozie puts the job in status **PAUSEDWITHERROR**.
-When pause time reaches for a bundle job that is in *RUNNING* status, oozie puts the job in status *PAUSED*. When pause time reaches for a bundle job that is in *RUNNINGWITHERROR* status, oozie puts the job in status *PAUSEDWITHERROR*.
+Conversely, when a user requests to resume a **SUSPENDED** bundle job, oozie puts the job in status **RUNNING**. Similarly, when a user requests to resume a **SUSPENDEDWITHERROR** bundle job, oozie puts the job in status **RUNNINGWITHERROR**. And when pause time is reset for a bundle job and job status is **PAUSED**, oozie puts the job in status **RUNNING**. Similarly, when the pause time is reset for a bundle job and job status is **PAUSEDWITHERROR**, oozie puts the job in status **RUNNINGWITHERROR**
-Conversely, when a user requests to resume a *SUSPENDED* bundle job, oozie puts the job in status *RUNNING*. Similarly, when a user requests to resume a *SUSPENDEDWITHERROR* bundle job, oozie puts the job in status *RUNNINGWITHERROR*. And when pause time is reset for a bundle job and job status is *PAUSED*, oozie puts the job in status *RUNNING*. Similarly, when the pause time is reset for a bundle job and job status is *PAUSEDWITHERROR*, oozie puts the job in status *RUNNINGWITHERROR*
+When all the coordinator jobs finish, oozie updates the bundle status accordingly. If all coordinators reaches to the _same_ terminal state, bundle job status also move to the same status. For example, if all coordinators are **SUCCEEDED**, oozie puts the bundle job into **SUCCEEDED** status. However, if all coordinator jobs don't finish with the same status, oozie puts the bundle job into **DONEWITHERROR**.
-When all the coordinator jobs finish, oozie updates the bundle status accordingly. If all coordinators reaches to the _same_ terminal state, bundle job status also move to the same status. For example, if all coordinators are *SUCCEEDED*, oozie puts the bundle job into *SUCCEEDED* status. However, if all coordinator jobs don't finish with the same status, oozie puts the bundle job into *DONEWITHERROR*.
-
----+++ 4.3. Bundle Application Definition
+### 4.3. Bundle Application Definition
A bundle definition is defined in XML by a name, controls and one or more coordinator application specifications:
- * *%BLUE% name: %ENDCOLOR%* The name for the bundle job.
- * *%BLUE% controls: %ENDCOLOR%* The control specification for the bundle.
- * *%BLUE% kick-off-time: %ENDCOLOR%* It defines when the bundle job should start and submit the coordinator applications. This field is optional and the default is *NOW* that means the job should start right-a-way.
- * *%BLUE% coordinator: %ENDCOLOR%* Coordinator application specification. There should be at least one coordinator application in any bundle.
- * *%BLUE% name: %ENDCOLOR%* Name of the coordinator application. It can be used for referring this application through bundle to control such as kill, suspend, rerun.
- * *%BLUE% enabled: %ENDCOLOR%* Enabled can be used to enable or disable a coordinator. It is optional. The default value for enabled is true.
- * *%BLUE% app-path: %ENDCOLOR%* Path of the coordinator application definition in hdfs. This is a mandatory element.
- * *%BLUE% configuration: %ENDCOLOR%* A hadoop like configuration to parameterize corresponding coordinator application. This is optional.
- * *%BLUE% Parameterization: %ENDCOLOR%* Configuration properties that are a valid Java identifier, [A-Za-z_][0-9A-Za-z_]*, are available as =${NAME}= variables within the bundle application definition. Configuration properties that are not a valid Java identifier, for example =job.tracker=, are available via the =${bundle:conf(String name)}= function. Valid Java identifier properties are available via this function as well.
+ * **<font color="#0000ff"> name: </font>** The name for the bundle job.
+ * **<font color="#0000ff"> controls: </font>** The control specification for the bundle.
+ * **<font color="#0000ff"> kick-off-time: </font>** It defines when the bundle job should start and submit the coordinator applications. This field is optional and the default is **NOW** that means the job should start right-a-way.
+ * **<font color="#0000ff"> coordinator: </font>** Coordinator application specification. There should be at least one coordinator application in any bundle.
+ * **<font color="#0000ff"> name: </font>** Name of the coordinator application. It can be used for referring this application through bundle to control such as kill, suspend, rerun.
+ * **<font color="#0000ff"> enabled: </font>** Enabled can be used to enable or disable a coordinator. It is optional. The default value for enabled is true.
+ * **<font color="#0000ff"> app-path: </font>** Path of the coordinator application definition in hdfs. This is a mandatory element.
+ * **<font color="#0000ff"> configuration: </font>** A hadoop like configuration to parameterize corresponding coordinator application. This is optional.
+ * **<font color="#0000ff"> Parameterization: </font>** Configuration properties that are a valid Java identifier, [A-Za-z_][0-9A-Za-z_]*, are available as `${NAME}` variables within the bundle application definition. Configuration properties that are not a valid Java identifier, for example `job.tracker`, are available via the `${bundle:conf(String name)}` function. Valid Java identifier properties are available via this function as well.
+
+**<font color="#800080">Syntax: </font>**
-*%PURPLE% Syntax: %ENDCOLOR%*
-<verbatim>
- <bundle-app name=[NAME] xmlns='uri:oozie:bundle:0.1'>
+```
+ <bundle-app name=[NAME] xmlns='uri:oozie:bundle:0.1'>
<controls>
<kick-off-time>[DATETIME]</kick-off-time>
</controls>
@@ -119,16 +120,17 @@ A bundle definition is defined in XML by a name, controls and one or more coordi
</configuration>
</coordinator>
...
-</bundle-app>
-</verbatim>
+</bundle-app>
+```
+
+**<font color="#008000"> Examples: </font>**
-*%GREEN% Examples: %ENDCOLOR%*
+**A Bundle Job that maintains two coordinator applications:**
-*A Bundle Job that maintains two coordinator applications:*
-<verbatim>
-<bundle-app name='APPNAME' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns='uri:oozie:bundle:0.1'>
+```
+<bundle-app name='APPNAME' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns='uri:oozie:bundle:0.1'>
<controls>
<kick-off-time>${kickOffTime}</kick-off-time>
</controls>
@@ -159,18 +161,19 @@ A bundle definition is defined in XML by a name, controls and one or more coordi
</configuration>
</coordinator>
</bundle-app>
-</verbatim>
+```
----+++ 4.4. Bundle Formal Parameters
-As of schema 0.2, a list of formal parameters can be provided which will allow Oozie to verify, at submission time, that said
-properties are actually specified (i.e. before the job is executed and fails). Default values can also be provided.
+### 4.4. Bundle Formal Parameters
+As of schema 0.2, a list of formal parameters can be provided which will allow Oozie to verify, at submission time, that said
+properties are actually specified (i.e. before the job is executed and fails). Default values can also be provided.
-*Example:*
+**Example:**
The previous Bundle Job application definition with formal parameters:
-<verbatim>
-<bundle-app name='APPNAME' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns='uri:oozie:bundle:0.2'>
+
+```
+<bundle-app name='APPNAME' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns='uri:oozie:bundle:0.2'>
<parameters>
<property>
<name>appPath</name>
@@ -210,17 +213,17 @@ The previous Bundle Job application definition with formal parameters:
</configuration>
</coordinator>
</bundle-app>
-</verbatim>
+```
-In the above example, if =appPath= is not specified, Oozie will print an error message instead of submitting the job. If
-=appPath2= is not specified, Oozie will use the default value, =hdfs://foo:8020/user/joe/job/job.properties=.
+In the above example, if `appPath` is not specified, Oozie will print an error message instead of submitting the job. If
+`appPath2` is not specified, Oozie will use the default value, `hdfs://foo:8020/user/joe/job/job.properties`.
----++ 5. User Propagation
+## 5. User Propagation
-When submitting a bundle job, the configuration must contain a =user.name= property. If security is enabled, Oozie must ensure that the value of the =user.name= property in the configuration match the user credentials present in the protocol (web services) request.
+When submitting a bundle job, the configuration must contain a `user.name` property. If security is enabled, Oozie must ensure that the value of the `user.name` property in the configuration match the user credentials present in the protocol (web services) request.
-When submitting a bundle job, the configuration may contain the =oozie.job.acl= property (the =group.name= property
+When submitting a bundle job, the configuration may contain the `oozie.job.acl` property (the `group.name` property
has been deprecated). If authorization is enabled, this property is treated as as the ACL for the job, it can contain
user and group IDs separated by commas.
@@ -228,15 +231,15 @@ The specified user and ACL are assigned to the created bundle job.
Oozie must propagate the specified user and ACL to the system executing its children jobs (coordinator jobs).
----++ 6. Bundle Application Deployment
+## 6. Bundle Application Deployment
A bundle application consist exclusively of bundle application definition and associated coordinator application specifications. They must be installed in an HDFS directory. To submit a job for a bundle application, the full HDFS path to bundle application definition must be specified.
----+++ 6.1. Organizing Bundle Applications
+### 6.1. Organizing Bundle Applications
TBD.
----++ 7. Bundle Job Submission
+## 7. Bundle Job Submission
When a bundle job is submitted to Oozie, the submitter must specified all the required job properties plus the HDFS path to the bundle application definition for the job.
@@ -245,9 +248,10 @@ The bundle application definition HDFS path must be specified in the 'oozie.bund
All the bundle job properties, the HDFS path for the bundle application, the 'user.name' and 'oozie.job.acl' must be
submitted to the Oozie using an XML configuration file (Hadoop XML configuration file).
-*%GREEN% Example: %ENDCOLOR%*:
+**<font color="#008000"> Example: </font>**:
-<verbatim>
+
+```
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<property>
@@ -260,23 +264,24 @@ submitted to the Oozie using an XML configuration file (Hadoop XML configuration
</property>
...
</configuration>
-</verbatim>
+```
----++ 8. Bundle Rerun
----+++ Rerunning a Bundle Job
+## 8. Bundle Rerun
+### 8.1 Rerunning a Bundle Job
Oozie provides a way of rerunning a bundle job. The user could request to rerun a subset of coordinators within a bundle by defining a list of coordinator's names. In addition, a user could define a list of dates or ranges of dates (in UTC format) to rerun for those time windows.
-There is a way of asking whether to cleanup all output directories before rerun. By default, oozie will remove all output directories. Moreover, there is an option by which a user could ask to re-calculate the dynamic input directories defined by latest function in coordinators.
+There is a way of asking whether to cleanup all output directories before rerun. By default, oozie will remove all output directories. Moreover, there is an option by which a user could ask to re-calculate the dynamic input directories defined by latest function in coordinators.
+
+### 8.2 Rerun Arguments
----+++ Rerun Arguments
-<verbatim>
+```
$oozie job -rerun <bundle_Job_id> [-coordinator <list of coordinator name separate by comma>
[-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z]
[-nocleanup] [-refresh]
-</verbatim>
+```
- * The =rerun= option reruns a bundle job that is *not* in (=KILLED=, =FAILED=, =PREP=, =PREPPAUSED=, =PREPSUSPENDED=).
- * Rerun a bundle job that is in =PAUSED= state will reset the paused time.
+ * The `rerun` option reruns a bundle job that is *not* in (`KILLED`, `FAILED`, `PREP`, `PREPPAUSED`, `PREPSUSPENDED`).
+ * Rerun a bundle job that is in `PAUSED` state will reset the paused time.
* The option -coordinator determines the name of coordinator that will be rerun. By default all coordinators are rerun.
* Multiple ranges can be used in -date. See the above examples.
* The dates specified in -date must be UTC.
@@ -285,18 +290,19 @@ $oozie job -rerun <bundle_Job_id> [-coordinator <list of coordinator name separa
* If -refresh is set, all dependencies will be re-checked; otherwise only missed dependencies will be checked for the corresponding coordinators.
-After the command is executed the rerun bundle job will be in =RUNNING= status.
+After the command is executed the rerun bundle job will be in `RUNNING` status.
+
+Refer to the [Rerunning Coordinator Actions](DG_CoordinatorRerun.html) for details on rerun of coordinator job.
-Refer to the [[DG_CoordinatorRerun][Rerunning Coordinator Actions]] for details on rerun of coordinator job.
+## Appendixes
----++ Appendixes
+### Appendix A, Oozie Bundle XML-Schema
----+++ Appendix A, Oozie Bundle XML-Schema
+#### Oozie Bundle Schema 0.1
----++++ Oozie Bundle Schema 0.1
-<verbatim>
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:bundle="uri:oozie:bundle:0.1"
elementFormDefault="qualified" targetNamespace="uri:oozie:bundle:0.1">
@@ -340,11 +346,12 @@ Refer to the [[DG_CoordinatorRerun][Rerunning Coordinator Actions]] for details
</xs:sequence>
</xs:complexType>
</xs:schema>
-</verbatim>
+```
----++++ Oozie Bundle Schema 0.2
+#### Oozie Bundle Schema 0.2
-<verbatim>
+
+```
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:bundle="uri:oozie:bundle:0.2"
elementFormDefault="qualified" targetNamespace="uri:oozie:bundle:0.2">
@@ -403,9 +410,9 @@ Refer to the [[DG_CoordinatorRerun][Rerunning Coordinator Actions]] for details
</xs:sequence>
</xs:complexType>
</xs:schema>
-</verbatim>
+```
+
+[::Go back to Oozie Documentation Index::](index.html)
-[[index][::Go back to Oozie Documentation Index::]]
-</noautolink>